Large (huge) pages in Lazarus

[B4R]

Previously, I worked with Delphi 5-8 and a little bit with XE.
Now, I am looking to create an application that scans through large files and computes HMAC SHA256 from blocks of data contained therein. I thought that Lazarus with Indy can handle the task.

Ideally, the buffer in which these files will be kept, should be allocated as large pages also known as huge pages. This is trivial for me in C++. I am very reluctant to code this program in C++ because it needs a GUI, and I am not up to speed with that.

I searched and failed to find information on whether they are available in Lazarus. Are they, and if yes, then how can I enable them?

Many thanks!

[440bx]

You didn't mention the O/S you're using. 

If you're using Windows then you use VirtualAlloc (or its Ex sibling) to allocate 4MB pages for a given allocation.

If you're using a different O/S then I cannot provide an answer.

HTH.

[B4R]

Windows.

Is VirtualAlloc wrapped in Lazarus, or am I to import it from Windowze?

And will only my own buffers be allocated as large pages? I mean that when I will be calling Indy to compute HMACs, will it use regular pages internally? Is there a way to use large pages globally for all allocations in the whole process?

[Red_prig]

If you want this globally, then you need to write your own custom memory manager, otherwise it will be an individual allocation via VirtualAlloc

[440bx]

Windows.

Is VirtualAlloc wrapped in Lazarus, or am I to import it from Windowze?

And will only my own buffers be allocated as large pages? I mean that when I will be calling Indy to compute HMACs, will it use regular pages internally? Is there a way to use large pages globally for all allocations in the whole process?

Answer to the first question: it's included in the Windows unit.  As long as you have "uses Windows", you can use VirtualAlloc and VirtualAllocEx.

Answer to second question: _you_ tell VirtualAlloc to size of the page to use.  You need to do this for every buffer you allocate.  Therefore, controlling page size can only be done for the buffers you allocate.   Strictly speaking that last sentence is not absolutely true, it is possible to force code in other used units to also use large pages but, that's a lot, and I really mean a LOT, more complicated.

Therefore, it is not possible NOR desirable to use large pages for all allocations in the whole process.  It makes no sense to grab a 4MB page to store a 10 character window title or class name or whathaveyou. 

Another thing which is very important when allocating large pages is that the O/S must find a single  _physical_ block that is 4MB which becomes the page.  The problem is, since most allocations consist of 4KB pages it is possible that there are no blocks or very few blocks of 4MB.   IOW, if you ask for 1GB made of large pages, the O/S needs to find 256 blocks of 4MB.  Physical memory fragmentation may cause the number of 4MB blocks to be less than that in which case the allocation will fail.  OTH, 1GB is 262,144 pages of memory, none of those pages need to be consecutive (though many will be), therefore the allocation is almost guaranteed to succeed because Windows can shuffle things around to satisfy the request.  Another big disadvantage of large pages, unlike regular size pages, is that those are _not_ doled out on demand (because Windows won't take the risk of running out of 4MB blocks, a problem it doesn't have to face when dealing with 4KB pages.)

The bottom line is this: it is _extremely_ rare for a user mode program to derive significant benefit from using large pages.  From what you mentioned, it sounds like you should simply map the files.  It's very doubtful you'll have a perceptible gain using 4MB pages.

HTH.

[Thaddy]

I also think he just means memory mapped files, since that at least makes sense.
There should not be much difference between C++ and FPC in handling that.

[Laur]

Previously, I worked with Delphi 5-8 and a little bit with XE.
Now, I am looking to create an application that scans through large files and computes HMAC SHA256 from blocks of data contained therein. I thought that Lazarus with Indy can handle the task.

Ideally, the buffer in which these files will be kept, should be allocated as large pages also known as huge pages. This is trivial for me in C++. I am very reluctant to code this program in C++ because it needs a GUI, and I am not up to speed with that.

I searched and failed to find information on whether they are available in Lazarus. Are they, and if yes, then how can I enable them?

Many thanks!

Use simply GetMem... this is unlimited (should be).

or directly: in win GlobalAlloc... still works.

But loading of GBs to memory is rater stupid idea.

[Joanna from IRC]

rater stupid idea

what an amazing post please tell us more about yourself 

[Thaddy]

I am eating, Joanna, don't make me ROFL. You are right.

[Joanna from IRC]

 

[MarkMLl]

rater stupid idea

what an amazing post please tell us more about yourself 

Joanna, stop it. He's making a good point: going out of ones way to use something like this is of questionable use on a modern OS since stuff will start getting paged out to disc way before all memory is used.

However in this particular case I suggest that the important thing is the maximum block size that the library implementing the algorithm can use, whether that in any way falls foul of the block sizes allocated by the OS/runtimes, and the most efficient way to map those blocks onto the files being processed.

It might even be that enough work has been put into solutions using mmap() etc. that attempting to transcribe the code into "pure" Pascal is counterproductive :-/

MarkMLl

[Joanna from IRC]

Joanna, stop it. He's making a good point:

That’s right!, why bother with translators ,spell checkers or politeness nonsense when he can use  misspelled words and insults to prove that he gives good advice ? Laur +1 keep up the good work 

[duralast]

rater

OMG, someone mistakenly left out the H. Oh, the horrors!!!!!!

An admin needs to change Joanna's name to Chernobyl, which would better reflect her radioactivity.

[440bx]

I believe Joanna has a tendency to overreact but, in this case, I think she  has a valid point.

Laur's advice on how to deal with the problem is, to put it in a nice way, "not particularly good" and in addition to that it is topped with a put down.

Joanna is right, that was quite a post.

[Joanna from IRC]

rater

OMG, someone mistakenly left out the H. Oh, the horrors!!!!!!

An admin needs to change Joanna's name to Chernobyl, which would better reflect her radioactivity.

I hope you are single because I want to marry you!!