Recent

Author Topic: Large (huge) pages in Lazarus  (Read 4731 times)

B4R

  • Newbie
  • Posts: 6
Large (huge) pages in Lazarus
« on: June 29, 2024, 07:33:58 pm »
Previously, I worked with Delphi 5-8 and a little bit with XE.
Now, I am looking to create an application that scans through large files and computes HMAC SHA256 from blocks of data contained therein. I thought that Lazarus with Indy can handle the task.

Ideally, the buffer in which these files will be kept, should be allocated as large pages also known as huge pages. This is trivial for me in C++. I am very reluctant to code this program in C++ because it needs a GUI, and I am not up to speed with that.

I searched and failed to find information on whether they are available in Lazarus. Are they, and if yes, then how can I enable them?

Many thanks!

440bx

  • Hero Member
  • *****
  • Posts: 4751
Re: Large (huge) pages in Lazarus
« Reply #1 on: June 29, 2024, 07:45:04 pm »
You didn't mention the O/S you're using. 

If you're using Windows then you use VirtualAlloc (or its Ex sibling) to allocate 4MB pages for a given allocation.

If you're using a different O/S then I cannot provide an answer.

HTH.


(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

B4R

  • Newbie
  • Posts: 6
Re: Large (huge) pages in Lazarus
« Reply #2 on: June 29, 2024, 07:56:01 pm »
Windows.

Is VirtualAlloc wrapped in Lazarus, or am I to import it from Windowze?

And will only my own buffers be allocated as large pages? I mean that when I will be calling Indy to compute HMACs, will it use regular pages internally? Is there a way to use large pages globally for all allocations in the whole process?
« Last Edit: June 29, 2024, 07:57:55 pm by B4R »

Red_prig

  • Full Member
  • ***
  • Posts: 153
Re: Large (huge) pages in Lazarus
« Reply #3 on: June 29, 2024, 08:10:17 pm »
If you want this globally, then you need to write your own custom memory manager, otherwise it will be an individual allocation via VirtualAlloc

440bx

  • Hero Member
  • *****
  • Posts: 4751
Re: Large (huge) pages in Lazarus
« Reply #4 on: June 29, 2024, 08:33:06 pm »
Windows.

Is VirtualAlloc wrapped in Lazarus, or am I to import it from Windowze?

And will only my own buffers be allocated as large pages? I mean that when I will be calling Indy to compute HMACs, will it use regular pages internally? Is there a way to use large pages globally for all allocations in the whole process?
Answer to the first question: it's included in the Windows unit.  As long as you have "uses Windows", you can use VirtualAlloc and VirtualAllocEx.

Answer to second question: _you_ tell VirtualAlloc to size of the page to use.  You need to do this for every buffer you allocate.  Therefore, controlling page size can only be done for the buffers you allocate.   Strictly speaking that last sentence is not absolutely true, it is possible to force code in other used units to also use large pages but, that's a lot, and I really mean a LOT, more complicated.

Therefore, it is not possible NOR desirable to use large pages for all allocations in the whole process.  It makes no sense to grab a 4MB page to store a 10 character window title or class name or whathaveyou. 

Another thing which is very important when allocating large pages is that the O/S must find a single  _physical_ block that is 4MB which becomes the page.  The problem is, since most allocations consist of 4KB pages it is possible that there are no blocks or very few blocks of 4MB.   IOW, if you ask for 1GB made of large pages, the O/S needs to find 256 blocks of 4MB.  Physical memory fragmentation may cause the number of 4MB blocks to be less than that in which case the allocation will fail.  OTH, 1GB is 262,144 pages of memory, none of those pages need to be consecutive (though many will be), therefore the allocation is almost guaranteed to succeed because Windows can shuffle things around to satisfy the request.  Another big disadvantage of large pages, unlike regular size pages, is that those are _not_ doled out on demand (because Windows won't take the risk of running out of 4MB blocks, a problem it doesn't have to face when dealing with 4KB pages.)

The bottom line is this: it is _extremely_ rare for a user mode program to derive significant benefit from using large pages.  From what you mentioned, it sounds like you should simply map the files.  It's very doubtful you'll have a perceptible gain using 4MB pages.

HTH.
« Last Edit: June 29, 2024, 08:35:38 pm by 440bx »
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

Thaddy

  • Hero Member
  • *****
  • Posts: 16199
  • Censorship about opinions does not belong here.
Re: Large (huge) pages in Lazarus
« Reply #5 on: June 29, 2024, 08:58:27 pm »
I also think he just means memory mapped files, since that at least makes sense.
There should not be much difference between C++ and FPC in handling that.
« Last Edit: June 29, 2024, 09:01:31 pm by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

Laur

  • New Member
  • *
  • Posts: 35
Re: Large (huge) pages in Lazarus
« Reply #6 on: July 12, 2024, 05:46:28 pm »
Previously, I worked with Delphi 5-8 and a little bit with XE.
Now, I am looking to create an application that scans through large files and computes HMAC SHA256 from blocks of data contained therein. I thought that Lazarus with Indy can handle the task.

Ideally, the buffer in which these files will be kept, should be allocated as large pages also known as huge pages. This is trivial for me in C++. I am very reluctant to code this program in C++ because it needs a GUI, and I am not up to speed with that.

I searched and failed to find information on whether they are available in Lazarus. Are they, and if yes, then how can I enable them?

Many thanks!

Use simply GetMem... this is unlimited (should be).

or directly: in win GlobalAlloc... still works.

But loading of GBs to memory is rater stupid idea.


Joanna from IRC

  • Hero Member
  • *****
  • Posts: 1236
Re: Large (huge) pages in Lazarus
« Reply #7 on: July 12, 2024, 06:27:46 pm »
Quote
rater stupid idea
what an amazing post please tell us more about yourself  :D
✨ 🙋🏻‍♀️ More Pascal enthusiasts are needed on IRC .. https://libera.chat/guides/ IRC.LIBERA.CHAT  Ports [6667 plaintext ] or [6697 secure] channel #fpc  #pascal Please private Message me if you have any questions or need assistance. 💁🏻‍♀️

Thaddy

  • Hero Member
  • *****
  • Posts: 16199
  • Censorship about opinions does not belong here.
Re: Large (huge) pages in Lazarus
« Reply #8 on: July 12, 2024, 06:34:40 pm »
I am eating, Joanna, don't make me ROFL. :-[ You are right.
If I smell bad code it usually is bad code and that includes my own code.

Joanna from IRC

  • Hero Member
  • *****
  • Posts: 1236
Re: Large (huge) pages in Lazarus
« Reply #9 on: July 12, 2024, 06:49:51 pm »
 :D
✨ 🙋🏻‍♀️ More Pascal enthusiasts are needed on IRC .. https://libera.chat/guides/ IRC.LIBERA.CHAT  Ports [6667 plaintext ] or [6697 secure] channel #fpc  #pascal Please private Message me if you have any questions or need assistance. 💁🏻‍♀️

MarkMLl

  • Hero Member
  • *****
  • Posts: 8041
Re: Large (huge) pages in Lazarus
« Reply #10 on: July 12, 2024, 07:59:01 pm »
Quote
rater stupid idea
what an amazing post please tell us more about yourself  :D

Joanna, stop it. He's making a good point: going out of ones way to use something like this is of questionable use on a modern OS since stuff will start getting paged out to disc way before all memory is used.

However in this particular case I suggest that the important thing is the maximum block size that the library implementing the algorithm can use, whether that in any way falls foul of the block sizes allocated by the OS/runtimes, and the most efficient way to map those blocks onto the files being processed.

It might even be that enough work has been put into solutions using mmap() etc. that attempting to transcribe the code into "pure" Pascal is counterproductive :-/

MarkMLl
MT+86 & Turbo Pascal v1 on CCP/M-86, multitasking with LAN & graphics in 128Kb.
Logitech, TopSpeed & FTL Modula-2 on bare metal (Z80, '286 protected mode).
Pet hate: people who boast about the size and sophistication of their computer.
GitHub repositories: https://github.com/MarkMLl?tab=repositories

Joanna from IRC

  • Hero Member
  • *****
  • Posts: 1236
Re: Large (huge) pages in Lazarus
« Reply #11 on: July 13, 2024, 01:53:53 am »
Quote
Joanna, stop it. He's making a good point:
That’s right!, why bother with translators ,spell checkers or politeness nonsense when he can use  misspelled words and insults to prove that he gives good advice ? Laur +1 keep up the good work  :D
« Last Edit: July 13, 2024, 02:00:06 am by Joanna »
✨ 🙋🏻‍♀️ More Pascal enthusiasts are needed on IRC .. https://libera.chat/guides/ IRC.LIBERA.CHAT  Ports [6667 plaintext ] or [6697 secure] channel #fpc  #pascal Please private Message me if you have any questions or need assistance. 💁🏻‍♀️

duralast

  • New Member
  • *
  • Posts: 23
Re: Large (huge) pages in Lazarus
« Reply #12 on: July 13, 2024, 02:13:33 pm »
Quote
rater
OMG, someone mistakenly left out the H. Oh, the horrors!!!!!!

An admin needs to change Joanna's name to Chernobyl, which would better reflect her radioactivity.

440bx

  • Hero Member
  • *****
  • Posts: 4751
Re: Large (huge) pages in Lazarus
« Reply #13 on: July 13, 2024, 02:44:06 pm »
I believe Joanna has a tendency to overreact but, in this case, I think she  has a valid point.

Laur's advice on how to deal with the problem is, to put it in a nice way, "not particularly good" and in addition to that it is topped with a put down.

Joanna is right, that was quite a post.
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

Joanna from IRC

  • Hero Member
  • *****
  • Posts: 1236
Re: Large (huge) pages in Lazarus
« Reply #14 on: July 13, 2024, 05:11:03 pm »
Quote
rater
OMG, someone mistakenly left out the H. Oh, the horrors!!!!!!

An admin needs to change Joanna's name to Chernobyl, which would better reflect her radioactivity.
I hope you are single because I want to marry you!!  :D
✨ 🙋🏻‍♀️ More Pascal enthusiasts are needed on IRC .. https://libera.chat/guides/ IRC.LIBERA.CHAT  Ports [6667 plaintext ] or [6697 secure] channel #fpc  #pascal Please private Message me if you have any questions or need assistance. 💁🏻‍♀️

 

TinyPortal © 2005-2018