Recent

Author Topic: Optimize memory manager on x64  (Read 971 times)

Artem3213212

  • New member
  • *
  • Posts: 6
Optimize memory manager on x64
« on: November 11, 2018, 02:21:13 pm »
In theme http://forum.lazarus.freepascal.org/index.php/topic,42852.0.html was:
3) I think that we must make possible to use more than 4 GB*sizeof(T) memory in one TDeque => 4294967295 - isn't memory limit.
It is, at least it is the memory limit of the default memorymanager for one single entity. It is not the limit of total memory.
As such it would require a specialized memory manager. This is not a limitation of FreePascal perse, you will also find this limitation in most other languages.
Why fpc don't use 32 memory manager on 32bit platforms and 64 on 64bit?

Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #1 on: November 11, 2018, 02:33:38 pm »
It does. It is entries div ( SizeOf(NativePtr) minus already allocated memory minus allocated stacks minus OS allocated memory minus everything allocated by other processes) or there about.
It is just that the size of a single entry in a dequeue is limited to High(Cardinal), which makes sense, otherwise a dequeue can have just one (1) entry... <sigh  >:D >:D >
« Last Edit: November 11, 2018, 02:42:22 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

Artem3213212

  • New member
  • *
  • Posts: 6
Re: Optimize memory manager on x64
« Reply #2 on: November 11, 2018, 02:40:54 pm »
Ok, why in 64bit memory manager i can't get more than 4 GB memory?

Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #3 on: November 11, 2018, 02:42:50 pm »
Ok, why in 64bit memory manager i can't get more than 4 GB memory?
Per entry...? <sigh again  >:D > Most programmers are very good in logic. Of course FPC can address any amount of memory, but not in the context of a single dequeue element.
« Last Edit: November 11, 2018, 02:44:39 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

Artem3213212

  • New member
  • *
  • Posts: 6
Re: Optimize memory manager on x64
« Reply #4 on: November 11, 2018, 04:22:39 pm »
You say that i can get more then 4Gb mem but not in one block?
I think it is very strange.
« Last Edit: November 11, 2018, 04:32:50 pm by Artem3213212 »

Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #5 on: November 11, 2018, 04:35:54 pm »
No. It is not strange. Separate the two questions...
-- Do you need a dequeue single element to be the size of all memory? Then you are not thinking. To have a structure like a dequeue it needs to be at most half the size of available memory to have a dequeue...THINK! and even less than that if you want to use it...THINK!
-- Do you need one huge contiguous block of memory over 4 GB? That can be arranged, but only if it is available.
The OS can fragment it as well. E.g. Windows can not - and does not - guarantee it without tricks like file mappings, because even windows has a page mapping of 4 K per page not even 4 G per page.
What do you want? As it stands you are on the border of being called someone closely resembling of having the brains of an amoeba. Which I respectfully consider not an option.. O:-)
« Last Edit: November 11, 2018, 04:46:39 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

Zoran

  • Hero Member
  • *****
  • Posts: 1283
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: Optimize memory manager on x64
« Reply #6 on: November 11, 2018, 04:59:09 pm »
But, Thaddy, suppose your 64-bit application has 16 gb available... Why the deque could not have three 5-gb elements?

The addressable space in 64-bit application is not just two times bigger, but 232 times bigger than in 32-bit, right?
So, to me it would make sense to have more than one element whose size is larger than 4 gb.

Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #7 on: November 11, 2018, 05:31:39 pm »
But, Thaddy, suppose your 64-bit application has 16 gb available... Why the deque could not have three 5-gb elements?

The addressable space in 64-bit application is not just two times bigger, but 232 times bigger than in 32-bit, right?
So, to me it would make sense to have more than one element whose size is larger than 4 gb.
That is a limitation only for the memory manager, not to be confused by accessible memory. The current memory manager makes a completely sane choice, (One of my Windows has 128 GB and 8 cores, I still never needed more than a single 4GB on a single entry in a dequeue, far less: is is everyone handling data in the way I am? I know there are, they will give you the same answer.)
If you want to handle big data at least have a grasp of how the memory manager works: the code is there, fpc is open source..... writing one to suit your needs is very,very,very simple. < >:D >:D >:D >
Writing a good one is not as simple. FPC has a good one.
« Last Edit: November 11, 2018, 05:33:31 pm by Thaddy »
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

LemonParty

  • New member
  • *
  • Posts: 13
Re: Optimize memory manager on x64
« Reply #8 on: November 11, 2018, 06:17:15 pm »
Have fun!

By the way, sometimes you can't get a solid chunk of memory (even if there is enough free space) because of memory fragmentation. Programers usually splits the large chunks into a small ones.

ASerge

  • Hero Member
  • *****
  • Posts: 998
Re: Optimize memory manager on x64
« Reply #9 on: November 11, 2018, 06:51:28 pm »
Ok, why in 64bit memory manager i can't get more than 4 GB memory?
You can. This work in Win64:
Code: Pascal  [Select]
  1. {$APPTYPE CONSOLE}
  2. program Project1;
  3.  
  4. var
  5.   P: Pointer;
  6. begin
  7.   P := GetMem(5 * 1024 * 1024 * 1024);
  8.   try
  9.     Writeln(MemSize(P));
  10.     Readln;
  11.   finally
  12.     FreeMem(P);
  13.   end;
  14. end.

Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #10 on: November 11, 2018, 09:23:14 pm »
Yes. we know that. But He also asked why the element size for a dequeue is limited...
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.

Zoran

  • Hero Member
  • *****
  • Posts: 1283
    • http://wiki.lazarus.freepascal.org/User:Zoran
Re: Optimize memory manager on x64
« Reply #11 on: November 12, 2018, 01:16:10 am »
The current memory manager makes a completely sane choice,

Probably so, but I was just questioning your arguments in previous posts...



Thaddy

  • Hero Member
  • *****
  • Posts: 7182
Re: Optimize memory manager on x64
« Reply #12 on: November 12, 2018, 07:45:17 am »
Probably so, but I was just questioning your arguments in previous posts...
I explained I believe that all contiguous available memory is addressable, but not in the context of members of a dequeue or any other list.
inline variables like in D10.3 are a bit like Brexit: if you are given the wrong information it sounds like a good idea. Every kid loves candy, but it makes you fat and your teeth will disappear.