Recent

Author Topic: Laz 1.8.2 No memory left  (Read 32270 times)

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #45 on: March 09, 2018, 07:55:28 am »
I thought on 32 bits machines limit was 4GB allocation more or less... It happens around 2GB to HANG
BTW: On Windows (win32 parte) you can use more than 2G RAM if you declare it to the system in the PE -Header.

I have 24 GB RAM on my machine, but in win32 this is meaningless. Can you share the code for tests ?

Andreas
regards
Andreas

sam707

  • Guest
Re: Laz 1.8.2 No memory left
« Reply #46 on: March 09, 2018, 05:33:49 pm »
I thought on 32 bits machines limit was 4GB allocation more or less... It happens around 2GB to HANG
BTW: On Windows (win32 parte) you can use more than 2G RAM if you declare it to the system in the PE -Header.

I have 24 GB RAM on my machine, but in win32 this is meaningless. Can you share the code for tests ?

Andreas

YES there is a registry key to tell win32 to manage more than 4GB (large adddresses), I can't remember which key at the moment (Im focused on linux) but I used to switch that one ON

code is simple I described it before... make a stress loop on a button click

here is a kind of algo to run under GDB

for i:=1 to 1000 do begin
 list.add(getmem(10*1024*1024) (* list of PChar *)
 for j:=0 to (10*1024*1024)-1 do list.items [i,j]:=#$FA; (*fill to ensure true alloc *)
end;

(n.b: the above algo is a skeleton, not a snippet, u'll need to make it work in pascal... easy)

when it hangs, watch local variables with debuger, youll get where it hangs
actually around 2GB (not around 4 as expected, ... and never uses swapfile) ===> (I*10*1024*1024)+J < 2GB
« Last Edit: March 10, 2018, 07:10:08 am by sam707 »

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #47 on: March 10, 2018, 10:11:42 am »
You mean a sample like:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.BuTestitClick(Sender: TObject);
  2. const
  3.   coTestsize = 10*1024*1024;
  4.   coKB = 1024;
  5.   coMB = coKB * 1024;
  6. var
  7.   list : TList;
  8.   i,j,aktIDX : Integer;
  9. begin
  10.   list := TList.Create;
  11.   //
  12.   for i := 1 to 10000 do begin
  13.     if i mod 5 = 0 then Memo1.Append('Used size: ' + IntToStr((i*coTestsize) div (coMB))+ ' MB');
  14.     aktIDX := list.Add(getmem(coTestsize)); // List of PChar
  15.     if list.Items[aktIDX] = pointer(0) then begin
  16.       Memo1.Append('No memory at : ' + IntToStr(i));
  17.       exit;
  18.     end;
  19.     for j := 0 to coTestsize-1 do begin
  20.       PChar(list.Items[aktIDX])[j] := #$FA;
  21.     end;
  22.   end;
  23.   list.Free;
  24. end;
  25.  
used componets: aButton and a memo

it stops on my machine  around 1.7 GB with a runtimeError 203 in the IDE

Andreas

Edit:
1.7 GB in win32

90 GB !!! in win64 mode - after 20GB Windows was compressing the heap, but not swapping  O:-) BUT Make a complete breakdown - black screen -  of the system above 90 GB , but other threads are running (my background music stream). I have to shutdown hard - eg. powerreset - to work with the pc again. BTW: I have 24GB physical memory
« Last Edit: March 10, 2018, 10:27:32 am by af0815 »
regards
Andreas

sam707

  • Guest
Re: Laz 1.8.2 No memory left
« Reply #48 on: March 10, 2018, 10:38:33 am »
Welcome to the jungle @af0815  :D

it should hang around 4GB minus size of app and stack, IF FPC was using the paging RAMfile !!!

It worked for me using "cmem unit" which is a wrapper to C libraries' ram managemennt methods, BUT when using FPC TRADITIONAL Mem Manager = it hangs before 2GB (even if you switched the registry key telling the system there is large ram) yep WELCOME

LoOoL

on windows, some C memory managers use to broadcast a WM_COMPACTING message or something among all sibling processes, to ask them to make room in RAM and swap to pagedramfile if possible, I dont know on nix systems... could it be a clue to improve FPC MemManager?!?

Its not very important THO... Who needs 3GB ram allocated? few particular cases... like a world game simulator or 15+ big jpg pictures to process at the same time for example... I can't imagine the CPU presure at morphing ... hehehe
« Last Edit: March 10, 2018, 10:49:05 am by sam707 »

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #49 on: March 10, 2018, 11:09:08 am »
if i ONLY set the Flag for 'App can hande > 2GB address space' with an programm to manipulate the PE-Header of windows, it can adress more space (but with some side effects). So the PE-Flag is important for some out of memory indications under win32.  The flag is not set from fpc in win 32. But can this controlled at compiletime in lazarus ?

And this definitly no Reg-Key, it is a Flag in the PE-Header

And it is sometime VERY IMPORTANT, because fpc have troubles with building a larger lazarusversion at buildtime The needed Ram reach the 2G limit. This is the reason for me to keep this in my mind.


Andreas

Edit: Only the PE-Flag changing the (modified) Testprogramm is able to run up to 3,75 GB.
       The changes was only to define the constants as QWord, to fix the negative numbers shown in the Memo

Edit2: {$setpeflags $20} in the .lpr solves the issue and it is possible to use 3.75 GB
« Last Edit: March 10, 2018, 11:41:21 am by af0815 »
regards
Andreas

sam707

  • Guest
Re: Laz 1.8.2 No memory left
« Reply #50 on: March 10, 2018, 09:06:30 pm »
https://msdn.microsoft.com/en-us/library/windows/desktop/bb870880(v=vs.85).aspx

manipulating these regKeys .. plus using -LARGEADDRESSAWARE option with MS linker ( its your $setpeflags)  8)
« Last Edit: March 10, 2018, 09:09:27 pm by sam707 »

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #51 on: March 11, 2018, 09:07:53 am »
The key is for win32 machines, meaningless on 64 bit machines.

 O:-) The info is only valid for ancient systems  >:D hehehe
This is the reason for to be out of focus.

Andreas
regards
Andreas

sam707

  • Guest
Re: Laz 1.8.2 No memory left
« Reply #52 on: March 11, 2018, 11:55:14 pm »
I am NEVER out of focus EXCEPT when people do not express CLEARLY what they are talking about! In such cases these people are about 1000000% much more out of focus than meh  :D

SO you were talking win32 I was focusing win32 and now you go win64 telling i am out of ....... MY ASS!!!! HAHAHAHAHAHAHAH poor guy c'mon!

@af0815 why don't you start talking DOS and then complain my answers aren't compliant with OS/2 ROFLLL you def made my day I can't stop laughing

find a dictionnary, have a good day looking "focus" definition HAHAHHAAH
thanks for the moment!
« Last Edit: March 12, 2018, 12:02:51 am by sam707 »

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Laz 1.8.2 No memory left
« Reply #53 on: March 12, 2018, 09:06:27 am »
It looks like the issue is not just an isolated bug, triggered by some strange settings. @avra since you're able to reproduce with 100% accuracy, can you please run a few tests with FPC 3.0.2 and Lazarus Trunk?
I have used fpcupdeluxe to install FPC 3.0.2 + TrunkLaz and after adding zillion packages I was able to get "No memory left". Again, using external linker for IDE rebuild (-Xe switch) helped.

Then I have tried FPC 3.0.5 + Laz 1.8.3 (which has "No memory left" problem) to Win7 VM. Since that was originally built with fpcupdeluxe I was able to move it by copying dir to the same location and using the same shortcut. The error was not showing.

Then I have tried FPC 3.0.5 + Laz 1.8.3 (which has "No memory left" problem) to Win10 VM which was never updated over internet. Guess what, the error was not showing.

Then on my real pc I killed WebRoot antivirus and I still had the error. Then I killed in task manager everything that I could kill. Strangely, at first the situation was better and error was not showing, but immediatelly after adding 2-3 new packages the error was returned. Like it give me just a little bigger window for not showing the error.

So, it is either Win10 after some updates, or something else on my pc. I do not know. Anyway, since -Xe helps I will stop further investigation. Now I would not even know what to write in a bug report. I just hope that other people running into this issue will find this thread and apply -Xe to fix it. Otherwise the frustration will be huge.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

snorkel

  • Hero Member
  • *****
  • Posts: 817
Re: Laz 1.8.2 No memory left
« Reply #54 on: March 12, 2018, 03:33:06 pm »
It's likely related to a windows 10 patch for one of those CPU vulnerabilities or some other "issue" LOL
On my windows 7 work laptop I don't get the out of memory error installing packages the corporate patching is usually behind so who knows I might see the same thing on windows 7 in a month or so when they catch up.


***Snorkel***
If I forget, I always use the latest stable 32bit version of Lazarus and FPC. At the time of this signature that is Laz 3.0RC2 and FPC 3.2.2
OS: Windows 10 64 bit

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #55 on: March 13, 2018, 07:11:45 am »
It's likely related to a windows 10 patch for one of those CPU vulnerabilities or some other "issue" LOL
Definitly no, i have fighted again the memory left issue long time before. See the bug report. And i have spent a lot of time to understand, why this issue rises.

IMHO rises the issu because the programms are bigger at linktime and reaches more and more the 2GB Barrier. If the enviroment goes to win64 and cross compile to win 32 the issue will gone.

If you use the testprogramm posted some posts before you can see the impacts and test the issue. I give thanks to sam707, because his idea of a very simple test makes it clearer and show the impact easilier. And he shows the problem on winXP, win7 and other, need an additinal fix in the registry - this was out of my point of view (focus) because i work last time only on win 10 systems (work and home) and never build the IDE on win32 systems anymore.

But with the knowledge it was easy to find out why the building of the IDE on RasPi doesnt wor -> i have to create a realy bigger swap - because fpc was running out of memeory at linktime of Lazarus :-)

Andreas


 
regards
Andreas

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
Re: Laz 1.8.2 No memory left
« Reply #56 on: March 13, 2018, 09:42:15 am »
IMHO rises the issu because the programms are bigger at linktime and reaches more and more the 2GB Barrier. If the enviroment goes to win64 and cross compile to win 32 the issue will gone.
I agree that some limit is reached, but remember that patch you have proposed does not solve "No memory left" error in my case.

Btw, when hit by this error, if -Xe switch is used then linking stage is passed and size of produced optimized IDE executable is just about 50MB.
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

af0815

  • Hero Member
  • *****
  • Posts: 1288
Re: Laz 1.8.2 No memory left
« Reply #57 on: March 13, 2018, 11:56:50 am »
can you send a log when it fail, to my personal mail + info on which system it runs. I'll try to look when/why it was happen.

Andreas

Edit:
I have found a goog diagnostic tool on github and forked. With a little extension it shows the state of the IMAGE_FILE_LARGE_ADDRESS_AWARE Bit (and what type of executable you have 32/64).

https://github.com/afriess/pesp

I have precompiled a version https://github.com/afriess/pesp/releases/tag/V0 here. Normal you can use the pse32.exe (or compile by yourself).

My ppc386.exe have the bit set.
« Last Edit: March 13, 2018, 06:32:22 pm by af0815 »
regards
Andreas

avra

  • Hero Member
  • *****
  • Posts: 2514
    • Additional info
[SOLVED] Re: Laz 1.8.2 No memory left
« Reply #58 on: March 16, 2018, 11:45:37 pm »
Andreas insisted via mail that I should try to recompile FPC with PP.PAS fix, so I did it once more:
Code: Pascal  [Select][+][-]
  1. cd \myfpcsrcdir
  2. make clean
  3. make all
  4. make install
Unfortunately after that Lazarus still had "No memory left" error on recompilation (when not using -Xe switch), although \myfpcsrcdir\compiler\ppc386.exe had IMAGE_FILE_LARGE_ADDRESS_AWARE flag when checked with pse32.exe (as suggested by Andreas). This time I searched for all ppc386 executables in my fpc/laz dir and there were 3. Besides already mentioned one there was one in fpcbootstrap dir and one in \fpc\bin\i386-win32 dir. When last one was replaced with new ppc386.exe which has IMAGE_FILE_LARGE_ADDRESS_AWARE, the error does not show any more. I can repeat the error simply by putting original ppc386.exe back in place. The reason why I originally thought that suggested fix was not working was that my Lazarus installation was actually using different ppc386.exe from the one I thought it used. Silly me.

So, Andreas was right from the start and {$setpeflags $20} fix does help. I am really glad this fix ended up in 3.1.1, but I think it should be back ported for 3.0.6.

To summarize, "No memory left" error during Lazarus recompilation can be solved by applying the mentioned compiler fix, or by using -Xe switch in Lazarus build profile.

Thank you Andreas for your persistence and assistance!
ct2laz - Conversion between Lazarus and CodeTyphon
bithelpers - Bit manipulation for standard types
pasettimino - Siemens S7 PLC lib

armandoboza

  • New Member
  • *
  • Posts: 13
Re: [SOLVED] Re: Laz 1.8.2 No memory left
« Reply #59 on: March 22, 2018, 01:34:29 pm »
Andreas insisted via mail that I should try to recompile FPC with PP.PAS fix, so I did it once more:
Code: Pascal  [Select][+][-]
  1. cd \myfpcsrcdir
  2. make clean
  3. make all
  4. make install
Unfortunately after that Lazarus still had "No memory left" error on recompilation (when not using -Xe switch), although \myfpcsrcdir\compiler\ppc386.exe had IMAGE_FILE_LARGE_ADDRESS_AWARE flag when checked with pse32.exe (as suggested by Andreas). This time I searched for all ppc386 executables in my fpc/laz dir and there were 3. Besides already mentioned one there was one in fpcbootstrap dir and one in \fpc\bin\i386-win32 dir. When last one was replaced with new ppc386.exe which has IMAGE_FILE_LARGE_ADDRESS_AWARE, the error does not show any more. I can repeat the error simply by putting original ppc386.exe back in place. The reason why I originally thought that suggested fix was not working was that my Lazarus installation was actually using different ppc386.exe from the one I thought it used. Silly me.

So, Andreas was right from the start and {$setpeflags $20} fix does help. I am really glad this fix ended up in 3.1.1, but I think it should be back ported for 3.0.6.

To summarize, "No memory left" error during Lazarus recompilation can be solved by applying the mentioned compiler fix, or by using -Xe switch in Lazarus build profile.

Thank you Andreas for your persistence and assistance!

avra, could you attach the changed ppc386.exe file for me to test?

 

TinyPortal © 2005-2018