Lazarus

Programming => General => Topic started by: snorkel on March 05, 2018, 09:49:20 pm

Title: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 09:49:20 pm
I recently installed 1.8.2 and when I went to rebuild the IDE like I usually do after a update I get this error now:

Code: Pascal  [Select][+][-]
  1. Build IDE: Exit code 2, Errors: 1
  2. Fatal: No memory left

I reinstalled over the 1.8 and upgraded the config then went to rebuild and in the past this always worked, never had to reinstall packages now it looks like I will have to start over and completely uninstall and reinstall each packages.

This is going from 1.8 to 1.8.2 so no compiler differences right?


Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 10:06:46 pm
Seem to be getting the out of memory error after I install a few packages.
I was able to install a few and now everyone after that results in the out of memory error.

I went into package manager and removed every package I installed and then started adding each one back and rebuilding the IDE.
Guess I will just blow the whole install away and install from scratch.
Title: Re: Laz 1.8.2 No memory left
Post by: JuhaManninen on March 05, 2018, 10:22:09 pm
I also once got a similar error after my local code changes. A clean build of Lazarus solves it like it solves most compilation problems.
Select "Clean all" in Configure Build Lazarus dialog or delete all .ppu and .o files manually.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 10:53:45 pm
Well, I completely uninstalled and began installing each package and I am getting the fatal out of memory error again after using the online package manager to install indy 10.
 I used the uninstall programs to uninstall and then I completely removed the c:\lazarus dir and then ran the setups again (32bit) ide and cross compiler for 64bit
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 10:57:06 pm
I was able to install about 7 packages now all the ones I try result in the out of memory error when linking lazarus.exe.
My PC has 16gb of ram by the way, and I just installed 1.8.0 last week on this PC (new PC, never had laz on it before 1.8.0) and did not have this out of memory issue when linking lazarus.exe.  Something appears to be messed up.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 11:02:59 pm
Ok, blowing it away again and this time removing the laz directory in the appdata directory for my user.
It's completely gone now no traces anywhere.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 11:13:51 pm
Still getting the out of memory error after I install some packages, this time I installed Indy first via online package manager and it worked no problem, then VirtualTreeview and LCL extensions via online package manager as well as powerpdf and Industrial stuff controls.
Then went to install lazreport and the PDF export for it and bam out of memory.

This was after a complete purge of my system, c:\lazarus completely removed as well as the lazarus folder in local appdata directory for my user account.

It's almost as if it's erroring  out when a certain number of packages are installed.  I am going to remove the laz report and see if anything else will install.

Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 05, 2018, 11:22:25 pm
Ok, looks like LazReport is the culprit i.e. the one in the install/uninstall packages and Zeoslib from the online package manager also causes a out of memory error when linking lazarus.exe.

Something is messed up either with the packages or with 1.8.2.  I guess I am going to go back to 1.8.0.

I guess I will uninstall Virtualtreeview and indy and see if lazreport will then install.   Not for sure clear if it's certain packages or just the number of packages installed.

Update1: I removed Indy 10 and then Lazreport installed, now going to see if Zeos lib will install.
update2: zeos fails, going to remove another package and see if Zeos installs.

Ok, removed lazreport and then zeos installed, it's looking like a package number issue.  weird.  going back to 1.8.0
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 12:09:13 am
installing 1.8 and will report back if there are any issues.

It is super odd that I had package issues with 1.8 as the compiler did not change.  I am not sure if it was the packages or something else that was causing these out of memory issues.
Title: Re: Laz 1.8.2 No memory left
Post by: wp on March 06, 2018, 12:26:42 am
When you reinstalled Laz did you do delete your user settings? It could be that the issue is in there.

Alternatively you can do a secondary installation which leaves the old one alone and writes your settings into a folder of your choice.
Title: Re: Laz 1.8.2 No memory left
Post by: Martin_fr on March 06, 2018, 12:29:09 am
If the error happens while linking, try to disable smart linking.
Just for building the IDE, you can use it in your apps.
Title: Re: Laz 1.8.2 No memory left
Post by: jamie on March 06, 2018, 12:42:38 am
I do know if it is being built with debug info and not using an external file, the EXE will be huge!!!!!

Is It possible this is happening ?
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 02:31:39 am
Well, I went back to 1.8 and it's still doing it :-(

I even under clocked the memory in case it was due to the fact I had it overclocked for a Ryzen CPU.
I just don't get what's going on here, all this stuff installed fine a week and half ago...
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 02:41:29 am
I watched the memory consumption and it's using around 450mb while compiling and linking, there is gobs of free memory.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 02:58:06 am
I wonder if it's some stupid windows 10 update for specter or that other damn thing with the CPUs...
Title: Re: Laz 1.8.2 No memory left
Post by: taazz on March 06, 2018, 03:33:58 am
I wonder if it's some stupid windows 10 update for specter or that other damn thing with the CPUs...
some times (I've seen it mostly in older windows eg xp,vista etc) you got an out of memory when the system run out of resources. You can test against resources if you login in safemode and run build from there. If that succeeds then some recently install application or service consumes to many resources. If not then try installing a clean system on a VM with older windows (7 comes to mind) and see if that helps at all.

Keep in mind I have no experience with windows 10 so I'll not be able to provide any definitive observations.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 04:47:18 am
well, I got all my packages installed and I did it by uninstalling the sparta docked form package.
I then installed that package last and bam Fatal: No memory left
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 04:59:09 am
Well after some more messing around I got even the sparta docked form editor to work i used the Optimized IDE profile and then it linked with no issues.
Maybe some issue with Ryzen CPUs?
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 06, 2018, 10:25:00 am
I can confirm "No memory left" problem. My system is 32-bit Laz 1.8.3 + FPC 3.0.5 on Win10x64. I didn't have that problem with previous 1.8.1+3.0.5. I have Intel 4670 with 16GB. It happens when you install looots of packages with or without OPM. When it happens you have to uninstall at least 10-12 existing packages and it then allows you to install the package you thought was problematic. Unfortunately every "problematic" package passes after this big package uninstall happens first.

I would not know normally, but I am testing ct4laz repo and it needs lots of OPM packages and also many ct4laz packages. So I can reproduce described behavior. You install packages normally, and after a while when it shows, packages can only be compiled but not installed. Only some very small package can install then. After that not even small. I have the same problem with optimized and normal IDE buildings. Clean build does not help. Since this was a fpcupdeluxe installation, I have a clean install that I zipped. I can reproduce the problem when starting from that clean zip archive.

We both have Win10 in common.  >:(

I will test if old 1.8.1+3.0.5 now has this problem, too.

In my first iteration of tests I had only 32-bit Vivaldi browser running. In my second iteration I had only Total Commander running. Therefore I do not think I was out of resources (although I didn't check at the time).
Title: Re: Laz 1.8.2 No memory left
Post by: marcov on March 06, 2018, 10:30:54 am
The strange thing is that it seems to be a linker/compiler error, which should be the same for all these versions. (3.0.4, and 3.0.5 is barely different)
Title: Re: Laz 1.8.2 No memory left
Post by: Fabio Souza on March 06, 2018, 12:45:13 pm
Try using the -CX option in the Lazarus Build.

Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 03:58:14 pm
Avra, are you by chance using any type of AMD CPU?
On my intel work laptop I don't have any issues but it is windows 7.

The odd thing is I didnt have any issues last week with 1.8 when I set up the new Ryzen CPU based desktop.

Using the optimized profile when building the IDE did allow the linker to complete without error and the optimized profile enables the smart link option.  Works with smartlink and does not with it disabled. 

I really wonder if it's some patch they pushed through for windows 10 for some CPU vulnerability.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 04:03:03 pm
I reverted back to the stock 1.8 and it did it there as well, and with the same 1.8.0 setup a week and half ago no issues at all installing the packages.
Title: Re: Laz 1.8.2 No memory left
Post by: Cyrax on March 06, 2018, 04:18:57 pm
Have you tried to remove the overclocking? Also try underclocking the cpu and memory speeds.

Test your memory with these programs : http://www.memtest.org/ https://www.memtest86.com/

If you can, get Linux Live CD and test if your Ryzen have the segmentation fault bug during compiling. See this link for more info : https://www.phoronix.com/scan.php?page=article&item=new-ryzen-fixed&num=1

Also see this link : https://hardforum.com/threads/ryzen-segmentation-fault-test-for-windows.1943854/
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 06, 2018, 05:53:42 pm
Hi,
The cpu is not overclocked at all, running at stock speeds, but I was running the memory at 2933 and I did lower it back down to 2133 and it made no difference.
The DDR4 ram I have is rated at 3200mzh but with current bios it will only run stable at 2933.

Only thing that allowed me to install all my required packages was to build the IDE with the Optimized profile which adds come compiler switches(in that screen shot I posted) which seems to enable the smart linking as well.  I was using the default profile which did not enable smart linking.

I was unaware of any segmentation fault with Ryzen CPUs, but will check that out.
UPDATE:
It's not likely I have a CPU made way back in June, I just got it a couple of weeks ago and these things sell like hot cakes.  Only way to tell for sure is to look at the heat spreader :-( and it's a pain in rear to take off the bolted on heatsink/fan.
Title: Re: Laz 1.8.2 No memory left
Post by: jamie on March 06, 2018, 11:52:49 pm
maybe its not worth anything but, did you go into the Settings:Updates /Security, scroll down
where it says developer mode and make sure you have some lead way there?

Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 07, 2018, 01:19:25 am
The strange thing is that it seems to be a linker/compiler error
I tried internal linker -Xi and error was still there. Then I tried external linker with -Xe and error was gone both in Optimized IDE and Normal IDE profile build modes. I checked several times and I can reproduce on will. It is too late now so I will do further testing tomorrow, but right now it looks to me as this is a work around. Another work around is not just check "Clean all" in already mentioned build modes (which I wrongly asumed was enough), but to use last "Clean up + Build all" profile for building IDE. That solved the problem until next build is needed, but it was too slow to do it every time so I looked further as you could see.

Try using the -CX option in the Lazarus Build.
It didn't help.

Avra, are you by chance using any type of AMD CPU?
No, Intel i5 4670.

Have you tried to remove the overclocking? Also try underclocking the cpu and memory speeds.
Test your memory with these programs : http://www.memtest.org/ https://www.memtest86.com/
No overclocking. BIOS defaults for CPU and MEM. MemTest86+ runs rock solid.

maybe its not worth anything but, did you go into the Settings:Updates /Security, scroll down
where it says developer mode and make sure you have some lead way there?
I am in developer mode, if that is what you wanted me to check.

Title: Re: Laz 1.8.2 No memory left
Post by: jamie on March 07, 2018, 03:00:50 am
Have you tried shutting down the IDE between package install

and also check the Task manager for running apps related to Laz between shutdowns.

Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 07, 2018, 04:48:48 am
Have you tried shutting down the IDE between package install

and also check the Task manager for running apps related to Laz between shutdowns.

Yep, did that and mad no difference.  I checked the task manager the only other app that was launching was multiple copies of make.exe.
As I mentioned before the optimized profile seems to work around the issue whatever it really is.
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 07, 2018, 09:07:52 am
Have you tried shutting down the IDE between package install
Restarting IDE does not solve the problem.

Quote
check the Task manager for running apps related to Laz between shutdowns.
Single Lazarus executable in task manager, no hanging executables.

As I mentioned before the optimized profile seems to work around the issue whatever it really is.
In my case it just gave more room before it strikes again. My guess is that it will strike you too if you install another 10-15 packages. If you need bunch of packages for testing then you can use these: https://bitbucket.org/avra/ct4laz/downloads/ct4laz-preview.zip.

The only permanent work around so far is to use external linker (add -Xe in active build profile). I have so many packages installed that at the moment I remove -Xe and try to rebuild IDE, "No memory left" error shows every time. So yes, so far I can reproduce it whenever needed.
Title: Re: Laz 1.8.2 No memory left
Post by: Cyrax on March 07, 2018, 10:03:15 am
Does this bug occur if you are compiling Lazarus with 64-bit compiler? Note that you need native compiler, crosscompiler may have the same bug.
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 07, 2018, 04:51:39 pm
Maybe an old Bug 0031517 ?!

One change is in the pp.pas aroud line 150. Now i patched the actual trunk 35551. Now fpc compiles the 'Big' Lazarus for me.

===============

{$ifdef win32}
  { 256 MB stack }
  { under windows the stack can't grow }
  {$MAXSTACKSIZE 256000000}
  {$setpeflags $20}
{$else win32}
  {$ifdef win64}
    { 512 MB stack }
    { under windows the stack can't grow }
    {$MAXSTACKSIZE 512000000}
  {$else win64}
    { 1 MB stack }
    {$MINSTACKSIZE 1000000}
  {$endif win64}
{$endif win32}

--------------------------
The PE must be set to compile Lazarus with more than 2GB at linktime.

Andreas
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 07, 2018, 08:09:20 pm
As I mentioned before the optimized profile seems to work around the issue whatever it really is.
In my case it just gave more room before it strikes again. My guess is that it will strike you too if you install another 10-15 packages. If you need bunch of packages for testing then you can use these: https://bitbucket.org/avra/ct4laz/downloads/ct4laz-preview.zip.

The only permanent work around so far is to use external linker (add -Xe in active build profile). I have so many packages installed that at the moment I remove -Xe and try to rebuild IDE, "No memory left" error shows every time. So yes, so far I can reproduce it whenever needed.

That could be as I only have a fairly small amount of packages I install.   I have been installing the same packages every time I update Lazarus ( or install new)  and never saw this problem before.
Even a couple of weeks again when I installed 1.8 on a new PC using same packages this error never popped up.  Is it possible some windows 10 update has messed with the stack size or something?
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 07, 2018, 09:58:47 pm
from https://msdn.microsoft.com/en-us/library/aa384219(VS.85).aspx

Quote
WOW64 enables 32-bit applications to take advantage of the 64-bit kernel. Therefore, 32-bit applications can use a larger number of kernel handles and window handles. However, 32-bit applications may not be able to create as many threads under WOW64 as they can when running natively on x86-based systems because WOW64 allocates an additional 64-bit stack (usually 512 KB) for each thread. In addition, some amount of address space is reserved for WOW64 itself and the data structures it uses. The amount reserved depends on the processor; more is reserved on the Intel Itanium than on the x64 or ARM64 processors.

If the application has the IMAGE_FILE_LARGE_ADDRESS_AWARE flag set in the image header, each 32-bit application receives 4 GB of virtual address space in the WOW64 environment. If the IMAGE_FILE_LARGE_ADDRESS_AWARE flag is not set, each 32-bit application receives 2 GB of virtual address space in the WOW64 environment.

Maybe Lazarus is reaching the 2GB line at linktime of FPC. This is the reason for $setpeflag $20.
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 08, 2018, 06:49:53 pm
Maybe an old Bug 0031517 ?!
No, sorry. Adding {$setpeflags $20} does not help with "No memory left" error. I have even tried to set {$MAXSTACKSIZE 512000000} for win32 but that didn't help either. Thank you for your input.

The only thing that helps is -Xe switch to force external linker. Should I report this as a bug?
Title: Re: Laz 1.8.2 No memory left
Post by: balazsszekely on March 08, 2018, 08:44:32 pm
I installed almost every package from OPM and I did not see the "No memory left" error. Few notes:
1. The whole process took approximately 30 minutes
2. A few packages(3-4) failed to install for various reasons
3. I saw at least two "duplicate unit" warning
4. The lazarus.exe after install is ~0.8GB

As a conclusion the "No memory left" error is not a generic bug. Something specific to your system triggers the issue and I have no idea what is it.

PS: Lazarus Trunk/FPC 3.0.4, Win7(64 bit)
      CPU: i7-3770 3.40 GHz, RAM: 16 GB
Title: Re: Laz 1.8.2 No memory left
Post by: marcov on March 08, 2018, 08:49:22 pm
Maybe some security software enforces some memory limit ?
Title: Re: Laz 1.8.2 No memory left
Post by: balazsszekely on March 08, 2018, 08:59:36 pm
@marcov
Quote
Maybe some security software enforces some memory limit ?
That's a possibility. Personally I use the lightweight Microsoft Security Essentials, both FPC and Lazarus directories are white listed.
Title: Re: Laz 1.8.2 No memory left
Post by: zeljko on March 08, 2018, 09:21:01 pm
Maybe some security software enforces some memory limit ?

Not sure, used 3.0.2 (or even it was 3.0.0) on Win10 virtual machine, when switched to fpc-3.0.4 got "No memory left" when rebuilding lazarus. My bigide contains extra > 25 packages.
Title: Re: Laz 1.8.2 No memory left
Post by: jamie on March 09, 2018, 01:14:40 am
0.8G after a some packages being compiled tells me debug info is getting attached to the file.

That is an insane EXE file size, one that I only see if Debug info is on...

maybe if debug info is to be generated one could use an external file instead...

I have a huge program myself that I compile with gobs of large units and it fits nicely in
 6 Megs of EXE(32 bit mode), loaded with GUI stuff...
 
 BUT!, the debug file is HUGE and very slow to debug...

 I am glad for the new feature lazarus has now to RUN your app without debug! I was setting
up an ICON on the desktop to run it outside the debugger before.

Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 09, 2018, 01:48:30 am
@Jamie
debug infos have nothing to do with mem allocation issues.

1st of all, a debugger lives inside its own process, accessing its own mem mapping

2) if you run an application without a debuger, then the Header of the EXE is interpreted in such a way that the debug info are never loaded by the subsystem into memory

SO merging or not debug infos inside the app file have no impact at all in memory while not using debugers... you can forget your "INSANE" qualifier
Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 09, 2018, 01:58:49 am
keep going! folks, I can confirm there is a bug something Strange at mem allocating

laz 1.8.3

I made a loop (say 1..1000), allocating blocks, and a subloop writing bytes in the allocated blocks of 10MB (to ensure they are truely in allocation, not marked allocated by pointers)

when effective RAM ends (and OFTEN before, see below), it raises a SIGSEGV, it doesn't try to allocate virtual mems nor tell other applications running to swap to pagefile if possible!

AT the MOMENT I use cmem unit and it no more happens ;-)

I was expecting a swap attempt! as it goes on VC++ ... and longer time for allocating function to get mem before effective fail if swapfile is full. Nothing like that never happened...

the test is simple for you to reproduce.

I thought on 32 bits machines limit was 4GB allocation more or less... It happens around 2GB to HANG

I don't know on 64bits machines, because the app I actually develop doesn't need huge mem
Title: Re: Laz 1.8.2 No memory left
Post by: jamie on March 09, 2018, 02:53:37 am
 Say what you want, but considering the linker needs to attach the debug file onto the
end of the EXE file, when large as it can get could cause an issue if the linker is trying to do this
in memory. I've switched using external debug files, it seems to work better for me.

 But on the other hand, it seems  you are the only one having this issue apparently.

 I know that most of us are happy with our results, sorry you can't get what you are looking for.

 Since you claimed it worked before with 1.8.0 and now it does not should tell you something..

 Have a good day.

Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 09, 2018, 04:02:19 am
@Jamie
yeah have a good day AND explore PE or ELF headers you'll see I DO NOT say what I want BUT exactly WHAT SYSTEMS LOADERS DO (linux/windows) while loading LINKERED linked program

meaning => a linker CAN use specia/optional sections (segments some OSes) to write infos, resources and so on (called overlays in old good times, nowadays useless) THAT ARE NEVER LOADED EXCEPT for debugers purposes ON debugers DEMANDS

Have fun! come back when you have study a lil bit more around "insanity" hehehe

for the Strange behavior, do yourself a loop allocation test you'll see if i am the only one HAHAHAHAHAH if you go forward 2GB you'll hang like me myself all alone HAHAHAHAH
Title: Re: Laz 1.8.2 No memory left
Post by: balazsszekely on March 09, 2018, 06:14:29 am
It looks like the issue is not just an isolated bug, triggered by some strange settings. @avra since you're able to reproduce with 100% accuracy, can you please run a few tests with FPC 3.0.2 and Lazarus Trunk?
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 09, 2018, 07:55:28 am
I thought on 32 bits machines limit was 4GB allocation more or less... It happens around 2GB to HANG
BTW: On Windows (win32 parte) you can use more than 2G RAM if you declare it to the system in the PE -Header.

I have 24 GB RAM on my machine, but in win32 this is meaningless. Can you share the code for tests ?

Andreas
Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 09, 2018, 05:33:49 pm
I thought on 32 bits machines limit was 4GB allocation more or less... It happens around 2GB to HANG
BTW: On Windows (win32 parte) you can use more than 2G RAM if you declare it to the system in the PE -Header.

I have 24 GB RAM on my machine, but in win32 this is meaningless. Can you share the code for tests ?

Andreas

YES there is a registry key to tell win32 to manage more than 4GB (large adddresses), I can't remember which key at the moment (Im focused on linux) but I used to switch that one ON

code is simple I described it before... make a stress loop on a button click

here is a kind of algo to run under GDB

for i:=1 to 1000 do begin
 list.add(getmem(10*1024*1024) (* list of PChar *)
 for j:=0 to (10*1024*1024)-1 do list.items [i,j]:=#$FA; (*fill to ensure true alloc *)
end;

(n.b: the above algo is a skeleton, not a snippet, u'll need to make it work in pascal... easy)

when it hangs, watch local variables with debuger, youll get where it hangs
actually around 2GB (not around 4 as expected, ... and never uses swapfile) ===> (I*10*1024*1024)+J < 2GB
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 10, 2018, 10:11:42 am
You mean a sample like:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.BuTestitClick(Sender: TObject);
  2. const
  3.   coTestsize = 10*1024*1024;
  4.   coKB = 1024;
  5.   coMB = coKB * 1024;
  6. var
  7.   list : TList;
  8.   i,j,aktIDX : Integer;
  9. begin
  10.   list := TList.Create;
  11.   //
  12.   for i := 1 to 10000 do begin
  13.     if i mod 5 = 0 then Memo1.Append('Used size: ' + IntToStr((i*coTestsize) div (coMB))+ ' MB');
  14.     aktIDX := list.Add(getmem(coTestsize)); // List of PChar
  15.     if list.Items[aktIDX] = pointer(0) then begin
  16.       Memo1.Append('No memory at : ' + IntToStr(i));
  17.       exit;
  18.     end;
  19.     for j := 0 to coTestsize-1 do begin
  20.       PChar(list.Items[aktIDX])[j] := #$FA;
  21.     end;
  22.   end;
  23.   list.Free;
  24. end;
  25.  
used componets: aButton and a memo

it stops on my machine  around 1.7 GB with a runtimeError 203 in the IDE

Andreas

Edit:
1.7 GB in win32

90 GB !!! in win64 mode - after 20GB Windows was compressing the heap, but not swapping  O:-) BUT Make a complete breakdown - black screen -  of the system above 90 GB , but other threads are running (my background music stream). I have to shutdown hard - eg. powerreset - to work with the pc again. BTW: I have 24GB physical memory
Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 10, 2018, 10:38:33 am
Welcome to the jungle @af0815  :D

it should hang around 4GB minus size of app and stack, IF FPC was using the paging RAMfile !!!

It worked for me using "cmem unit" which is a wrapper to C libraries' ram managemennt methods, BUT when using FPC TRADITIONAL Mem Manager = it hangs before 2GB (even if you switched the registry key telling the system there is large ram) yep WELCOME

LoOoL

on windows, some C memory managers use to broadcast a WM_COMPACTING message or something among all sibling processes, to ask them to make room in RAM and swap to pagedramfile if possible, I dont know on nix systems... could it be a clue to improve FPC MemManager?!?

Its not very important THO... Who needs 3GB ram allocated? few particular cases... like a world game simulator or 15+ big jpg pictures to process at the same time for example... I can't imagine the CPU presure at morphing ... hehehe
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 10, 2018, 11:09:08 am
if i ONLY set the Flag for 'App can hande > 2GB address space' with an programm to manipulate the PE-Header of windows, it can adress more space (but with some side effects). So the PE-Flag is important for some out of memory indications under win32.  The flag is not set from fpc in win 32. But can this controlled at compiletime in lazarus ?

And this definitly no Reg-Key, it is a Flag in the PE-Header

And it is sometime VERY IMPORTANT, because fpc have troubles with building a larger lazarusversion at buildtime The needed Ram reach the 2G limit. This is the reason for me to keep this in my mind.


Andreas

Edit: Only the PE-Flag changing the (modified) Testprogramm is able to run up to 3,75 GB.
       The changes was only to define the constants as QWord, to fix the negative numbers shown in the Memo

Edit2: {$setpeflags $20} in the .lpr solves the issue and it is possible to use 3.75 GB
Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 10, 2018, 09:06:30 pm
https://msdn.microsoft.com/en-us/library/windows/desktop/bb870880(v=vs.85).aspx (https://msdn.microsoft.com/en-us/library/windows/desktop/bb870880(v=vs.85).aspx)

manipulating these regKeys .. plus using -LARGEADDRESSAWARE option with MS linker ( its your $setpeflags)  8)
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 11, 2018, 09:07:53 am
The key is for win32 machines, meaningless on 64 bit machines.

 O:-) The info is only valid for ancient systems  >:D hehehe
This is the reason for to be out of focus.

Andreas
Title: Re: Laz 1.8.2 No memory left
Post by: sam707 on March 11, 2018, 11:55:14 pm
I am NEVER out of focus EXCEPT when people do not express CLEARLY what they are talking about! In such cases these people are about 1000000% much more out of focus than meh  :D

SO you were talking win32 I was focusing win32 and now you go win64 telling i am out of ....... MY ASS!!!! HAHAHAHAHAHAHAH poor guy c'mon!

@af0815 why don't you start talking DOS and then complain my answers aren't compliant with OS/2 ROFLLL you def made my day I can't stop laughing

find a dictionnary, have a good day looking "focus" definition HAHAHHAAH
thanks for the moment!
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 12, 2018, 09:06:27 am
It looks like the issue is not just an isolated bug, triggered by some strange settings. @avra since you're able to reproduce with 100% accuracy, can you please run a few tests with FPC 3.0.2 and Lazarus Trunk?
I have used fpcupdeluxe to install FPC 3.0.2 + TrunkLaz and after adding zillion packages I was able to get "No memory left". Again, using external linker for IDE rebuild (-Xe switch) helped.

Then I have tried FPC 3.0.5 + Laz 1.8.3 (which has "No memory left" problem) to Win7 VM. Since that was originally built with fpcupdeluxe I was able to move it by copying dir to the same location and using the same shortcut. The error was not showing.

Then I have tried FPC 3.0.5 + Laz 1.8.3 (which has "No memory left" problem) to Win10 VM which was never updated over internet. Guess what, the error was not showing.

Then on my real pc I killed WebRoot antivirus and I still had the error. Then I killed in task manager everything that I could kill. Strangely, at first the situation was better and error was not showing, but immediatelly after adding 2-3 new packages the error was returned. Like it give me just a little bigger window for not showing the error.

So, it is either Win10 after some updates, or something else on my pc. I do not know. Anyway, since -Xe helps I will stop further investigation. Now I would not even know what to write in a bug report. I just hope that other people running into this issue will find this thread and apply -Xe to fix it. Otherwise the frustration will be huge.
Title: Re: Laz 1.8.2 No memory left
Post by: snorkel on March 12, 2018, 03:33:06 pm
It's likely related to a windows 10 patch for one of those CPU vulnerabilities or some other "issue" LOL
On my windows 7 work laptop I don't get the out of memory error installing packages the corporate patching is usually behind so who knows I might see the same thing on windows 7 in a month or so when they catch up.


Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 13, 2018, 07:11:45 am
It's likely related to a windows 10 patch for one of those CPU vulnerabilities or some other "issue" LOL
Definitly no, i have fighted again the memory left issue long time before. See the bug report. And i have spent a lot of time to understand, why this issue rises.

IMHO rises the issu because the programms are bigger at linktime and reaches more and more the 2GB Barrier. If the enviroment goes to win64 and cross compile to win 32 the issue will gone.

If you use the testprogramm posted some posts before you can see the impacts and test the issue. I give thanks to sam707, because his idea of a very simple test makes it clearer and show the impact easilier. And he shows the problem on winXP, win7 and other, need an additinal fix in the registry - this was out of my point of view (focus) because i work last time only on win 10 systems (work and home) and never build the IDE on win32 systems anymore.

But with the knowledge it was easy to find out why the building of the IDE on RasPi doesnt wor -> i have to create a realy bigger swap - because fpc was running out of memeory at linktime of Lazarus :-)

Andreas


 
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 13, 2018, 09:42:15 am
IMHO rises the issu because the programms are bigger at linktime and reaches more and more the 2GB Barrier. If the enviroment goes to win64 and cross compile to win 32 the issue will gone.
I agree that some limit is reached, but remember that patch you have proposed does not solve "No memory left" error in my case.

Btw, when hit by this error, if -Xe switch is used then linking stage is passed and size of produced optimized IDE executable is just about 50MB.
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 13, 2018, 11:56:50 am
can you send a log when it fail, to my personal mail + info on which system it runs. I'll try to look when/why it was happen.

Andreas

Edit:
I have found a goog diagnostic tool on github and forked. With a little extension it shows the state of the IMAGE_FILE_LARGE_ADDRESS_AWARE Bit (and what type of executable you have 32/64).

https://github.com/afriess/pesp

I have precompiled a version https://github.com/afriess/pesp/releases/tag/V0 here. Normal you can use the pse32.exe (or compile by yourself).

My ppc386.exe have the bit set.
Title: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: avra on March 16, 2018, 11:45:37 pm
Andreas insisted via mail that I should try to recompile FPC with PP.PAS fix, so I did it once more:
Code: Pascal  [Select][+][-]
  1. cd \myfpcsrcdir
  2. make clean
  3. make all
  4. make install
Unfortunately after that Lazarus still had "No memory left" error on recompilation (when not using -Xe switch), although \myfpcsrcdir\compiler\ppc386.exe had IMAGE_FILE_LARGE_ADDRESS_AWARE flag when checked with pse32.exe (as suggested by Andreas). This time I searched for all ppc386 executables in my fpc/laz dir and there were 3. Besides already mentioned one there was one in fpcbootstrap dir and one in \fpc\bin\i386-win32 dir. When last one was replaced with new ppc386.exe which has IMAGE_FILE_LARGE_ADDRESS_AWARE, the error does not show any more. I can repeat the error simply by putting original ppc386.exe back in place. The reason why I originally thought that suggested fix was not working was that my Lazarus installation was actually using different ppc386.exe from the one I thought it used. Silly me.

So, Andreas was right from the start and {$setpeflags $20} fix does help. I am really glad this fix ended up in 3.1.1, but I think it should be back ported for 3.0.6.

To summarize, "No memory left" error during Lazarus recompilation can be solved by applying the mentioned compiler fix, or by using -Xe switch in Lazarus build profile.

Thank you Andreas for your persistence and assistance!
Title: Re: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: armandoboza on March 22, 2018, 01:34:29 pm
Andreas insisted via mail that I should try to recompile FPC with PP.PAS fix, so I did it once more:
Code: Pascal  [Select][+][-]
  1. cd \myfpcsrcdir
  2. make clean
  3. make all
  4. make install
Unfortunately after that Lazarus still had "No memory left" error on recompilation (when not using -Xe switch), although \myfpcsrcdir\compiler\ppc386.exe had IMAGE_FILE_LARGE_ADDRESS_AWARE flag when checked with pse32.exe (as suggested by Andreas). This time I searched for all ppc386 executables in my fpc/laz dir and there were 3. Besides already mentioned one there was one in fpcbootstrap dir and one in \fpc\bin\i386-win32 dir. When last one was replaced with new ppc386.exe which has IMAGE_FILE_LARGE_ADDRESS_AWARE, the error does not show any more. I can repeat the error simply by putting original ppc386.exe back in place. The reason why I originally thought that suggested fix was not working was that my Lazarus installation was actually using different ppc386.exe from the one I thought it used. Silly me.

So, Andreas was right from the start and {$setpeflags $20} fix does help. I am really glad this fix ended up in 3.1.1, but I think it should be back ported for 3.0.6.

To summarize, "No memory left" error during Lazarus recompilation can be solved by applying the mentioned compiler fix, or by using -Xe switch in Lazarus build profile.

Thank you Andreas for your persistence and assistance!

avra, could you attach the changed ppc386.exe file for me to test?
Title: Re: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: avra on March 23, 2018, 01:35:16 am
avra, could you attach the changed ppc386.exe file for me to test?
Here it is: http://anonfile.com/69IdS2dcb2/ppc386.7z
Title: Re: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: armandoboza on March 23, 2018, 02:39:20 am
avra, could you attach the changed ppc386.exe file for me to test?
Here it is: http://anonfile.com/69IdS2dcb2/ppc386.7z

Thanks Avra, but it did not work.

I am using Lazarus 1.8.2 + fpc 3.0.4, is this ppc386.exe you attached to this version?
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 23, 2018, 06:37:50 am
armandobaza, the best way is to compile the fpc by yourself. The line to insert is descript in the bugreport Bug 0031517 (see my post of the code in this thread one page before)

The line to insert in the code of the pp.pas is now near line 170.  pp.pas is in the <fpc-sources>/compiler

This patch is ONLY for WIN32 (and meaningless for WIN64 and the other plattforms)

Code: Pascal  [Select][+][-]
  1. { Don't care about minstacksize or maxstacksize not beeing supported by current OS }
  2. {$WARN 2077 OFF}
  3. {$WARN 2078 OFF}
  4.  
  5. {$ifdef win32}
  6.   { 256 MB stack }
  7.   { under windows the stack can't grow }
  8.   {$MAXSTACKSIZE 256000000}
  9.   {$setpeflags $20}  // <------------------------------------here, this line is inserted
  10. {$else win32}
  11.   {$ifdef win64}
  12.  

Are you able to build fpc from the sources ? If not, look for fpcupdeluxe. It is a swiss-army-knife for building fpc&lazarus.

Andreas
Title: Re: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: avra on March 23, 2018, 09:34:03 am
Thanks Avra, but it did not work.
I am using Lazarus 1.8.2 + fpc 3.0.4, is this ppc386.exe you attached to this version?
Sorry, I use fixes for both (1.8.3 + 3.0.5). Anyway, you can make it yourself if you patch pp.pas as Andreas mentioned and follow command line make instructions that you have already quoted. Alternativelly, without any patching you can use -Xe switch in your build profile and problem is solved.
Title: Re: Laz 1.8.2 No memory left
Post by: ttomas on March 23, 2018, 10:34:24 am
Is there any set like {$setpeflags $20} for ARM Pi3.
Last year i have not problem to compile lazarus trunk/trunk and newpascal on Pi3 using zram (much faster  then swap). Now trunk/trunk compile/link with fpcupdeluxe, but fail lazarus rebuild when install any component. After fail I can rebuld lazarus with fpcupdeluxe(external linker) and new component install without problem.
Title: Re: [SOLVED] Re: Laz 1.8.2 No memory left
Post by: armandoboza on March 23, 2018, 12:41:53 pm
Thanks Avra, but it did not work.
I am using Lazarus 1.8.2 + fpc 3.0.4, is this ppc386.exe you attached to this version?
Sorry, I use fixes for both (1.8.3 + 3.0.5). Anyway, you can make it yourself if you patch pp.pas as Andreas mentioned and follow command line make instructions that you have already quoted. Alternativelly, without any patching you can use -Xe switch in your build profile and problem is solved.

I downloaded 1.8.3 + 3.0.5 by fpcupdeluxe and it worked perfectly with your ppc386.exe file, thank you very much.

I'll try later with 1.8.2 + 3.0.4 by changing the pp.pas file as Andreas mentioned.

armandobaza, the best way is to compile the fpc by yourself. The line to insert is descript in the bugreport Bug 0031517 (see my post of the code in this thread one page before)

The line to insert in the code of the pp.pas is now near line 170.  pp.pas is in the <fpc-sources>/compiler

This patch is ONLY for WIN32 (and meaningless for WIN64 and the other plattforms)

Code: Pascal  [Select][+][-]
  1. { Don't care about minstacksize or maxstacksize not beeing supported by current OS }
  2. {$WARN 2077 OFF}
  3. {$WARN 2078 OFF}
  4.  
  5. {$ifdef win32}
  6.   { 256 MB stack }
  7.   { under windows the stack can't grow }
  8.   {$MAXSTACKSIZE 256000000}
  9.   {$setpeflags $20}  // <------------------------------------here, this line is inserted
  10. {$else win32}
  11.   {$ifdef win64}
  12.  

Are you able to build fpc from the sources ? If not, look for fpcupdeluxe. It is a swiss-army-knife for building fpc&lazarus.

Andreas


Thanks Andreas, I used the fpcupdeluxe.
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 23, 2018, 12:51:50 pm
I have asked on the mailinglist of freepascal for backporting the fix. Because i think more and more people using the stable win32 Lazarus will come into troubles. And the OPM is definitly a good tool to make bigger Ide's :-)

Andreas
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 23, 2018, 03:20:45 pm
Is there any set like {$setpeflags $20} for ARM Pi3.
You might try to put -Xe switch into your current build profile.
Title: Re: Laz 1.8.2 No memory left
Post by: avra on March 23, 2018, 03:21:34 pm
I have asked on the mailinglist of freepascal for backporting the fix.
Thank you very much!
Title: Re: Laz 1.8.2 No memory left
Post by: armandoboza on March 23, 2018, 07:48:42 pm
I have asked on the mailinglist of freepascal for backporting the fix. Because i think more and more people using the stable win32 Lazarus will come into troubles. And the OPM is definitly a good tool to make bigger Ide's :-)

Andreas

Thank you Andreas.
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 23, 2018, 07:51:02 pm
BTW. Memory problems on PasPi 3 Modell B: I have found the original setting of the swap is to low. I have set the swap to 1GB and now i have no problem with Lazarus on RasPi 3B
Quote
sudo swapoff -a
sudo nano /etc/dphys-swapfile
sudo swapon -a
Normally the defaultvalue is 100, i have rised to 1000, and now the problems with compiling and linking of lazarus on the RasPi3B are gone.

Andreas
Title: Re: Laz 1.8.2 No memory left
Post by: chenyuchih on October 04, 2018, 02:34:16 am
Hello,

Is there any solution for this issue? I encountered the same problem recently. I installed Lazarus 1.8.4(FPC 3.0.4) official release on a new(2018) Windows 10 laptop, which has 8GB RAM. When building a dll files(about 8MB size), the memory usage goes up to 800MB then the error "Fatal: No memory left" showed up. But the same version of lazarus on another 6 year old, 4GB RAM, Windows 7 computer works without pain! I checked the antivirus on new laptop and no interception message/log was found. I read the related topic thread on this forum. It seems this problem happens almost only on Windows 10. Does Windows 10 have special memory limit for single process/program?

BTW, the new laptop is company's computer and MIS locks the superuser permission for safety issue, so I cannot do any advanced system configuration.

Thank you!

Best Regard,
Chen, Yu-Chih
Title: Re: Laz 1.8.2 No memory left
Post by: jamie on October 05, 2018, 12:37:31 am
maybe your virtual memory has been turned off or you have a set limit ?
Title: Re: Laz 1.8.2 No memory left
Post by: wp on October 05, 2018, 12:45:24 am
Are you talking of 32-bit or 64-bit Lazarus? I've not seen this issue on 64-bit, but quite often on 32-bit when too many packages are installed. For 32-bit there is another thread in the forum (https://forum.lazarus.freepascal.org/index.php/topic,42234.msg294599.html#msg294599) in which af0815 gives a good hint how to shift the out-of-memory error away to some degree, but it will eventaully come back, again when too many components are installed (now at a higher limit)
Title: Re: Laz 1.8.2 No memory left
Post by: chenyuchih on October 05, 2018, 05:19:04 am
My Lazarus is 32-bit.

I tried using 64-bit Lazarus, it's still fail to cross compile 32-bit dll. I found that it works fine when build 64-bit dll with 64-bit Lazarus. (Using 32-bit Lazarus to cross compile 64-bit dll fails, too) But I still need to get the 32-bit dll....

I noticed that when building 64-bit dll with 64-bit Lazarus, the memory usage can over 1500MB. Does it make sense? My source code is just 8 MB!

And, why it doesn't happen on win7-64bit-4G RAM machine?

BTW, The virtual memory limit setup of new laptop is 9367 MB and never used out during compilation.

Anyway, thanks for all of your comment and advice.

//EDITED---

After several tests, miab3's suggestion in another thread helps! Just setting lazarus in compatibility mode win 7 solve everything!! It looks Windows 10 puts more constraints on programs.....

//-----------

Best Regards,
Chen, Yu-Chih
Title: Re: Laz 1.8.2 No memory left
Post by: six1 on March 14, 2019, 12:08:12 pm
thanks a lot @chenyuchih  ;)

works like a charme on WIN10 / 64Bit Lazarus 2.0.0 fpc 3.0.4

Lazarus in compatibility Mode to WIN7 and that's it!

best, michel
Title: Re: Laz 1.8.2 No memory left
Post by: af0815 on March 14, 2019, 12:40:47 pm
fpc 3.2 / 32 Bit solves the issue actual. It shifts the menorylimit from 1.8 GB to approx. 3.5 GB. But this is a hard limit - no way out, in 32Bit (with fpc). 
Title: Re: Laz 1.8.2 No memory left
Post by: Thaddy on March 14, 2019, 12:51:49 pm
That's not correct. Since fpc 2.6.4.
Title: Re: Laz 1.8.2 No memory left
Post by: avra on June 15, 2019, 10:56:12 am
Follow up info:

After translation to Lazarus 2.0 fixes and FPC 3.2 fixes (32bit on Win10x64), and after installing tons of components I have again faced "No memory left" error. This time I hit a hard limit. It happened with normal non optimized build mode, and my Lazarus executable came to an extreme size of 790MB. I was forced to do it like that because my favorite optimized settings "-O3 -g- -Xs -CpPENTIUMM -CfSSE3" after a while bring me to incompatible PPU issues with already installed components (clean all does not help). So, now I could not continue installing packages for one or the other reason. Being desperate, I started playing again with optimization switches. Finally, everything works again and I could continue installing packages without issues and bringing Lazarus executable size down to 64MB, so I want to share this in case anyone else bumps into the same problem.

These are optimized IDE options that work:
Code: Pascal  [Select][+][-]
  1. -O2 -g- -Xs -CpPENTIUMM -CfSSE3
TinyPortal © 2005-2018