[Solved] Out of Memory

[bobonwhidbey]

I get this message, and of course the program terminates, when I try to create a rather large TStringlist. I think it's about 100 MB.  Is there a limitation that I can change to eliminate this problem, or perhaps move the limit up if there is one?

[jamie]

in 32 bit mode you have 2G to play with...

If you have a lot of items created in your app you will get this error.

If you are generating a 64bit app then you'll have plenty of space...

 You need to rethink your game plain on conserving memory in your app.

 Its possible you do have 100 MB but, your memory gets a little fragmented and things can have issues trying
to allocate a single chunk that large..

 you could try allocate that at the start of the program before you create all the other little items and see what happens.

[Martin_fr]

How are the 100 MB distributed? How many strings.
And how are the added?

If you add strings one by one, the stringlist will grow its capacity to much more than you may need.
Eg if you have space for 100 strings, and you add the 101st, then the capacity goes to 400, next 1600 and so on.

A stringlist on 32 bit can have 500 million entries (each 4 byte pointer => 2GB). So that leaves no memory for the strings itself.

As soon as memory fragments (which it will do) that number drops dramatically.

If you know the amount of strings you will have in the end, then do

Code: Pascal  [Select][+][-]

1. mystringlist.capacity := x;

to set it.

[bobonwhidbey]

Basically it's created in a loop like this

for i := 0 to 99000 do begin
  Page = MakePage; // each page is about 200 bytes
  SL.Add(Page);
end;

[Martin_fr]

And?

What happens if you do
SL.capacity := 99001;
before the loop?

[jamie]

looks like a bad approach.

 I would allocate a chunk of memory at once to cover all of it and then write a little class to manage that chunk.

 Also this would be a good time to use a virtual memory chunk handler..

[bobonwhidbey]

I tried

Code: Pascal  [Select][+][-]

1. SL.capacity := TotalRows +1000;

and it still bombed, about the same place.


I know nothing about memory handlers

[Mike.Cornflake]

My wild guess is that it's unlikely to be the List that's causing your problems.   

What's going on inside MakePage?  Is it small enough to post?

[Thaddy]

Indeed, the stringlist is not likely an issue on its own, unless there is already a lot of fragmentation before hand caused by something elsewhere in the code.
So we need a little more info like what MakePage does and how.

[440bx]

Basically it's created in a loop like this

for i := 0 to 99000 do begin
  Page = MakePage; // each page is about 200 bytes
  SL.Add(Page);
end;

Rounding things up and presuming your estimate of 200 bytes is correct, that's 200 * 100,000 = 20,000,000.  That's not even 20MB, barely a drop in the bucket. 

As Thaddy above mentioned, it's likely that the problem is elsewhere in the code but shows up when executing that loop.

Here is something you can do to get some idea of what is happening, when you first start your program put a call to SuspendThread (assuming single threaded) to stop your program from executing and, put another call to SuspendThread just before the loop and another just after it, start process hacker (free download, nice utility) and use it to look at your process memory space every time the main thread suspends itself, particularly the heaps but, look at the entire address space  (copy paste the process memory map, that will help when comparing), after that you can tell process hacker to resume the thread which will stop just before the loop, look at process memory map again, odds are good, it will already be messed up, for good measure look at the process memory map after executing the loop.

To find out where the problem is happening sprinkle calls to suspendthread in your program (it would be good for you to use a compiler directive to control the inclusion or exclusion of those calls).  With judicious placement of calls to SuspendThread, it should only take you a few minutes to figure out the origin of the memory problem.

Once you know where the problem is being created then, a solution is possible.

HTH.

[WooBean]

Basically it's created in a loop like this

for i := 0 to 99000 do begin
  Page = MakePage; // each page is about 200 bytes
  SL.Add(Page);
end;

THIS IN LOOP ABOVE COMPARES ONLY variable (?property) Page to result of function MakePage.
Result of comparison is not used but ... unitialized Page (string actually) is then used for TStringList.Add.

What would happen then? - Almost everything - try to use any random combination of bytes as a valid string structure.

WooBean

[Mike.Cornflake]

THIS IN LOOP ABOVE COMPARES ONLY variable (?property) Page to result of function MakePage.

Well spotted.

Code: [Select]

  Page = MakePage; // each page is about 200 bytes
Should be

Code: [Select]

  Page := MakePage; // each page is about 200 bytesNote the :

[lucamar]

THIS IN LOOP ABOVE COMPARES ONLY variable (?property) Page to result of function MakePage.
Result of comparison is not used but ... unitialized Page (string actually) is then used for TStringList.Add.

I'd guess that is just a copy/paste typing artifact; a line like
   Page = MakePage; // each page is about 200 bytes
will produce a compilation error ("Illegal expression").


edited: typo

[bobonwhidbey]

Yes that was a typo. The app has worked fine with smaller files.  My example was far too much a simplification of what's going on. The actual code is much, much larger. When running from an EXE, like a user would, the error message is "Out of Memory".  Running from the IDE the message is something like Heap Overflow, or Out of Heap.  This is a much closer representation of what's going on.

Code: Pascal  [Select][+][-]

1. for i := 0 to FileSize do
2.   for k := 0 to 57 do
3.     SL.Add(aPage);  // average page size is 16 bytes
4.  

So at 100,000 file records this generates APPROXIMATELY 5.7 MB records x 16 bytes or 91 MB.

I use heaptrc from time to time to insure there are no memory leaks.

[bobonwhidbey]

I realized that the stringlist record count is about 60 times the Total Rows in the SQL database file. So I changed

Code: Pascal  [Select][+][-]

1. SL.Capacity := TotalRows*60;

Next I opened up Task Manager (Win10) and watched the total memory used as I ran the program. Memory increased gradually from 159MB to 1.1GB when it hit 100% on my Progress Bar. Changing capacity to an appropriate number allowed me to complete the main loop. I had finished creating the large stringlist. 

However, I RUN OUT OF MEMORY when I tried to execute SL.SaveToFile after the main loop.

I need another approach to create the output file. Any suggestions?