intercepting exceptions

[Handoko]

I tested the code. It caused massive slow down and crashed when the counter on the caption showed 2 thousands something. It seems my Linux machine with 4 GB RAM wasn't able to handle the huge request of memory.

[Thaddy]

I didn't even begin to test that code....
I can give you a code review if you like?
Don't be offended... if you seriously want me to do that.

[Rik]

I tested the code. It caused massive slow down and crashed when the counter on the caption showed 2 thousands something. It seems my Linux machine with 4 GB RAM wasn't able to handle the huge request of memory.

Assuming that each pixel takes 16 bits (2 bytes) the size a 500 x 500 Bitmap is +/- 500kB (500x500x2). So filling 6000 of these Bitmaps would indeed consume about 3GB, what might crash the program.
But the aim the program was to provoke this, and find a way to know (calculate) how many 500x500 Biptmaps I can fill without crashing.

I adapted the program (see below) in order to check how many MB RAM are used and how many MB remain.
On a Windows 7 PC with 8MB RAM the program stops after filling 3324 Bitmaps. This took 2632MB (so about 800kB per Bitmap) and 2099 MB remained free.
I would expect this last value to be (close to) zero, but maybe I am checking the wrong variable?

Rik

Code: Pascal  [Select][+][-]

1. unit MainUnit;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes, SysUtils, FileUtil, Forms, Controls, Graphics, Dialogs;
9.  
10. type
11.  
12.   { TMainForm }
13.  
14.   TMainForm = class(TForm)
15.     procedure FormActivate(Sender: TObject);
16.   private
17.     { private declarations }
18.   public
19.     { public declarations }
20.   end;
21.  
22. var
23.   MainForm: TMainForm;
24.  
25. implementation
26.  
27. {$R *.lfm}
28.  
29. { TMainForm }
30.  
31. type
32.   tMemoryStatusEx = record
33.    dwLength: DWORD;
34.    dwMemoryLoad: DWORD;
35.    ullTotalPhys: uint64;
36.    ullAvailPhys: uint64;
37.    ullTotalPageFile: uint64;
38.    ullAvailPageFile: uint64;
39.    ullTotalVirtual: uint64;
40.    ullAvailVirtual: uint64;
41.    ullAvailExtendedVirtual: uint64;
42.   end;
43.  
44. var
45.   MyBitmap: array of tBitmap;
46.  
47.   function GlobalMemoryStatusEx(var Buffer: tMemoryStatusEx): Boolean; stdcall; external 'kernel32' name 'GlobalMemoryStatusEx';
48.  
49.   function GetFreeMem: uint64;
50.   var
51.     MS_Ex: tMemoryStatusEx;
52.   begin
53.     FillChar(MS_Ex,SizeOf(tMemoryStatusEx),#0);
54.     MS_Ex.dwLength:= SizeOf(tMemoryStatusEx);
55.     GlobalMemoryStatusEx(MS_Ex);
56.     Result:= MS_Ex.ullAvailPhys;
57.   end;
58.  
59. procedure TMainForm.FormActivate(Sender: TObject);
60. var
61.   i: Integer;
62.   fm1,fm2: uint64;
63. begin
64.   SetLength(MyBitmap,6000); // create enough Bitmaps to provoke the exception
65.   FillChar(MyBitmap[0],length(MyBitmap)*sizeof(MyBitmap[0]),#0);
66.   for i:= 0 to 5999 do
67.   begin
68.     fm2:= GetFreeMem;
69.     Caption:= 'Creating '+IntToStr(i);
70.     MyBitmap[i]:= tBitmap.Create;
71.   end;
72.   fm1:= GetFreeMem;
73.   for i:= 0 to 5999 do
74.   begin
75.     fm2:= GetFreeMem;
76.     Caption:= 'Filling '+IntToStr(i)+' ('+IntToStr((fm1-fm2) div 1000000)+'MB used, '+IntToStr(fm2 div 1000000)+'MB remaining)';
77.     with MyBitmap[i] do
78.     begin
79.       Width:= 500;
80.       Height:= 500;
81.       Canvas.Brush.Style:= bsSolid;
82.       Canvas.Brush.Color:= clGray;
83.       Canvas.Pen.Color:= clBlue;
84.       try
85.         Canvas.FillRect(0,0,500,500);
86.       except
87.         On E :Exception do
88.         begin
89.           ShowMessage(E.Message);
90.         end;
91.       end;
92.     end;
93.   end;
94. end;
95.  
96. end.

[Thaddy]

If it is a 64 bit windows you are somewhere using the wrong integer type? (amongst many things).
The code you showed doesn't show the actual bitmap maximum size, for one.
If I do:

Code: Pascal  [Select][+][-]

1.   SetLength(MyBitmap,6000); // create enough Bitmaps to provoke the exception
2.   FillChar(MyBitmap[0],length(MyBitmap)*sizeof(MyBitmap[0]),#0);
3.  

What happens?
SizeOf(Tbitmap) is SizeOf(NativePointer).........
Well you are allocating space for 6000 pointers to bitmap..
Now calculate times Tbitmap.InstanceSize times actual size of the bitmap.
I need 18+ Gb of ram for that given a conservative 800Kb bitmap size. 18 is not 8.

[Rik]

I didn't even begin to test that code....
I can give you a code review if you like?
Don't be offended... if you seriously want me to do that.

To get things right: the aim of the program is to provoke the error / exception.
Next step is to find a way to create and fill as many of this Bitmaps as possible, without crashing.
Either by intercepting the exception and then stop filling the Bitmaps or to be able to calculate ahead how many Bitmaps can be filled.

I am not offended at all, all help is welcome.

Rik

[Thaddy]

Our posts crossed. See my edited reply.

[Rik]

The "Canvas does not allow drawing" is probably because you are running out of Windows resources. Windows can only create so many handles (or maybe the canvas needs GDI resources, which are also limited)

The same is true for brushes and pens, you only can create a limited amount. Limited by Windows.

Just a thought, since your real app will also be out of resources, it is possible that then the LCL will encounter the same error for some other operation (repainting the form, or whatever). It is possible that the LCL itself has a bug, not handling this correct, and leading to the sigsegv.

I think you are right about the number of recources / handles:
Filling 500x500 Bitmaps stops after 3324 Bitmaps, taking 2632MB and 2099 MB remaining free.
Filling 1000x1000 Bitmaps stops also after 3324 Bitmaps, now taking 4283MB and remaining 430MB free.

The obvious next question: Is there a way to get the number of available resources / handles?

Rik

[Martin_fr]

But the aim the program was to provoke this, and find a way to know (calculate) how many 500x500 Biptmaps I can fill without crashing.

I adapted the program (see below) in order to check how many MB RAM are used and how many MB remain.
On a Windows 7 PC with 8MB RAM the program stops after filling 3324 Bitmaps. This took 2632MB (so about 800kB per Bitmap) and 2099 MB remained free.
I would expect this last value to be (close to) zero, but maybe I am checking the wrong variable?

I tested on a 64 bit system with 32 GB Ram. Still got the error. It is not memory (try with 10x10 bitmaps). It is system/OS resources. (Handles, GDI Objects,...)

The problem: If you do not free them after the exception, then other components on your form may fail too.

E.g., if I put your loop into FormCreate, and break out after the exception, then the form will be invisible. It will be there, but not painted.

[Rik]

Our posts crossed. See my edited reply.

If I quadruple the size of the Bitmaps (from 500x500 to 1000x1000) the program still stops after filling 3324 Biptmaps (but now taking 4238MB instead of 2632MB).
It seems to be an issue with the number of available Windows resources rather than a memory issue, as Martin_fr suggested (see http://forum.lazarus.freepascal.org/index.php/topic,42251.msg295066.html#msg295066).

Rik

[Martin_fr]

Why do you need all of them with system resources?

Temporary allocate them and release after use. Store the raw bitmap data in memory without a handle.
Not done that myself, but maybe someone else can elaborate on the how to.

[Martin_fr]

If we assume (no idea if correct) that there are 3 gdi objects per canvas (pen, brush, canvas??), and add some used by the form etc..., then this would match with https://stackoverflow.com/questions/9723470/whats-the-upper-limit-on-gdi-objects-for-one-process-in-windows-7

That said, I do NOT recommend to increase the limit.

[Rik]

Store the raw bitmap data in memory without a handle.
Not done that myself, but maybe someone else can elaborate on the how to.

Good idea. But I have to do it in a way that porting the bitmap data to an Image can be done fast.

Rik

[lucamar]

TBitmap and his brothers and sisters are relatively heavy consumers of system ressources: pen, brush, font, canvas, etc. And for graphics-heavy applications the answer to "how many ressources do I have" is always "not enough" 

Martin's answer of "temporary allocate them and release after use; store the raw bitmap data in memory without a handle" is, indeed, the usual way this kind of things are done---among other things because it's not only your application which will suffer, but also the others executing at the moment and the system's general stability.

There are many ways of storing raw bitmap data in memory; a simple one is to allocate an array of word for each bitmap and blitting from/to a single one TBitmap whenever you need it. Simple and, if done right, not very inefficient. Your constraint then would really be memory size to contain the arrays collection.

[Thaddy]

The maximum number of handles on windows for any given process is 65535 according to https://docs.microsoft.com/en-us/windows/desktop/SysInfo/gdi-objects, but actually 10.000 (see msdn and this blog: https://blogs.msdn.microsoft.com/oldnewthing/20070718-00/?p=25963/ ). Also on windows 64. So that may also be an issue indeed.

So that means, with a 6000 collection of bitmaps, you can paint, brush and maybe half pen, but no font.... (Of course exagirated, but that's a possible cause.)
Actually this seems - not quite sure, but a little test led me to believe it - a (kind of serious) bug in Lazarus, because handles should only be used when needed. Lazarus takes all of them, in use or not. It should just have the bitmap and release any handles when not needed. Delphi behaves like that. And KOL too, even on Lazarus.

[440bx]

Rik,

As others above have already pointed out, there is a limit (which can be increased) of 10,000 GDI handles and you've hit it.  (btw, increasing the limit is _not_ a good idea.)

In addition to that, as you have already noticed, your program is a memory hog.  I attached a small screenshot showing that your program did hit the limit and that it was using 2.5GB of memory when it did.

You mentioned that the reason you were doing this is because your computer was too slow to render the images in real time.  Using 10,000 handles and several gigabytes of memory is not going to make any computer faster, on the contrary, that guarantees that most machines would be overwhelmed if it worked.

One possibility you may want to look into is creating a Windows Metafile which your program can play.  For the record, I have no idea if there is a component in Lazarus to play a WMF but, if there is, that might provide a reasonable, as well as, reasonably fast and simple solution.

In other words, your program needs to manage resources a little more efficiently.

Hopefully, someone who knows what Lazarus can do, can give you an example on how to play a Windows metafile which you could use for what you need to do.

HTH.