Heap Overflow with millions of rows in a TDrawGrid

[grawlix]

Hello all.  Here's a question that's likely irrelevant, but still holds some interest for me.  My apologies if it's been covered somewhere, but I cannot find much even with excellent search terms.

Using fp 3.2.2 in a dated 32-bit Lazarus install (which will be sorted out shortly,) I get heap overflow errors when setting a TDrawGrid's RowCount to millions of rows.  The number of rows at which the kill screen appears varies a bit, and would probably be higher outside of debug, but...

Q: What TDrawGrid heap allocation is scaling with RowCount?

- - - - - - - - - - -

Bonus Q1: Also of interest, here, is how to reasonably display millions of rows of information from a useability perspective--a scrollbar isn't it.

Bonus Q2: Do you have any personal recommendations for an open source virtual grid?

Thanks for reading.

- - - - - - - - - - -

Code: Pascal  [Select][+][-]

1.   GridTree.DefaultColWidth := 200;
2.   GridTree.DefaultDrawing  := False;
3.   GridTree.FixedCols       := 0;
4.   GridTree.FixedRows       := 0;
5.   GridTree.ColCount        := 1;
6.   GridTree.RowCount        := 0;
7.   GridTree.Flat            := False;
8.   GridTree.Options         := [goColSizing, goRowSelect, goHeaderPushedLook, goTruncCellHints, goFixedHorzLine, goFixedVertLine, goHorzLine, goVertLine];
9.   GridTree.UseXORFeatures  := False;
10.  

[Martin_fr]

I don't know much about TDrawGrid, but diving in it seems to have a list
  FCols,FRows: TIntegerList;

And that grows as you increase the amount of rows (and it will have a capacity that grows quicker than the actual count).
So you need a block of continuous memory for this, and especially in 32bit that will be limited.
I don't know what it is used for, or if it is a good idea.... 

And not sure what else in it uses memory...

Afaik VTV (LazVTV), that is VirtualTreeView, is especially made to handle huge amount of rows (probably limited at "high(integer)" / not sure if it uses 32 or 64 bit indexes, but I guess 32). That can be used to display a grid.

[grawlix]

That's definitely interesting.  All of those pointers in the TIntegerList end up on the heap?  That's a side issue I guess.

It's probably keeping track of row heights.  Thanks for the tip, I'll see if I can relieve it of that obligation.

Edited to add: It goes all the way down to TCustomGrid and is intrinsic to its function.

[Martin_fr]

end up on the heap?

"heap" can be seen as a term for your general memory. (or those parts of it, that are available for allocation at will)

On most of the big architectures, you got
- the "stack" which often is pre-allocated fixed size, and can be exceeded with deeply nested (often recursive) function calls.
- the heap, which is memory that can be gotten with AllocMem, etc (and the max size depends on your hardware)

The stack has local vars. Except that some local vars (ansistring, dyn array, class-instances) are internally pointers, and all except for the pointer goes into the heap.

[wp]

TDrawGrid by itself is "virtual" in the sense that it does not store any data (like the TStringGrid does). There are some administrative lists which scale with the number of rows, but I think  if this can be avoided - even the VirtualTreeView has administrative data per node.

I played with the attached project having 1 million rows in a TDrawGrid, and I can crank up the row count up to 20 millions and do not see a memory overlfow even when running inside the 32-bit IDE. My system has 32 GB RAM, only 4 can be used in 32-bit, and I don't know how much is reserved for OS and IDE and others.

My demo just draws some number in the cells which is derived from the column and row indices - no additional data is required for this. But in real life, you want to display something which is stored somewhere - and this needs memory, too, probably more than the administrative data used by the grid.

So, it all boils down to memory requirement and memory availability. Why don't you compile your application to 64-bit? I could build and run the test project with 50 millions of rows in the 64-bit IDE, but initialization time for the grid becomes very noticeable then since it seems to grow quadratically with row count (50 millions of rows --> 25 sec, 25 milions of rows --> 5 sec).

From a usability perspective, displaying so many rows is nonsense, nobody will capture this information. Think about more conceivable ways to display your data.

[grawlix]

Thank you very much for your informed replies.  Here's just a bit more background in case you're interested. 

I'd written an efficient-enough routine for analyzing text files of a particular format, by cutting it up into strings in memory.  It really works great for the scale I need.  It even works well at a significantly larger scale, but it blows up sooner than my vanity allows.

A memory stream and a skeletal tree index has gotten me where I want with performance vs memory consumption.  It was my own miscomprehension of TDrawGrid that left me wondering why it scales with RowCount.

It's not really a problem, though, as this grid is in a test skid for data-handling routines.  That data does not need to be displayed at all.

Thanks again.

Edited to add: I've gotten myself upgraded to 64-bit and TDrawGrid has memory for <nasal>billions and billions of</nasal> rows.  Thanks x3

[grawlix]

I've thought of something else.  If you're game:

If I declare a string member in a class, but never explicitly initialize it, will it have some small memory overhead for each object instantiated?

[Thaddy]

If I declare a string member in a class, but never explicitly initialize it, will it have some small memory overhead for each object instantiated?

Yes, within a class, it will. depending on platform between 16 and 24 bytes per string declaration.
But as a class member it will be initialized to length zero.
For length, location and refcount and even if length is 0. But even a dumb PChar will allocate at least the pointer space.

[Martin_fr]

I've thought of something else.  If you're game:

If I declare a string member in a class, but never explicitly initialize it, will it have some small memory overhead for each object instantiated?

Talking Ansistring:

If you never initialize it => that is if you leave it empty forever => then no overhead.

It will take the size of a pointer (4 or 8 bytes) inside the class.  Nothing else.
That is, there won't actually be an empty string, there will just be a (hidden/internal) nil pointer.

If the string has a value then you get some overhead. That overhead stores the length (as strings can contain #0), and the refcount, and the codepage.
FPC also adds a #0 at the end (in case you typecast to pchar)


Additionally, the mem-manager may add some overhead. (depends on which one you use). The mem manager, may round up the required memory (e.g. to the next multiply of 16), it may also store the size that it allocated.