No decent tutorial for the Tdrawgrid - need to output millions of data

[teochris]

Hello, I need to output nearly 25 million rows and of 6 columns data from array cell calculations in a Tstringgrid - but it only manages to output only 4 million rows and then the form hangs, I believe because of Ram leaking.

Is there a way to output such amount of data in a Tdrawgrid and everything would work properly? I have not found a decent tutorial about how can i output data in a Tdrawgrid can someone please point to me such an example project or send me a sample code?

If i export data first on a txt file then maybe the Tdrawgrid or even better the Tstringgrid could output them properly without any problem - an example project would be really helpful.

Thanks

[balazsszekely]

You ask this question before: http://forum.lazarus.freepascal.org/index.php/topic,32646.msg210593.html#msg210593 . Do you have issues with VST?

[teochris]

I have not managed to make virtualtree output my data properly so now I am trying to use Tdrawgrid or Tstringgrid to output them properly

[wp]

You'll need a 64-bit machine probably. And a good time reading through 150,000,000 cells for the rest of your life 

Anyway, here is the code for a TDrawGrid. I had to go down to 2.5 millions rows since I work with Laz-32bit.

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. const
3.   n = 6;
4.   m = 2500000;
5. var
6.   i,j: Integer;
7. begin
8.   SetLength(data, n, m);
9.   for i:=0 to n-1 do
10.     for j:=0 to m-1 do
11.       data[i,j] := 10*i+j;
12.   DrawGrid1.DefaultColWidth := 80;
13.   DrawGrid1.RowCount := m+1;
14.   DrawGrid1.ColCount := n+1;
15.   DrawGrid1.Invalidate;
16. end;
17.  
18. procedure TForm1.DrawGrid1DrawCell(Sender: TObject; aCol, aRow: Integer;
19.   aRect: TRect; aState: TGridDrawState);
20. var
21.   s: String;
22.   grid: TDrawGrid;
23. begin
24.   if Length(data) = 0 then
25.     exit;
26.   if aCol = 0 then
27.     s := Format('Row %d', [aRow])
28.   else
29.   if aRow = 0 then
30.     s := Format('Col %d', [aCol])
31.   else
32.     s := FloatToStr(data[aCol-1, aRow-1]);
33.   grid := TDrawGrid(Sender);
34.   grid.Canvas.TextOut(ARect.Left + constCellpadding, ARect.Top + ConstCellPadding, s);
35. end;

[teochris]

You'll need a 64-bit machine probably. And a good time reading through 150,000,000 cells for the rest of your life 

Identifier not found "data"

what type is it?

[wp]

Sorry, should have posted complete project...

Code: Pascal  [Select][+][-]

1. type
2.   TDataArray = array of array of Double;

[teochris]

Sorry, should have posted complete project...

Code: Pascal  [Select][+][-]

1. type
2.   TDataArray = array of array of Double;

I do not only feel newbie but I know I am!

After I add this declaration in my form still I am getting the same error Identifier not found "data"

[wp]

Then declare the variable:

Code: Pascal  [Select][+][-]

1. var
2.   Data: TDataArray;

But here's the full project.

[balazsszekely]

If you choose VST...

[teochris]

Then declare the variable:

Code: Pascal  [Select][+][-]

1. var
2.   Data: TDataArray;

But here's the full project.

Thanks for your time and help!

When I try to output 2500000 rows it hangs and then raises exception - I have win 10 - 64bit

Do you know if I first write all calculations to a txt file and then write this file data to a Tstringgrid or Tdrawgrid If I would have better chances?

[mas steindorff]

I ran into this TString limit as well on my 32 bit OS. my limit at the time was around 2G. What I did was to not use the .saveToFile() and coded my own version.  I move a .string[] into a local var and save it one line at a time.  it even run faster that the .SavetoFile() probably due to the  simpler code but it also has worked on 10G data sets. 

[teochris]

I ran into this TString limit as well on my 32 bit OS. my limit at the time was around 2G. What I did was to not use the .saveToFile() and coded my own version.  I move a .string[] into a local var and save it one line at a time.  it even run faster that the .SavetoFile() probably due to the  simpler code but it also has worked on 10G data sets.

Can you please post here your solution or even better post the certain part of your project?

[wp]

Win 10 64bit is one part of the story. The other one is Lazarus, it must be 64 bit as well.

I only have an old Laz 1.44 installation with 64 bit, because I normally work with 32 bit. But with this 64-bit system my demo runs without a crash even with 25 million rows. It's no fun working with it, though, because scrolling is very slow. Part of it is certainly due to the fact that these 150 millions of double values bring reach the limits of my system memory. Buf even an empty drawgrid with that many cells has already halfe the memory consumptions and does not scroll smoothly either.

I don't know about the performance of VirtualTreeview, but I would not expect it to be substantially faster. We are at the limits of today's PC's performance here.

And no: writing to a file and loading into any visual control will not be faster.

But to be honest: why do you want to show that many data? Nobody will look at them, not even you. You must write the results of these calculations to file which you evaluate with some dedicated analysis to extract the information that you need. Since I do not know what you want to achieve I cannot give you more specific hints.

[teochris]

Win 10 64bit is one part of the story. The other one is Lazarus, it must be 64 bit as well.

I only have an old Laz 1.44 installation with 64 bit, because I normally work with 32 bit. But with this 64-bit system my demo runs without a crash even with 25 million rows. It's no fun working with it, though, because scrolling is very slow. Part of it is certainly due to the fact that these 150 millions of double values bring reach the limits of my system memory. Buf even an empty drawgrid with that many cells has already halfe the memory consumptions and does not scroll smoothly either.

I don't know about the performance of VirtualTreeview, but I would not expect it to be substantially faster. We are at the limits of today's PC's performance here.

And no: writing to a file and loading into any visual control will not be faster.

But to be honest: why do you want to show that many data? Nobody will look at them, not even you. You must write the results of these calculations to file which you evaluate with some dedicated analysis to extract the information that you need. Since I do not know what you want to achieve I cannot give you more specific hints.

These data will not have any real use, I just want them to be shown to users because this will add a feature that similar programs like mine do not have at all and users of my app requested it from me

[tk]

I would implement some paging mechanism here to display reasonable amount of data at a time in a grid.
And I would not even try to load all those data into memory, rather I would store them in a large file on disk and then read the given chunk into memory.
TFileStream should be able to do it, or see here: http://stackoverflow.com/questions/4615286/what-is-the-fastest-way-for-reading-huge-files-in-delphi