Recent

Author Topic: Issue with greater input in TLineSeries.  (Read 713 times)

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Issue with greater input in TLineSeries.
« on: October 25, 2022, 02:07:26 pm »
I am trying to display data from an oscilloscope (4 channels, 10 000 000 samples each).
But I cannot display the data from more than two channels - I get an out of memory error, which is sometimes shown ever after displaying the data for even one channel.
Is there a way to solve the issue?

Here is my code snippet (also attached), Button1 and Button2 contain two slightly different attempts.

Code: Pascal  [Select][+][-]
  1.  
  2. unit Unit1;
  3.  
  4.  
  5. {$mode objfpc}{$H+}
  6.  
  7.  
  8. interface
  9.  
  10.  
  11. uses
  12.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, TAGraph, TASeries;
  13.  
  14.  
  15. type
  16.  
  17.  
  18.   tLSarr = array of TLineSeries;
  19.   Double2D = array of array of double;
  20.  
  21.  
  22.   { TForm1 }
  23.  
  24.  
  25.   TForm1 = class(TForm)
  26.     Button1: TButton;
  27.     Button2: TButton;
  28.     Chart1: TChart;
  29.     procedure Button1Click(Sender: TObject);
  30.     procedure Button2Click(Sender: TObject);
  31.     procedure FormCreate(Sender: TObject);
  32.   private
  33.  
  34.  
  35.   public
  36.  
  37.  
  38.   end;
  39.  
  40.  
  41. var
  42.   Form1: TForm1;
  43.   LS0: TLineSeries;
  44.   LS1: TLineSeries;
  45.   LS2: TLineSeries;
  46.   LS3: TLineSeries;
  47.  
  48.  
  49. implementation
  50.  
  51.  
  52. {$R *.lfm}
  53.  
  54.  
  55. { TForm1 }
  56.  
  57.  
  58. procedure TForm1.FormCreate(Sender: TObject);
  59. begin
  60.   ls0:=TLineSeries.create(Chart1);
  61.   ls1:=TLineSeries.create(Chart1);
  62.   ls2:=TLineSeries.Create(Chart1);
  63.   ls3:=TLineSeries.create(Chart1);
  64.  
  65.  
  66.   Chart1.AddSeries(ls0);
  67.   Chart1.AddSeries(ls1);
  68.   Chart1.AddSeries(ls2);
  69.   Chart1.AddSeries(ls3);
  70.  
  71.  
  72.   ls0.SeriesColor := clRed;
  73.   ls1.SeriesColor := clBlue;
  74.   ls2.SeriesColor := clGreen;
  75.   ls3.SeriesColor := clYellow;
  76. end;
  77.  
  78.  
  79. procedure TForm1.Button1Click(Sender: TObject);
  80. var
  81.   ChanCount:integer=4;
  82.   ChanIndex:integer=0;
  83.   row:integer=0;
  84.   Data: Double2D;
  85. begin
  86.   ls0.Clear;
  87.   ls1.Clear;
  88.   ls2.Clear;
  89.   ls3.Clear;
  90.  
  91.  
  92.   SetLength(Data,ChanCount);
  93.   for ChanIndex:= 0 to ChanCount-1 do
  94.     SetLength(Data[ChanIndex],10000000);
  95.  
  96.  
  97.   for ChanIndex:= 0 to ChanCount-1 do
  98.     for row := 0 to 10000000-1 do
  99.       Data[ChanIndex,row]:=random+ChanIndex*5;
  100.  
  101.  
  102.   ls0.AddArray(Data[0]);
  103.   ShowMessage('0');
  104.  
  105.  
  106.   if ChanCount > 1 then
  107.     ls1.AddArray(Data[1]);
  108.   ShowMessage('1');
  109.  
  110.  
  111.   if ChanCount > 2 then
  112.     ls2.AddArray(Data[2]);
  113.   ShowMessage('2');
  114.  
  115.  
  116.   if ChanCount > 3 then
  117.     ls3.AddArray(Data[3]);
  118.   ShowMessage('3');
  119. end;
  120.  
  121.  
  122. procedure TForm1.Button2Click(Sender: TObject);
  123. var
  124.   row:integer=0;
  125. begin
  126.   ls0.Clear;
  127.   ls1.Clear;
  128.   ls2.Clear;
  129.   ls3.Clear;
  130.  
  131.  
  132.   for row:=0 to 10000000-1 do
  133.     ls0.AddXY(row/1000,random);
  134.   ShowMessage('0');
  135.    for row:=0 to 10000000-1 do
  136.      ls1.AddXY(row/1000,random+5);
  137.  ShowMessage('1');
  138.   for row:=0 to 10000000-1 do
  139.     ls2.AddXY(row/1000,random+10);
  140.   ShowMessage('2');
  141.   for row:=0 to 10000000-1 do
  142.     ls3.AddXY(row/1000,random+15);
  143.   ShowMessage('3');
  144. end;
  145. end.
  146.  

If this info can be useful - I have tried how LabView (32 bit) behaves with the same size of data  - it shows 5 channels, before showing an out of memory error. It also draws the graphics much-much faster.
« Last Edit: November 16, 2022, 10:00:14 am by CM630 »
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: Issue with greater input in TLineSeries.
« Reply #1 on: October 25, 2022, 04:31:47 pm »
Seems that you are on a 32-bit system, or build the test application for 32-bit.

At first you add a matrix with 4 columns having each 10 million rows of double. A double is 8 bytes - so this occupies 8x4x10 millions = 320 MB. Then you add the data points to 4 series via AddXY - this creates a TChartDateItem for each of these 10 million data points per series:
Code: Pascal  [Select][+][-]
  1. type
  2.   TChartDataItem = packed record
  3.   public
  4.     X, Y: Double;                           // 2x8 bytes
  5.     Color: TChartColor;                     // 4 bytes
  6.     Text: String;                           // 4 bytes (on 32 bit), or 8 bytes (on 64 bit)
  7.     XList: TDoubleDynArray;                 // 4 bytes on (32 bit), or 8 bytes (on 64 bit)
  8.     YList: TDoubleDynArray;                 // 4 bytes on (32 bit), or 8 bytes (on 64 bit)
  9.     // (methods...)
  10.   end;
These are 32 bytes per data point for a 32-bit application, or 44 bytes for 64-bit. Having 10 million data points per series you need 320 (or 440) MB per series. Together with the 320 MB for the data matrix you occupy 640 MB after the first series (or 760 MB). Each following series would require another 320 MB (440 MB). But with the IDE running I am sure that you don't have enough memory for populating all series. No problem, however, for a 64-bit application: I have 32GB on my system - and here your test application runs correctly in the IDE with all 4 channels.

So, the first, dumb, hint is to switch to 64-bit and if required buy more RAM for your system...

The second, more clever, hint is: You already have the data in a matrix in memory. Why do you add the values again to the series's internal ListSource which massively increases the memory load? Instead, use a TUserDefinedChartSource which just takes the values from the data matrix and passes them over to the series in a way the series expects it - there is no additional memory load per series, only the 320MB for the data. Look at the attached modified demo. It runs correctly even as 32-bit application (although I'd recommend to go to 64-bit here, too).

[EDIT]
The UserDefinedChartSource is already about a factor 2 faster than the internal ListSource. I get an additional factor 2 or so by setting the Sorted property of the UserDefinedChartSource to true - this prevents it from trying to sort the values on its own. Add the following code to the source attached:
Code: Pascal  [Select][+][-]
  1. procedure TForm1.FormCreate(Sender: TObject);
  2. begin
  3.   CS0 := TUserDefinedChartSource.Create(Self);
  4.   CS1 := TUserDefinedChartSource.Create(Self);
  5.   CS2 := TUserDefinedChartSource.Create(Self);
  6.   CS3 := TUserDefinedChartSource.Create(Self);
  7.  
  8.   CS0.Sorted := true;
  9.   CS1.Sorted := true;
  10.   CS2.Sorted := true;
  11.   CS3.Sorted := true;
  12.  
  13.   CS0.OnGetChartDataItem := @DataToSeriesHandler;
  14.   CS1.OnGetChartDataItem := @DataToSeriesHandler;
  15.   CS2.OnGetChartDataItem := @DataToSeriesHandler;
  16.   CS3.OnGetChartDataItem := @DataToSeriesHandler;
  17.   ...
« Last Edit: October 25, 2022, 04:58:03 pm by wp »

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: Issue with greater input in TLineSeries.
« Reply #2 on: October 26, 2022, 12:20:29 pm »
Seems that you are on a 32-bit system, or build the test application for 32-bit.
Indeed, it is written in my signature, next time I will ad this info in the post, since the signature changes.

So, the first, dumb, hint is to switch to 64-bit and if required buy more RAM for your system...
If I have switched to 32 bit I would get the wrong impression that my code is okay.

The second, more clever, hint is: You already have the data in a matrix in memory. Why do you add the values again to the series's internal ListSource which massively increases the memory load? Instead, use a TUserDefinedChartSource which just takes the values from the data matrix and passes them over to the series in a way the series expects it - there is no additional memory load per series, only the 320MB for the data. Look at the attached modified demo. It runs correctly even as 32-bit application (although I'd recommend to go to 64-bit here, too).
Indeed, now it can handle up to 6 channels with the IDE running. And it is much much faster. I still have not figured out how it works, but I am an optimist that I will get the idea.
Thanks!
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: Issue with greater input in TLineSeries.
« Reply #3 on: October 26, 2022, 01:53:00 pm »
I still have not figured out how it works, but I am an optimist that I will get the idea.
Each series which descends from TChartSeries (i.e. TLineSeries, TBarSeries, TAreaSeries etc. - see the class diagram in https://wiki.lazarus.freepascal.org/TAChart_documentation#Series) expects the data values in a TCustomChartSource instance which passes each data point as a TChartDataItem record.

TCustomChartSource comes in many flavours.
  • The most important one is TListChartSource which stores every TChartDataItem in a list - this is what happens when you add data to a series directly (by calling Series.AddXY(x, y)) because every TChartSeries provides an internal TListChartSource for this purpose.
  • TUserDefinedChartSource works differently: it does not store any data at all; when the series needs a data value an event OnGetChartDataItem is fired by the UserDefinedChartSource, and your event handler must fill the provided TChartDataItem record for the given value index. So, when your original data are stored in an array simply read the array at the requested index and copy the data to the TChartDataItem record.
  • The other chart source types are explained in the documentation.
When the Source property of the series is empty the series uses data from the internal TListChartSource (accessible as public property ListSource). But when you hook another TCustomChartSource into the Source property, it will override the internal source and deliver the data for the series.

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: [SOLVED] Issue with greater input in TLineSeries.
« Reply #4 on: November 11, 2022, 12:53:43 pm »
I am trying (so far unsuccessfully - new series delete the content of the old ones) to implement this in a component.

I cannot understand what “HandlerCount” does.
I have removed it from your sample and nothing seems to change.

Also “CS0.Sorted” did not result in a significant improvement - drawing 6 curves may have decreased from 13 s to 12 s, but since I have measured with a manual stopwatch, I am not sure that there is any difference at all. Anyway, I think that the speed is absolutely okay..
« Last Edit: November 11, 2022, 02:26:08 pm by CM630 »
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: [SOLVED] Issue with greater input in TLineSeries.
« Reply #5 on: November 11, 2022, 02:30:13 pm »
I cannot understand what “HandlerCount” does.
I have removed it from your sample and nothing seems to change.
"HandlerCount"? This word is not mentioned anywhere in this thread... What do you mean? CS0.PointsNumber? This is the number of points to be plotted because the UserDefinedChartSource has no way of knowing this. It is starting with index=0 and fires the OnGetChartDateItem event in which it gets the data for the 1st (index=0) data point. Then it increments index and fires the event again, now for index=1, etc. etc. This is repeated and stops when index has reached the PointsNumber.


Also “CS0.Sorted” did not result in a significant improvement - drawing 6 curves may have decreased from 13 s to 12 s, but since I have measured with a manual stopwatch, I am not sure that there is any difference at all. Anyway, I think that the speed is absolutely okay..
I cannot remember what I did that had lead me to this statement, and when I repeat the "bigls-wp" project of the earlier post I don't see a signficant speed improvement with "Sorted := true" either: 7.8 sec vs 7.1 sec.
« Last Edit: November 11, 2022, 02:49:11 pm by wp »

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: [SOLVED] Issue with greater input in TLineSeries.
« Reply #6 on: November 11, 2022, 02:39:02 pm »
Indeed, there is no “HandlerCount” in your original source, the only believable explanation is that I have put it and forgotten about it.
Sorry for that.
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: Issue with greater input in TLineSeries.
« Reply #7 on: November 16, 2022, 10:00:05 am »
I am still having some issues.
One of them is that I get an exception on line TLineSeries (Chart1.Series[0]).Clear; , shown on the attached image, when executing the code below.
This line used to work, when adding the data with
for row:=0 to 10000000-1 do ls0.AddXY(row/1000,random);

Code: Pascal  [Select][+][-]
  1.  
  2. procedure TForm1.Button1Click(Sender: TObject);
  3. var
  4.   ChanCount:integer=6;
  5.   ChanIndex: integer=0;
  6.   row:integer=0;
  7.   points: integer=  10{000000};
  8. begin
  9.   SetLength(Data, ChanCount);
  10.   for ChanIndex:= 0 to ChanCount-1 do
  11.     SetLength(Data[ChanIndex],points);
  12.  
  13.  
  14.   for ChanIndex:= 0 to ChanCount-1 do
  15.     for row := 0 to points-1 do
  16.       Data[ChanIndex,row]:=random+ChanIndex*5;
  17.  
  18.  
  19.   CS0.PointsNumber := length(Data[0]);
  20.   CS1.PointsNumber := length(Data[1]);
  21.   CS2.PointsNumber := length(Data[2]);
  22.   CS3.PointsNumber := length(Data[3]);
  23.   CS4.PointsNumber := length(Data[4]);
  24.   CS5.PointsNumber := length(Data[5]);
  25.  
  26.  
  27.   TLineSeries (Chart1.Series[0]).Clear;
  28. end;

« Last Edit: November 16, 2022, 10:17:41 am by CM630 »
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: Issue with greater input in TLineSeries.
« Reply #8 on: November 16, 2022, 10:51:11 am »
Just like the error message says: The chartsource from which the series receives its data is not editable. This is because now you store data in an external array (Data[..,...]) which is connected to the series via the userdefined chartsources, and TUserDefinedChartSource is not editable... In your old code using series.AddXY the data were stored in a TListChartSource maintained internally by the series - and this is editable. Therefore, series.Clear used to work in your old code, but does not work any more in the new code.

I don't know why you are calling series.Clear at all. Maybe because you want to provide new data? And in fact, there is a potential problem in my code: Setting the PointsNumber of the TUserDefinedChartSource does not do anything with the series when the number of datapoints is not changed. When you want to "renew" the series with new data you should call TUserDefinedChartSource.Reset after setting its PointsNumber; this will erase previously aquired extent limits.

Code: Pascal  [Select][+][-]
  1. procedure TForm1.Button1Click(Sender: TObject);
  2. var
  3.   ChanCount:integer=6;
  4.   ChanIndex: integer=0;
  5.   row:integer=0;
  6.   points: integer=  10{000000};
  7. begin
  8.   SetLength(Data, ChanCount);
  9.   for ChanIndex:= 0 to ChanCount-1 do
  10.     SetLength(Data[ChanIndex],points);
  11.  
  12.   for ChanIndex:= 0 to ChanCount-1 do
  13.     for row := 0 to points-1 do
  14.       Data[ChanIndex,row]:=random+ChanIndex*5;
  15.  
  16.   CS0.PointsNumber := length(Data[0]);
  17.   CS1.PointsNumber := length(Data[1]);
  18.   CS2.PointsNumber := length(Data[2]);
  19.   CS3.PointsNumber := length(Data[3]);
  20.   CS4.PointsNumber := length(Data[4]);
  21.   CS5.PointsNumber := length(Data[5]);
  22.  
  23.   CS0.Reset;
  24.   CS1.Reset;
  25.   CS2.Reset;
  26.   CS3.Reset;
  27.   CS4.Reset;
  28.   CS5.Reset;
  29. end;

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: Issue with greater input in TLineSeries.
« Reply #9 on: November 23, 2022, 11:34:56 am »
So it makes sense that
LS0.SetYValue(0,7);  throws an exception.
I did

Code: Pascal  [Select][+][-]
  1.   Data[0,1]:= 7; //this does not cause a change
  2.   cs[0].Reset;   //This changes the Y of the point to 7


But how am I to change the X of the value?
In DataToSeriesHandler it is set by AItem.X := AIndex*some_value; but CS0.Item is not accessible.

Also, when I drag a point will the DATA array change according the new value of the dragged point?
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: Issue with greater input in TLineSeries.
« Reply #10 on: November 23, 2022, 02:19:18 pm »
So it makes sense that
But how am I to change the X of the value?
In DataToSeriesHandler it is set by AItem.X := AIndex*some_value; but CS0.Item is not accessible.
My example follows the data structure that you gave in the first post. This defines a matrix of floats where the columns correspond to the first index, and the y values correspond to the second index. There is no X value in this structure. For plotting, it is assumed that the x values are equal to the data point index, multiplied by some scaling factor.

The idea behind the UserDefinedChartSource is that you have your data in any structure, an array of x/y records, separate arrays for x and y, a TList, columns in a string grid, or whatever. The UserdefineChartSource is the link between this arbitrary data structure and the series. Whatever you want to do with the data points in the series, must be done in this external datastructure, and then redraw the series. So, when your data structure does not provide an x value because you plot against the index, you must exchange the values in the data structure if you want the move a data point to another x position.

Also, when I drag a point will the DATA array change according the new value of the dragged point?[/size]
You mean: dragging with the TDatapointDragTool? This is not possible because when you drag a datapoint on the screen the series must write the edited value back to the chartsource. But the UserDefinedChartSource does not know how data are stored. If you insist on the possibility to drag data points (how do you do this, BTW, when you have 10 millions per channel?) is to return to the TListChartSource, and AddXY values to the series as you did initially. And this brings you back the speed problem...

Is it absolutely necessary that you display all 10 millions of data values? You can't see any details here at all, just a broad colored band. Therefore I'd suggest that you have a zoomed view displaying only 1000 values (or so). In this case you'd only have to read these 1000 values into the series (AddXY), and everything will be much faster. And when you scroll through the original data set you erase the series and reload those 1000 in the current viewport.

In the attached project you can see the principle. When the checkbox is checked only a viewport with 1000 values is visible, but you can use the scrollbar to scroll through all data. In this state, the series gets the data from the internal chartsource (TListChartSource) and you thus are able to drag data points (CTRL+Left mouse button) (I restricted dragging to the y value only because I did not want to confuse by rearranging the original data array when the dragged value must be written back to the original data array. In this window mode, handling is very fast and fluent. On the other hand, when you uncheck the checkbox the series are connected to the UserdefinedChartSources to display all the data, and the program becomes slow again because millions of data points must be handled.

« Last Edit: November 23, 2022, 03:31:09 pm by wp »

CM630

  • Hero Member
  • *****
  • Posts: 1018
  • Не съм сигурен, че те разбирам.
    • http://sourceforge.net/u/cm630/profile/
Re: Issue with greater input in TLineSeries.
« Reply #11 on: November 23, 2022, 03:15:41 pm »
Is it absolutely necessary that you display all 10 millions of data values?
I am quite sure that I need these 10 million point, but I hope that I will not need dragging.
Using data with lower precision might possibly be more useful, but I rather think  the effort will not be worth.
If I wont need dragging, everything seems to work fine for now.

In the attached project you can see the principle. When the checkbox is checked only a viewport with 1000 values is visible, but you can use the scrollbar to scroll through all data. ...
Do you mean the sample from the last month? Because it contains no checkbox and nothing new is attached.
Лазар 2,2,4 32 bit; FPC3,2,2; rev: Lazarus_2_2_4 on W10 64bit.

wp

  • Hero Member
  • *****
  • Posts: 10296
Re: Issue with greater input in TLineSeries.
« Reply #12 on: November 23, 2022, 03:31:46 pm »
Do you mean the sample from the last month? Because it contains no checkbox and nothing new is attached.
Sorry, I forgot to attach the zipped project... It is attached now.

 

TinyPortal © 2005-2018