Recent

Author Topic: Correct way to plot rather large amount of data  (Read 629 times)

Kwadrum

  • New member
  • *
  • Posts: 9
Correct way to plot rather large amount of data
« on: August 11, 2020, 02:33:28 am »
Hi,

I have a question on how to plot rather large amount of data correctly.

The program I’ve made basically reads raw data from a file, does calculations and the output is two arrays of numbers, from which one is small (~20 numbers) and the other is quite big (several hundred to a thousand numbers).
I want to plot the both datasets on separate graphs. My first guess would be to use UserDefined ChartSource, as described here:
https://wiki.freepascal.org/TAChart_Tutorial:_Userdefined_ChartSource.
But I have a problem that all the calculations are performed inside the procedure which is a handler for a “Start” button and therefore the output data arrays are currently local. The handler for the OnGetChartDataItem needs access to the data, so I made those output data arrays global variables (which works for now, but maybe is not a very good idea).
My question is how should it be done correctly? Is it possible to make the OnGetChartDataItem handler nested in the main calculation procedure (probably not, but still)? I do not have much experience, so kind of lost so far…

Thank you.

wp

  • Hero Member
  • *****
  • Posts: 7525
Re: Correct way to plot rather large amount of data
« Reply #1 on: August 11, 2020, 09:51:35 am »
The UserDefinedChartSource provides an interface to make data of any structure accessible by a chart series. For example, when you perform a calculation and have the calculated data in an array then you can plot these data directly (by means of the UserDefinedchartSource); the conventional "series.AddXY()" would dupliate the data in an internal listsource of the series and waste memory. But of course, the array must exist as long as the series exists. If you have your data in a local array you must either make the array "global" (you can also add them to the corresponding form of the application), or you should use the internal chartsource of the series or an external ListChartSource.

I think there is no clear right or wrong; it depends on what you are doing and what you want to achieve.

The ListChartSource reserves space for additional information (multiple x, y values, individual point color, data point label), i.e. storing data in simple x/y arrays and using a UserDefinedChartSource is more memory efficient. And if you want a string grid to display the plotted values you'll have the data in memory a strings another time. Morover, to me it is also clearer to separate the data from the series.

Data in ListChartSources, however, can be added simply by calling the series' AddXY() method. Data can be sorted directly (well, using a UserDefinedChartSource you can do this too, but you need Laz trunk for it).

My personal strategy: Use the (internal/external) ListChartSource for "simple" plots and applications. More sophisticated programs where the same data are used at several places require data in their own data structures and UserDefinedChartSources.
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Kwadrum

  • New member
  • *
  • Posts: 9
Re: Correct way to plot rather large amount of data
« Reply #2 on: August 12, 2020, 01:41:17 am »
WP, thank you for detailed explanations,

So if I understood it correctly, it is advisable to use UserDefinedChartSource in order to avoid moving (raw) data into other types of sources (i.e. no duplication of big volumes of data). I am not planning to do anything specific with the plot (no sorting, etc), so I won't need any additional options provided by, say, ListChartSource.

Then if I have an array with data, generated in a procedure which in turn is a handler for one of the buttons (and I want to keep it like this), is making this data array "global" the only reasonable option for plotting via UserDefinedChartSource? I guess I can't nest UserDefinedChartSource0GetChartDataItem handler into a button handler (to make it "local" like the data array), right? And I do not want to copy the data from this array into other data source either.

Thank you.

wp

  • Hero Member
  • *****
  • Posts: 7525
Re: Correct way to plot rather large amount of data
« Reply #3 on: August 12, 2020, 09:57:12 am »
So, if you do not want to use TListChartSource and want to keep your data arrays local to the button-click handler then you only have the option to create the plot there and destroy it when you exit the click handler. This is because the chart must be able to redraw itself at any time, amd for this it needs the data. As a consequence your program will not be able to do anything else while the click handler runs.

OK - there is another option to make the chart more persistent: Draw it to a bitmap and display the bitmap after exiting the click handler. But you give up many advantages of the living chart: redraw when it is resized, zooming, panning, etc.
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Kwadrum

  • New member
  • *
  • Posts: 9
Re: Correct way to plot rather large amount of data
« Reply #4 on: August 12, 2020, 10:46:52 pm »
WP, thanks for the explanations!

As I mentioned, right now everything seems to work with data array made "global" and I'm kind of OK with that. Here is a heavily stripped excerpt of the code, but I hope it is clear enough.

Code: Pascal  [Select][+][-]
  1. unit Main;
  2.  
  3. {$mode objfpc}{$H+}
  4.  
  5. interface
  6.  
  7. uses
  8.   Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls, ExtCtrls,
  9.   Menus, ComCtrls, TAGraph, TASeries, TASources, StrUtils, Math, DateUtils,
  10.   MMSystem, TACustomSource;
  11.  
  12. type
  13.  
  14.   { TfMain }
  15.  
  16.   TfMain = class(TForm)
  17.     Chart1: TChart;
  18.     Chart1BarSeries0: TBarSeries;
  19.         .......
  20.     UserDefinedChartSource0: TUserDefinedChartSource;
  21.         .......
  22.     procedure bRunClick(Sender: TObject);       //main calculations, filling AR array
  23.        
  24.     procedure UserDefinedChartSource0GetChartDataItem(
  25.       ASource: TUserDefinedChartSource; AIndex: Integer;
  26.       var AItem: TChartDataItem);
  27.         .......
  28.        
  29.   private
  30.  
  31.   public
  32.  
  33.   end;
  34.  
  35. var
  36.   fMain: TfMain;
  37.   AR: array [0..4,0..4] of cardinal;
  38.   ........
  39.  
  40. implementation
  41.  
  42. {$R *.lfm}
  43.  
  44. { TfMain }
  45.  
  46. ........
  47.  
  48. procedure TfMain.UserDefinedChartSource0GetChartDataItem(
  49.   ASource: TUserDefinedChartSource; AIndex: Integer; var AItem: TChartDataItem);
  50. begin
  51.   AItem.X:=AIndex;
  52.   AItem.Y:=AR[0][AIndex];
  53. end;
  54. .........
  55. procedure TfMain.bRunClick(Sender: TObject);   //main calculations + filling data array AR
  56. .........
  57. UserDefinedChartSource0.PointsNumber := 5;
  58. UserDefinedChartSource0.Reset;                      //bar graph plotting
  59. ..........
  60.  
  61. end.

However I'm still curious about how it could be done with "local" data array. Is it possible "simply" to move the whole procedure UserDefinedChartSource0GetChartDataItem into the procedure TfMain.bRunClick or do you mean anything else?

Thank you.
« Last Edit: August 12, 2020, 10:50:07 pm by Kwadrum »

wp

  • Hero Member
  • *****
  • Posts: 7525
Re: Correct way to plot rather large amount of data
« Reply #5 on: August 12, 2020, 11:18:41 pm »
Your code seems to be correct. I probably misunderstood you because you were saying "output data arrays are currently local". But instead of a global array AR I would prefer to have the array within the form because when there are multiple instances of TfMain each one of them would access the same data - probably not what you want.:

Code: Pascal  [Select][+][-]
  1. type
  2.  
  3.   { TfMain }
  4.  
  5.   TfMain = class(TForm)
  6.     Chart1: TChart;
  7.     Chart1BarSeries0: TBarSeries;
  8.         .......
  9.     UserDefinedChartSource0: TUserDefinedChartSource;
  10.         .......
  11.     procedure bRunClick(Sender: TObject);       //main calculations, filling AR array
  12.        
  13.     procedure UserDefinedChartSource0GetChartDataItem(
  14.       ASource: TUserDefinedChartSource; AIndex: Integer;
  15.       var AItem: TChartDataItem);
  16.         .......
  17.        
  18.   private
  19.     AR: array[0..4, 0..4] of Cardinal;  // or move it to "public" when you need the array from somewhere else
  20.  
  21.   public
  22.  
  23.   end;

Is it possible "simply" to move the whole procedure UserDefinedChartSource0GetChartDataItem into the procedure TfMain.bRunClick
No. The OnGetChartDataItem event is declared as "procedure(...) of object" -- this means it must be used as a procedure (with the given parameters) of an object, in other words: as a method of the form (or some other class). In principle, it could also have been declared within TAChart as "nested" (i.e. a "local" procedure within another procedure) but this will not work because the event handler is needed as long as the UserDefinedChartSource exists since the chart must be able to redraw itself and its series at any time; in case of a nested procedure the event handler would no longer be available once the outer procedure has been left.


Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Kwadrum

  • New member
  • *
  • Posts: 9
Re: Correct way to plot rather large amount of data
« Reply #6 on: August 13, 2020, 02:01:28 am »
I probably misunderstood you because you were saying "output data arrays are currently local".
Sorry, that were me who was not clear enough. I wrote the "calculation" part of the code separately and then, trying to fit it into the form, realized the problem with local data array and UserDefinedChartSource handler. The first instinctive move was to make the array global, which I did, and it works fine. However, I thought that there might be another way of doing the same thing, while keeping the data array local.
In principle, it could also have been declared within TAChart as "nested" (i.e. a "local" procedure within another procedure) but this will not work because the event handler is needed as long as the UserDefinedChartSource exists since the chart must be able to redraw itself and its series at any time; in case of a nested procedure the event handler would no longer be available once the outer procedure has been left.
That might work for me, actually. So, if I declare procedure UserDefinedChartSource0GetChartDataItem within procedure bRunClick, I think I still will be able to plot the graph from within procedure bRunClick. This is the end of the whole program, and any other manipulations would mean starting the whole program from the very beginning and the current graph won't be needed any more. Not sure if I managed to explain it, though...
I can show the whole code or, if I manage to nest UserDefinedChartSource0GetChartDataItem in bRunClick I'll see if the result suits my needs  :). Or do I have to declare it somewhere else? Do I need to specifically alter the syntax of declaration of UserDefinedChartSource0GetChartDataItem when it's inside bRunClick or is it considered as a "regular" procedure within another procedure?

Thank you.

P.S. Sorry for bugging, just really want to know how to make it one and another way.
« Last Edit: August 13, 2020, 02:08:16 am by Kwadrum »

wp

  • Hero Member
  • *****
  • Posts: 7525
Re: Correct way to plot rather large amount of data
« Reply #7 on: August 13, 2020, 11:18:02 am »
So, if I declare procedure UserDefinedChartSource0GetChartDataItem within procedure bRunClick, I think I still will be able to plot the graph from within procedure bRunClick. This is the end of the whole program, and any other manipulations would mean starting the whole program from the very beginning and the current graph won't be needed any more.
Forget it. The compiler will not allow it because a "nested" procedure is not a "procedure of class" - that's what I tried to explain in the previous post. If you don't believe me try to compile the attached project - you will get compilation error message
Code: [Select]
unit1.pas(55,72) Error: Incompatible type for arg no. 1: Got "<address of procedure(TUserDefinedChartSource;LongInt;var TChartDataItem) is nested;Register>",
expected "<procedure variable type of procedure(TUserDefinedChartSource;LongInt;var TChartDataItem) of object;Register>"
On the other hand, if you select the non-nested event handler the program compiles and runs fine.

This is the end of the whole program, and any other manipulations would mean starting the whole program from the very beginning and the current graph won't be needed any more.
Since you have a GUI program the program flow is determined by user actions, not by your intentions. Even if nothing new may happen in your program after showing the chart the chart will still be visible and the user, for example could drag another window over the chart, and in this case the chart would have to be redrawn. And with every redraw the chart needs the data and, in the case of the UserdefinedChartSource, the OnGetChartDataItem event handler.
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

Kwadrum

  • New member
  • *
  • Posts: 9
Re: Correct way to plot rather large amount of data
« Reply #8 on: August 17, 2020, 06:14:27 pm »
Thank you, I see now how it works (or, better to say, why it does not).
If you do not mind, I have a couple of additional questions.

1. Do I understand it correctly, that in your code
Code: Pascal  [Select][+][-]
  1. UserDefinedChartSource1.OnGetChartDataItem := @UserDefinedChartSource1GetChartDataItem;
you point OnGetChartDataItem event to UserDefinedChartSource1GetChartDataItem procedure at run time instead of setting it ahead in the Object Inspector?

2. What is that parameter "ASource", which is not used and pops up in the compiler output as a hint? I understand it is a preset thing that should not be altered and the hints are not errors, but I did not find anything on how this parameter should be addressed (maybe some additional features?).

3. If I, for example, plot several (5-7) series on the same graph, is it any way to address them at ones (or to put them into a cycle) when resetting, etc? I tried a couple of things and walked though the properties/functions in the pop-up assistant in Lazarus, but do not see anything I could use (maybe missing?). What I mean are fragments like these:
Code: Pascal  [Select][+][-]
  1. Chart1BarSeries0.Active := False;
  2. Chart1BarSeries1.Active := False;
  3. // + several more
  4.  
  5. UserDefinedChartSource0.PointsNumber := 5;
  6. UserDefinedChartSource1.PointsNumber := 5;
  7. // + several more with the same value of 5
  8.  
  9. UserDefinedChartSource0.Reset;
  10. UserDefinedChartSource1.Reset;
  11. // + several more
  12.  

I think I can replace ChartBarSeries Active/Inactive with simple hiding the graph (Graph1.Hide), but all the rest look way too long :( I need this part for a case when the program (without restarting) opens another data file and creates another set of bar series (old series should disappear).

Thank you.
« Last Edit: August 17, 2020, 06:19:29 pm by Kwadrum »

wp

  • Hero Member
  • *****
  • Posts: 7525
Re: Correct way to plot rather large amount of data
« Reply #9 on: August 17, 2020, 07:18:13 pm »
1. Do I understand it correctly, that in your code
Code: Pascal  [Select][+][-]
  1. UserDefinedChartSource1.OnGetChartDataItem := @UserDefinedChartSource1GetChartDataItem;
you point OnGetChartDataItem event to UserDefinedChartSource1GetChartDataItem procedure at run time instead of setting it ahead in the Object Inspector?
Yes, this is the way how to assign an event handler to an event at runtime. Don't miss the '@' when your code is in {$mode objfpc}, but do not use it in {$mode Delphi}. The procedure itself can have any name, it is only required that it is a method (i.e. is declared within the context of a class) and it must have the same parameter list as required by the event.

2. What is that parameter "ASource", which is not used and pops up in the compiler output as a hint? I understand it is a preset thing that should not be altered and the hints are not errors, but I did not find anything on how this parameter should be addressed (maybe some additional features?).
The parameter "ASource" identifies the UserdefinedChartSource which is requesting the data values. Sometimes it is useful to reuse the same event handler for different sources. This way you can determine the source which is currently working. When the warning annoys you you can right-click on the warning message and select an option from the context menu to hide the "unused parameter" warning either for the current line or for the entire unit.

3. If I, for example, plot several (5-7) series on the same graph, is it any way to address them at ones (or to put them into a cycle) when resetting, etc? I tried a couple of things and walked though the properties/functions in the pop-up assistant in Lazarus, but do not see anything I could use (maybe missing?). What I mean are fragments like these:
Code: Pascal  [Select][+][-]
  1. Chart1BarSeries0.Active := False;
  2. Chart1BarSeries1.Active := False;
  3. // + several more
  4.  
  5. UserDefinedChartSource0.PointsNumber := 5;
  6. UserDefinedChartSource1.PointsNumber := 5;
  7. // + several more with the same value of 5
  8.  
  9. UserDefinedChartSource0.Reset;
  10. UserDefinedChartSource1.Reset;
  11. // + several more
  12.  
The chart contains a list of all series, named Series[index]; their count is SeriesCount. However, since TAChart supports many series types, this list returns only the must basic series type, TBasicChartSeries. If you want to address some particular properties of the series you must check the series type and make a type-cast. Please look at the TAChart documentation to learn about the inheritance of the series types (https://wiki.lazarus.freepascal.org/TAChart_documentation#Series). The most elemental series type which supports a chart source is TChartSeries. In the same way you can address the source of the chart series. You can use the following code instead of yours, and it will be valid for all series inheriting from TChartSeries, i.e. TLineSeries, TBarSeries, TAreaSeries etc.
Code: Pascal  [Select][+][-]
  1. var
  2.   i: Integer;
  3.   ser: TChartSeries;
  4.   src: TUserDefinedChartSource;
  5. begin
  6.   for i := 0 to Chart1.SeriesCount-1 do
  7.     if (Chart1.Series[i] is TChartSeries) then
  8.     begin
  9.       ser := TChartSeries(Chart1.Series[i]);
  10.       ser.Active := false;
  11.       if ser.Source is TUserDefinedChartSource then
  12.       begin
  13.         src := TUserDefinedChartSource(ser.Source);
  14.         src.PointsNumber := 5;
  15.         ser.Reset;
  16.       end;
  17.     end;
  18. end;

I think I can replace ChartBarSeries Active/Inactive with simple hiding the graph (Graph1.Hide), but all the rest look way too long :( I need this part for a case when the program (without restarting) opens another data file and creates another set of bar series (old series should disappear).
I think hiding and showing the Chart causes some disturbing flicker. In order to remove the old series you can set its Active to false, or you can delete it altogether (FreeAndNil(Chart1BarSeries1)), or you can keep the old series and you just make the data array used by the UserDefinedChartSource point to the new data.
Mainly Lazarus trunk / fpc 3.2.0 / all 32-bit on Win-10, but many more...

 

TinyPortal © 2005-2018