TChart: +32% Processor load while plotting more than 4000 datapoints

[KatiYusha]

Hello, I am designing a program that logs some serial data from an MCU and display them on a chart.
The MCU sends data every 600ms and I need it to work for at least 2 or 3 hours, it is working fine until I have more than 4000 to display (40minutes) then everything starts to flicker and when I monitor the load on my CPU it increases by 20% or so and sometimes the app crashes.
What could be the problem?

I'm using Lazarus 2.2.4
Intel Core i5 6500
Windows 10

[jamie]

Bits ? 32/64

[wp]

Show me the code how you are adding the new data values.

Did you enclose this in a BeginUpdate/EndUpdate block?

Code: Pascal  [Select][+][-]

1. procedure TForm1.Timer1Timer(Sender: TObject);
2. var
3.   x, y: Double;
4.   i: Integer;
5. begin
6.   Chart1LineSeries1.BeginUpdate;
7.   x := Chart1LineSeries1.GetXMax;
8.   for i := 0 to 999 do
9.   begin
10.     x := x + 1;
11.     y := 0.001 * x*x;
12.     Chart1LineSeries1.AddXY(x, y);
13.   end;
14.   Chart1LineSeries1.EndUpdate;
15. end;

[Curt Carpenter]

4000 points is a lot for a chart in human-readability terms.  Do you maybe need a different approach that won't require you to redraw all of the points every time you get a new one?  Something like two layered bitmaps, a base layer that is updated once every 100 or so new points, and an overlay that contains only the newest points?  Or a different style of chart?

[wp]

Basically, this is correct, but the limit is much higher. The attached demo adds 100,000 data points at the beginning, and then adds another 1000 (slightly offset) twice per seconds. And it runs very smoothly. With 1 million points it feels sticky when the window is dragged - but this is much, much more data that we are talking here.

[Thaddy]

The classic way to solve this is scaling: when there are more data points than can visually be displayed, scale to the maximum control resolution. An example is in e.g. Audio controls to display waveforms.
Also, TChart is a display component and should not be used to actually store data. That should be done on either file or, when available, in memory or both (using memory mapping).
So scale the data to what can actually be displayed. Anything else makes no sense.

[KatiYusha]

Sorry for the late reply.
I forgot to mention that I am displaying two TLineSeries

Bits ? 32/64

* I am using Windows 64 bits

Show me the code how you are adding the new data values.

Did you enclose this in a BeginUpdate/EndUpdate block?

Code: Pascal  [Select][+][-]

1. procedure TForm1.Timer1Timer(Sender: TObject);
2. var
3.   x, y: Double;
4.   i: Integer;
5. begin
6.   Chart1LineSeries1.BeginUpdate;
7.   x := Chart1LineSeries1.GetXMax;
8.   for i := 0 to 999 do
9.   begin
10.     x := x + 1;
11.     y := 0.001 * x*x;
12.     Chart1LineSeries1.AddXY(x, y);
13.   end;
14.   Chart1LineSeries1.EndUpdate;
15. end;

*I did not use BeginUpdate/EndUpdate in my original code but the problem persists even when using them.
 This is the code for loading and displaying the data from a string list:

Code: Pascal  [Select][+][-]

1. procedure TForm1.LoadButtonClick(Sender: TObject);
2. {Start loading a CSV file}
3. var
4.    Length : Integer = 0;
5.    a : Integer = 0;
6.    x : Integer = 0;
7.    y : Integer = 0;
8. begin
9. //code that loads a string List
10. Lenght := StringList.Count;
11.  while a < (Length - 1) do
12.  begin
13.          LongStr := strlist[a];
14.          x := (a + datapoints - 1) / divider;
15.          y := StrToInt(Copy(LongStr , 1 , 4));
16.          VoltageGraph.AddXY(x, y);
17.          y := StrToInt(Copy(LongStr , 6 , 4));
18.          CurrentGraph.AddXY(x, y);
19.          inc(a);
20.  end;
21. end;
22.  

4000 points is a lot for a chart in human-readability terms.  Do you maybe need a different approach that won't require you to redraw all of the points every time you get a new one?  Something like two layered bitmaps, a base layer that is updated once every 100 or so new points, and an overlay that contains only the newest points?  Or a different style of chart?

*My idea was to record the power consumption of IOT devices for at least 24 hours and then analyze them to improve battery consumption. I think <4k points aren't enough to capture current spikes on the PSU line.

The classic way to solve this is scaling: when there are more data points than can visually be displayed, scale to the maximum control resolution. An example is in e.g. Audio controls to display waveforms.
Also, TChart is a display component and should not be used to actually store data. That should be done on either file or, when available, in memory or both (using memory mapping).
So scale the data to what can actually be displayed. Anything else makes no sense.

*I am currently storing the data on a CSV File

[wp]

I don't understand: Your code in the previous commit suggests that you create the plot upon a buttonclick, but in the first post you said that the plot chokes when data array at 600ms intervals after some time.
So, is this a "live" plot which displays the data while they are arriving, or is the plot created upon request?

You say data are stored in a CSV file. In the former case of the "live" plot it looks unfavourable to me when you re-read the CSV file again and again. As an alternative to reload the CSV file again and again I'd propose to keep the data in an array of records containing time, current and current. You can use a TUserDefinedChartSource to pass the data from the array directly to the lineseries without having to "AddXY" them. This avoids double storage and is fast.

When you receive data from the external device, is this running in a separate thread? Are you sure that thread handling is correct?

[KatiYusha]

Actually, it has two main functions:

• Plot/record the received live data
• or load an existing CSV file when needed

For the serial part, I used TLazSerial to handle the incoming serial data.
A timer makes the "pace" of 600ms (sorry for my terrible english )

Code: Pascal  [Select][+][-]

1. procedure TForm1.Timer1Timer(Sender: TObject);
2. var
3.    Conf0 : TIniFile;
4.    temp: Integer = 0;
5.    qsdf : integer = 0;
6. begin
7.      if (Invalid = false) and (Serial.Active = true) then
8.      begin
9.                VoltageGraph.BeginUpdate;
10.                CurrentGraph.BeginUpdate;
11.                ZeroLine.BeginUpdate;
12.                temp := StrToInt(Voltage);
13.                ZeroLine.ADDXY(xindex, 0);
14.                VoltageGraph.AddXY(xindex, temp);
15.                temp := StrToInt(Current);
16.                CurrentGraph.AddXY(xindex, temp);
17.                VoltageGraph.EndUpdate;
18.                CurrentGraph.EndUpdate;
19.                ZeroLine.EndUpdate;
20.                inc(xindex);
21.       end;
22. end;
23.  

[wp]

This code looks correct (however,  the variable of type TIniFile might be an indication that there's more in your original procedure than you show?) How do you get the received data from Timer1Timer into the CSV file? The procedure as posted only loads the series.

Your ZeroLine series is a TLineSeries, isn't it? Alternatively you could use a TConstantLine series which does not require any data values except for the single Posiiton value (storing zeros all over the time is not very clever...). When you add it to the chart it already has the correct settings that you need: LineStyle = lsHorizontal and Position := 0.

Which of the two main functions causes the trouble?

[KatiYusha]

TIni file is for storing device parameters, it has nothing to do with the chart and runs only once.

Which of the two main functions causes the trouble?

I do not know either,
I designed it to be as simple as possible, the rest of the program is working fine except for the chart.
Maybe I should use 12 seconds sampling only as a last resort to decrase the datapoints?

[Curt Carpenter]

I would switch to just plotting pixels on a bitmap rather than using a chart since you have the data and seem to be using the chart as only a visual, qualitative guide rather than for any quantitative detail. 

Interesting application.

[Thaddy]

No you should scale the data points - as I wrote - and also find some different storage over CSV.

[KatiYusha]

No you should scale the data points - as I wrote - and also find some different storage over CSV.

I would switch to just plotting pixels on a bitmap rather than using a chart since you have the data and seem to be using the chart as only a visual, qualitative guide rather than for any quantitative detail. 

Interesting application.

How do I do that? I want to try both and see which suits me the best.

[ccrause]

Something else to consider: disable autoscale as much as possible, in your case the voltage should really be below 4.5V, so you can preconfigure the range of that axis. Also try to prescale the x-axis.  This may be more difficult if you don't know the total time for the plot, but one strategy is to manually increment the x-axis range by say 1 hour when your data reaches the current limit of the x-axis.

Here is an example which generates data for 3 line series.  When I set the loop to 10 hours, it generates over 100 000 data points which gets plotted in less than 2 seconds (which includes running a model to generate the OP series.  This was tested on my very modest dual core Celeron N3050 Linux laptop.