[SOLVED] Show progress on another form during data processing

[Jiří Huňáček]

Hi,

I am solving a problem with displaying the progress during data processing for one application.

I have an application where I import text output into a TStringList and then each line needs to be split with a separator and converted because the data in the columns contain spaces before and after the text. During the conversion, a new form will appear where there is only one text "Processing, please wait...". It occurred to me that I would add one more text there that would show the progress "line X / Y", where X means the current line and Y the total number of lines.
If there are few lines, then the conversion is fast and there is no need to give the user any info, but if there were more and it would take some time, it is ideal to show the progress to the user so that he knows that the program is doing something.

Now for the problem. I have 139000 rows and if I display the second form without the section

Code: Pascal  [Select][+][-]

1. with frmInformation do

it takes about 4 seconds to process. Then I insert this section and used two variants for the update

1. Application.ProcessMessages
The current and total rows information is updated during the loop and equals at the end of the loop, but processing 139K rows takes 44 seconds, 40 seconds longer.

2. Form2.Label.Refresh / Form2.Label.Invalidate
The information is updated, the processing takes about 4 seconds, but when the loop processing is finished, the information about the current line is not at the maximum value of 139K, but is at different values ​​each time (21K, 55K, etc.)

I don't know how or whether to get the value to be updated while the processing still takes the same amount of time, because there may not only be 139K rows, but even more.

Can anyone help? Thank you

The code section mentioned

Code: Pascal  [Select][+][-]

1. frmInformation.Show;
2. frmInformation.Label1.Caption := 'Conversion in progress, please wait...';
3. frmInformation.BorderIcons := [];
4. frmInformation.FormStyle:= fsStayonTop;
5. frmInformation.Refresh;
6.  
7. for StringsInputCurrentRow := 0 to StringInputRowCount - 1 do
8. begin
9.     // update the Information window
10.     with frmInformation do
11.     begin
12.         Label2.Caption := 'processing line ' + FormatFloat('# ### ###', StringsInputCurrentRow + 1) + ' / ' + FormatFloat('# ### ###', StringInputRowCount);
13. {
14.         Label2.Invalidate; // at the end of the loop  StringsInputCurrentRow <> StringInputRowCount - 1
15.         Label2.Refresh;    // at the end of the loop  StringsInputCurrentRow <> StringInputRowCount - 1
16. }
17.     end;
18.     Application.ProcessMessages; // processing 139000 lines takes 40 seconds longer
19.  
20. ... loop continuation ...
21.  
22. end;
23.  

[cdbc]

Hi
Use a worker thread and have that post messages to your form, for updating...
There should be plenty of examples here on the forum... Search is your friend.
Regards Benny

[Handoko]

The current and total rows information is updated during the loop and equals at the end of the loop, but processing 139K rows takes 44 seconds, 40 seconds longer.

Click the link below and go to the General category, there you can find Generating a huge text file demo. I shared a trick to improve program performance by sacrificing its responsiveness in the demo.

https://wiki.freepascal.org/Portal:HowTo_Demos

[Jiří Huňáček]

Thank you Handoko.

I just added this piece of code and it works great for numbers when I type "X / Y" and the conversion time goes down correctly to 4 seconds.

However, I have a problem with the ProgressBar, because it doesn't reach 100%, it always ends a little before the end.

Code: Pascal  [Select][+][-]

1.     ProgressBar1.Position := i;
2.     if (i mod 50) <> 0 then Continue; // this line is for better performance
3.     Application.ProcessMessages;

[Handoko]

Please provide a simple demo that can show the issue so we can download, run and inspect the problem.

[Thaddy]

Thank you Handoko.

I just added this piece of code and it works great for numbers when I type "X / Y" and the conversion time goes down correctly to 4 seconds.

However, I have a problem with the ProgressBar, because it doesn't reach 100%, it always ends a little before the end.

Code: Pascal  [Select][+][-]

1.     ProgressBar1.Position := i;
2.     if (i mod 50) <> 0 then Continue; // this line is for better performance
3.     Application.ProcessMessages;

Because the scale is from 0 to 99 for 100%, so mod 50 is wrong...? So set the scale explicitly to 1..100

[ASerge]

Use a worker thread...

+1

Example:

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes, SysUtils, Forms, Controls, ActnList, ExtCtrls, StdCtrls;
9.  
10. type
11.   TForm1 = class(TForm)
12.     actnStart: TAction;
13.     ActionList1: TActionList;
14.     Button1: TButton;
15.     Label1: TLabel;
16.     Timer1: TTimer;
17.     procedure actnStartExecute(Sender: TObject);
18.     procedure actnStartUpdate(Sender: TObject);
19.     procedure FormCreate(Sender: TObject);
20.     procedure Timer1Timer(Sender: TObject);
21.   private
22.     FCount, FPos: SizeInt;
23.     procedure Finished(Sender: TObject);
24.     procedure LongOperation;
25.     procedure ShowState;
26.   public
27.  
28.   end;
29.  
30. var
31.   Form1: TForm1;
32.  
33. implementation
34.  
35. {$R *.lfm}
36.  
37. { TForm1 }
38.  
39. procedure TForm1.actnStartExecute(Sender: TObject);
40. begin
41.   FCount := 0;
42.   FPos := 0;
43.   Timer1.Enabled := True;
44.   TThread.ExecuteInThread(@LongOperation, @Finished);
45. end;
46.  
47. procedure TForm1.actnStartUpdate(Sender: TObject);
48. begin
49.   (Sender as TAction).Enabled := not Timer1.Enabled;
50. end;
51.  
52. procedure TForm1.Finished(Sender: TObject);
53. begin
54.   Timer1.Enabled := False;
55.   ShowState;
56. end;
57.  
58. procedure TForm1.FormCreate(Sender: TObject);
59. begin
60.   Timer1.Enabled := False;
61.   Timer1.Interval := 300;
62. end;
63.  
64. procedure TForm1.LongOperation;
65. const
66.   CLinesCount = 1000000;
67. var
68.   i, k: SizeInt;
69. begin
70.   FCount := CLinesCount;
71.   for i := 0 to CLinesCount - 1 do
72.   begin
73.     FPos := i;
74.     // Do something
75.     for k := 1 to 10000 do
76.       ; // Nothing
77.   end;
78. end;
79.  
80. procedure TForm1.ShowState;
81. begin
82.   Label1.Caption := Format('Processed %d from %d', [FPos + 1, FCount]);
83. end;
84.  
85. procedure TForm1.Timer1Timer(Sender: TObject);
86. begin
87.   ShowState;
88. end;
89.  
90. end.

[Jiří Huňáček]

Now I tried it with the case where the ProgressBar is placed on the same form where the data is processed and the current line number is set to the maximum 50 000, but the ProgressBar is still catching up and not at the end.

Apparently I'll have to choose whether to show the progress with numbers or with a ProgressBar. Both are probably not possible for it to work correctly.

See attachment for my attempt.

[Handoko]

I couldn't reproduce the issue, perhaps because I was testing the code on Linux GTK2. My Lazarus wasn't prepared to cross compiled to Windows so I couldn't test it on Windows.

Try to change that line to:

Code: Pascal  [Select][+][-]

1.     if i < (Memo1.Lines.Count-1) then
2.       if (i mod 50) <> 0 then Continue;

Let me know if it doesn't work, so I will install Lazarus on a virtual machine to inspect the problem.

[Jiří Huňáček]

Changed in PB demo but unfortunately not working. I tried to make a screen at the moment when the current line is equal to the total number. From that moment on, PrograssBar just coasts to the end.

[Handoko]

Okay, give me some minutes. Or maybe 10 minutes.

[Bart]

Now I tried it with the case where the ProgressBar is placed on the same form where the data is processed and the current line number is set to the maximum 50 000, but the ProgressBar is still catching up and not at the end.

May or may not be your issue.

On Windows the progressbar lacks behind because MS made it animated.
If you want an immediate update, set Positions to Desired + 1, and after that to Position (he animation does not occur when you decrease Position).
Yes, it sucks, because you cannot do that for Position = Max.

There is a reason why so many people write their own progressbar.
Even I did, because I was fed up with this.

Bart

[Handoko]

I cannot reproduce the problem. I tested the code on Windows 7 64-bit virtual machine with Lazarus 4.0RC1 i386-win32. See the attached video below.

Mine only shows 1 point difference but OP's screenshot shows a big difference.

Can anyone please test the code on your computer and show/tell us the result?

[cdbc]

Hi
I took the liberty of rewriting you app, so now it works Ok.
Prerequisites:
The data you put in the memo, must be in a file in the app-dir, called 'input.txt'
Compiled with debug info, it runs in ~ 16 seconds
Compiled without debug info(release), it runs in ~ 5 seconds
Hope you don't mind...
Regards Benny

[cdbc]

Hi
I've sliced out 1/3 of the org input.txt, to get it under 500 kb, here it is
Regards Benny