Loop Iteration Speed Test

[Aruna]

Hi! Here's a simple example that measures how many iterations a given type of loop can execute in one second using the TDateTime functions. A fully working source zip is attached.

Between runs the number of iterations vary (Screenshots attached). I am trying to understand the reason why? Everything remaining the same should not the output be the same?

I was told by a friend "Aruna: your benchmark is flawed because calculating time offset is a part of the loop. It would make more sense to just loop until ALRM signal is caught"

So next question is can I use ALRM in a gui application? Or this is console only?

I am yet to test ALRM but if I see differences in the iterations between a console application and a gui application what could the possible reasons be please?

Sample output :
FOR..TO loop           :        1660000 iterations
WHILE..DO loop       :        1786679 iterations
REPEAT..UNTIL loop:        1767713 iterations
FOR..IN loop            :     558000000 iterations (array)

[ALLIGATOR]

Perhaps you should switch from measuring time to measuring the number of iterations.

In other words, instead of “How many iterations in 1000 milliseconds,” it would be “How many milliseconds in 1,000,000,000 iterations.”

Then everything will be fairer.

And, perhaps at the optimization level -O2 and above - all options (except, perhaps, for in) will be the same.

[ALLIGATOR]

I would like to share with you a benchmark that I once created for myself out of curiosity to determine the extent of the slowdown caused by for in and how much it can be reduced in an “ideal case”

Perhaps you will find it useful

Code: Pascal  [Select][+][-]

1. program test;
2. {$ifdef FPC}{$mode delphi}{$endif}
3. {$APPTYPE CONSOLE}
4. {$optimization ON}
5.  
6. {$DEFINE USE_INLINE}
7.  
8. uses
9.   SysUtils, Classes, DateUtils;
10.  
11. const
12.   N = 512*1024*1024;
13.  
14. type
15.   TEnumClass = class
16.     val: NativeInt;
17.     function GetCurrent: NativeInt; overload; {$IFDEF USE_INLINE}inline;{$ENDIF}
18.     function GetCurrent(i: NativeInt): NativeInt; overload; {$IFDEF USE_INLINE}inline;{$ENDIF}
19.     function MoveNext: Boolean; {$IFDEF USE_INLINE}inline;{$ENDIF}
20.     property Current: NativeInt read GetCurrent;
21.     function GetEnumerator: TEnumClass;
22.     property Get[i: NativeInt]: NativeInt read GetCurrent; default;
23.     function Count: NativeInt;
24.   end;
25.  
26. var
27.   timer: TDateTime;
28.   i, s: NativeInt;
29.   TE: TEnumClass;
30.  
31. function TEnumClass.GetCurrent: NativeInt;
32. begin
33.   Result:=val;
34. end;
35.  
36. function TEnumClass.GetCurrent(i: NativeInt): NativeInt;
37. begin
38.   Result:=i;
39. end;
40.  
41. function TEnumClass.MoveNext: Boolean;
42. begin
43.   Result := val < N;
44.   inc(val);
45. end;
46.  
47. function TEnumClass.GetEnumerator: TEnumClass;
48. begin
49.   Result:=Self;
50. end;
51.  
52. function TEnumClass.Count: NativeInt;
53. begin
54.   Result:=N;
55. end;
56.  
57. begin
58.   TE:=TEnumClass.Create;
59.   timer := Now;
60.   for i in TE do
61.   begin
62.     s := s + i;
63.   end;
64.   WriteLn('IEnumerable: ' + FormatDateTime('ss:zz', Now - timer));
65.  
66.   TE:=TEnumClass.Create;
67.   timer := Now;
68.   for i := 0 to TE.Count - 1 do
69.   begin
70.     s := s + TE[i];
71.   end;
72.   WriteLn('Iterator: ' + FormatDateTime('ss:zz', Now - timer));
73.  
74.   ReadLn;
75. end.

[paweld]

I have a similar opinion to @ALLIGATOR.
Additionally, you have an error when checking the for..in loop. In my case, after correction, the results are as follows:

Code: [Select]

FOR..TO loop:         15067401 iterations
WHILE..DO loop:       15115295 iterations
REPEAT..UNTIL loop:   15243195 iterations
FOR..IN loop:         14385271 iterations

[Martin_fr]

So the first 3 numbers are identical.

Unless you have a real time OS, and write the code to special rules for getting real time results, you will have variances. E.g. the OS scheduler may interrupt the code.

Now() is also not the most exact timer.

Then, your loops are not the same, you are comparing apples to oranges.

The for loop runs to "High(Int64). Well it may exit earlier. It has TWO exit conditions.

Code: Pascal  [Select][+][-]

1.   for Count := 1 to High(Int64) do
2.     if MilliSecondsBetween(Now, StartTime) >= 1000 then Break;
3.  

Code: Pascal  [Select][+][-]

1.   while MilliSecondsBetween(Now, StartTime) < 1000 do Inc(Count);
2.  

You don't check the "High(Int64)" exit condition. So this is completely different code that is run. Comparing the time to the "for" loop is meaningless.

Take the "MilliSecondsBetween(Now, StartTime) >= 1000" out of both loops, and the "for" loop will end, but the "while" loop will loop forever (overflowing Count eventually).


The "for in" has significantly less calls to "MilliSecondsBetween(Now, StartTime) >= 1000" => so of course it is faster. But then "A:=3.14;" is also significantly faster than "A := Round(Calculate1000DigitsOfPiFromScratch,2);".


Depending on your CPU there may be other factors.  With loops that small many modern INTEL/AMD cpu can be faster if the loop is aligned at a 32byte boundary. Putting the loops one after another may give them different alignments. But even putting each in a procedure of its own, will at best do 16byte alignment. You would need to
- specify the desired alignment
- benchmark with different settings

And, running them one by one,... Many cpu will have a "Boost" mode that lasts for a short time only. So some part of your code may run at significantly higher CPU clock that the rest.

In other words, benchmarking can be very complex.

Mind you, its possible that accounting for align and boost will get the same results. Or not...

But at least make the code for all loops "do the same work"

[Aruna]

Perhaps you should switch from measuring time to measuring the number of iterations.

In other words, instead of “How many iterations in 1000 milliseconds,” it would be “How many milliseconds in 1,000,000,000 iterations.”

Then everything will be fairer.

And, perhaps at the optimization level -O2 and above - all options (except, perhaps, for in) will be the same.

Hello @ALLIGATOR thank you for the suggestion and yes it makes a lot of sense. I have never used compiler optimizations (yet) never had a reason to so went with the defaults but I will explore this further now. I have attached a first attempt at what you suggested and to count to a billion my ancient system ( 2013 era) took 14-15 seconds.

[Aruna]

I would like to share with you a benchmark that I once created for myself out of curiosity to determine the extent of the slowdown caused by for in and how much it can be reduced in an “ideal case”

Perhaps you will find it useful

Thank you very much. Yes this will be helpful.

[Aruna]

I have a similar opinion to @ALLIGATOR.
Additionally, you have an error when checking the for..in loop. In my case, after correction, the results are as follows:

Code: [Select]

FOR..TO loop:         15067401 iterations
WHILE..DO loop:       15115295 iterations
REPEAT..UNTIL loop:   15243195 iterations
FOR..IN loop:         14385271 iterations

Hello @paweld what was the error? Please show me what I did wrong? And what you did to fix things please.

[Aruna]

So the first 3 numbers are identical.

Unless you have a real time OS, and write the code to special rules for getting real time results, you will have variances. E.g. the OS scheduler may interrupt the code.

Ah, now I am starting to understand. Thank you.

Now() is also not the most exact timer.

I always used Now() never thought of how accurate it is till now...

Then, your loops are not the same, you are comparing apples to oranges.

The for loop runs to "High(Int64). Well it may exit earlier. It has TWO exit conditions.

Code: Pascal  [Select][+][-]

1.   for Count := 1 to High(Int64) do
2.     if MilliSecondsBetween(Now, StartTime) >= 1000 then Break;
3.  

Hmm, I did not see that one or think of it until you pointed it out...

Code: Pascal  [Select][+][-]

1.   while MilliSecondsBetween(Now, StartTime) < 1000 do Inc(Count);
2.  

You don't check the "High(Int64)" exit condition. So this is completely different code that is run. Comparing the time to the "for" loop is meaningless.

Take the "MilliSecondsBetween(Now, StartTime) >= 1000" out of both loops, and the "for" loop will end, but the "while" loop will loop forever (overflowing Count eventually).

Alright so I have lots to learn

The "for in" has significantly less calls to "MilliSecondsBetween(Now, StartTime) >= 1000" => so of course it is faster. But then "A:=3.14;" is also significantly faster than "A := Round(Calculate1000DigitsOfPiFromScratch,2);".

LOL 

Depending on your CPU there may be other factors.  With loops that small many modern INTEL/AMD cpu can be faster if the loop is aligned at a 32byte boundary. Putting the loops one after another may give them different alignments. But even putting each in a procedure of its own, will at best do 16byte alignment. You would need to
- specify the desired alignment
- benchmark with different settings

Hey I did not know about this. Thank you for explaining all this. So if I put them into separate procedures then they get aligned ?

And, running them one by one,... Many cpu will have a "Boost" mode that lasts for a short time only. So some part of your code may run at significantly higher CPU clock that the rest.

In other words, benchmarking can be very complex.

Mind you, its possible that accounting for align and boost will get the same results. Or not...

But at least make the code for all loops "do the same work"

This was a first attempt at a very simple and basic benchmark tool for my personal use but very much appreciate all the explanations and pointers I have a question, what would be the best way to make them all do the same work? DO what @alligator and @paweld suggeted and use a count to 1 billion for all loops and measure the time?

[LV]

win64: 

Code: Text  [Select][+][-]

1. Loop timing for 1000000000 iterations
2. -------------------------------------------
3. FOR..TO loop time:         11175 ms
4. WHILE loop time:           11145 ms
5. REPEAT..UNTIL loop time:   11089 ms
6. FOR..IN loop (array):      11096 ms
7. Dummy result (ignore):     227722
8.  

win32: 

Code: Text  [Select][+][-]

1. Loop timing for 1000000000 iterations
2. -------------------------------------------
3. FOR..TO loop time:         7651 ms
4. WHILE loop time:           7563 ms
5. REPEAT..UNTIL loop time:   7592 ms
6. FOR..IN loop (array):      7581 ms
7. Dummy result (ignore):     227722
8.  

P.S. I replaced Int64 with Longint so that the test would run on a 32-bit system as well.

[LV]

If we move on to real problems, then for me, there is no question about choosing between a 32-bit or a 64-bit application. Now I have compiled and run a resource-intensive, multi-threaded application from the CFD field, which is written and optimized in pure Free Pascal. The application requires more than 10000 cycles to produce a result. Let's focus on the performance for 100 cycles:

- Windows 64-bit: 31 seconds 
- Windows 32-bit: 66 seconds   

[Aruna]

win64: 

Code: Text  [Select][+][-]

1. Loop timing for 1000000000 iterations
2. -------------------------------------------
3. FOR..TO loop time:         11175 ms
4. WHILE loop time:           11145 ms
5. REPEAT..UNTIL loop time:   11089 ms
6. FOR..IN loop (array):      11096 ms
7. Dummy result (ignore):     227722
8.  

win32: 

Code: Text  [Select][+][-]

1. Loop timing for 1000000000 iterations
2. -------------------------------------------
3. FOR..TO loop time:         7651 ms
4. WHILE loop time:           7563 ms
5. REPEAT..UNTIL loop time:   7592 ms
6. FOR..IN loop (array):      7581 ms
7. Dummy result (ignore):     227722
8.  

P.S. I replaced Int64 with Longint so that the test would run on a 32-bit system as well.

Hi @LV, this is really interesting. I would have thought the 64bit would have taken less time to count to 1 billion but obviously this is not the case? What would make a 32bit application faster than the same 64bit application? 

[ALLIGATOR]

Now() is also not the most exact timer.

I beg to differ. Now on Windows has a resolution of about 1 ms, while GetTickCount[64] has a resolution of about 15 ms.
Therefore, Now is a fairly accurate tool in this case.

[ALLIGATOR]

A simple way to check (again, this is only for Windows; I don't know how to do it on Linux):

Code: Pascal  [Select][+][-]

1. program app;
2. {$mode objfpc}
3.  
4. uses SysUtils;
5.  
6. var
7.   t1,t2: TDateTime;
8.   t3,t4: QWord;
9.   tickspersec: QWord;
10.  
11. begin
12.   t3:=GetTickCount64;
13.   Sleep(1000);
14.   tickspersec:=GetTickCount64-t3;
15.  
16.   t1:=Now;
17.   t2:=t1;
18.   while t2=t1 do t2:=Now;
19.   WriteLn('Now: ', (t2-t1)*MSecsPerDay:0:2);
20.  
21.   t3:=GetTickCount64;
22.   t4:=t3;
23.   while t4=t3 do t4:=GetTickCount64;
24.   WriteLn('GTC64: ', (t4-t3)*1000.0/tickspersec:0:2);
25.  
26.   ReadLn;
27. end.

Output (Win11):

Code: Pascal  [Select][+][-]

1. Now: 1.00
2. GTC64: 15.00

[Gustavo 'Gus' Carreno]

Hey ALLIGATOR,

Funny enough, this is what I get in my Ubuntu 25.04, Lazarus 4.99:

Code: Pascal  [Select][+][-]

1. Now: 1.00
2. GTC64: 1.00

Looks like it's a Windows quirk... Rather odd, I would say 
And I've run it 5 times, just to make sure no shenanigans were involved.

Cheers,
Gus