Recent

Author Topic: LGenerics: yet another generics collection  (Read 39340 times)

440bx

  • Hero Member
  • *****
  • Posts: 4486
Re: LGenerics: yet another generics collection
« Reply #105 on: December 24, 2022, 09:16:50 am »
@440bx, thank you very much for discovering this sad fact. Some time ago I excluded the TGOptional.OrElseDefault() method and forgot to fix it in this example. Now fixed.
My pleasure to be of some help. 
(FPC v3.0.4 and Lazarus 1.8.2) or (FPC v3.2.2 and Lazarus v3.2) on Windows 7 SP1 64bit.

zbyna

  • Jr. Member
  • **
  • Posts: 63
Re: LGenerics: yet another generics collection
« Reply #106 on: December 24, 2022, 05:34:09 pm »
Hi!
I would like to inform you that a unit has been added to LGenerics for exporting Pascal data structures to JSON format.
...

The performance should be noticeably better than in FpJsonRtti, but I would like to come back to that a little later.

Merry Christmas!

Thanks for great present. Merry Christmas!

Support for https://msgpack.org/ would be great too.

avk

  • Hero Member
  • *****
  • Posts: 756
Re: LGenerics: yet another generics collection
« Reply #107 on: December 25, 2022, 06:13:18 am »
Support for MessagePack was planned, but this is unlikely to happen anytime soon.

As promised earlier, let's try to roughly estimate the performance of export to JSON.
Datasets used: a collection of simple items(with three string properties and one integer property) and an array of records whose fields contain values that coincide with the values of the collection's item properties(i.e. the resulting JSONs should be the same). The number of elements in the datasets was chosen so that the size of the resulting JSON approximately corresponded to the values (20KB, 100KB, 1MB, 10MB, 50MB).
TJSONStreamer.CollectionToJSON() was taken as a reference implementation(FpJsonStreamer for short), other members:
  - PdoToJson() with a collection as parameter(PdoCollection);
  - PdoToJson() with array as parameter and field name list registration(PdoRecordFieldMap);
  - PdoToJson() with array as parameter and custom callback registration(PdoRecordCallback);
Each member exports the data several times(number of attempts is pre-defined), the best time is taken as the result.
Full benchmark code(Windows only, uses QueryPerformanceCounter):
Code: Pascal  [Select][+][-]
  1. program test;
  2.  
  3. {$MODE OBJFPC}{$H+}{$MODESWITCH ADVANCEDRECORDS}
  4. {$OPTIMIZATION NOORDERFIELDS}
  5.  
  6. uses
  7.   Windows, Classes, SysUtils, lgPdo, FpJsonRtti, lgJson;
  8.  
  9. const
  10.   AlphaLen = 64;
  11.   Alphabet: array[0..AlphaLen-1] of Char = '1234567890ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz_ ';
  12.  
  13. function RandomString(aLength: SizeInt): string;
  14. var
  15.   I: SizeInt;
  16.   p: PChar;
  17. begin
  18.   Result := '';
  19.   SetLength(Result, aLength);
  20.   p := PChar(Result);
  21.   for I := 0 to Pred(aLength) do
  22.     p[I] := Alphabet[Random(AlphaLen)];
  23. end;
  24.  
  25. type
  26.   TMyObj = class(TCollectionItem)
  27.   private
  28.     FId: Int64;
  29.     FName,
  30.     FValue,
  31.     FInfo: string;
  32.   published
  33.     property id: Int64 read FId write FId;
  34.     property name: string read FName write FName;
  35.     property value: string read FValue write FValue;
  36.     property info: string read FInfo write FInfo;
  37.   end;
  38.  
  39.   TMyRec = record
  40.     Id: Int64;
  41.     Name,
  42.     Value,
  43.     Info: string;
  44.     class procedure JsonWrite(r: Pointer; aWriter: TJsonStrWriter); static;
  45.   end;
  46.  
  47. class procedure TMyRec.JsonWrite(r: Pointer; aWriter: TJsonStrWriter);
  48. type
  49.   PMyRec = ^TMyRec;
  50. const
  51.   fnId    = 'id';
  52.   fnName  = 'name';
  53.   fnValue = 'value';
  54.   fnInfo  = 'info';
  55. begin
  56.   with PMyRec(r)^ do
  57.     aWriter
  58.       .BeginObject
  59.         .Add(fnId, Id)
  60.         .Add(fnName, Name)
  61.         .Add(fnValue, Value)
  62.         .Add(fnInfo, Info)
  63.       .EndObject;
  64. end;
  65.  
  66. const
  67.   MinLen = 5;
  68.   Range  = 16;
  69.  
  70. var
  71.   TestColl: TCollection = nil;
  72.   RecList: array of TMyRec;
  73.  
  74. procedure UpdateData(aElCount: Integer);
  75. var
  76.   I, OldSize: Integer;
  77. const
  78.   IdRange = 9007199254740991;
  79. begin
  80.   if TestColl = nil then
  81.     TestColl := TCollection.Create(TMyObj);
  82.   OldSize := TestColl.Count;
  83.   SetLength(RecList, aElCount);
  84.   for I := OldSize to Pred(aElCount) do
  85.     with TMyObj(TestColl.Add) do begin
  86.       Id   := Random(IdRange);
  87.       Name := RandomString(MinLen + Random(Range));
  88.       Value := RandomString(MinLen + Random(Range));
  89.       Info := RandomString(MinLen + Random(Range));
  90.       RecList[I].Id := Id;
  91.       RecList[I].Name := Name;
  92.       RecList[I].Value := Value;
  93.       RecList[I].Info := Info;
  94.     end;
  95. end;
  96.  
  97. function JsonStreamer: string;
  98. begin
  99.   with TJSONStreamer.Create(nil) do
  100.     try
  101.       Result := CollectionToJSON(TestColl);
  102.     finally
  103.       Free;
  104.     end;
  105. end;
  106.  
  107. function PdoCollection: string;
  108. begin
  109.   Result := PdoToJson(TypeInfo(TestColl), TestColl);
  110. end;
  111.  
  112. function PdoMap: string;
  113. begin
  114.   Result := PdoToJson(TypeInfo(RecList), RecList);
  115. end;
  116.  
  117. function PdoProc: string;
  118. begin
  119.   Result := PdoToJson(TypeInfo(RecList), RecList);
  120. end;
  121.  
  122. procedure RegisterFields;
  123. begin
  124.   RegisterRecordFields(TypeInfo(TMyRec), ['id','name','value','info']);
  125. end;
  126.  
  127. procedure RegisterCallback;
  128. begin
  129.   RegisterRecordJsonProc(TypeInfo(TMyRec), @TMyRec.JsonWrite);
  130. end;
  131.  
  132. procedure UnRegister;
  133. begin
  134.   UnRegisterPdo(TypeInfo(TMyRec));
  135. end;
  136.  
  137. type
  138.   TJsonFun = function: string;
  139.  
  140.   TMember = record
  141.     Fun: TJsonFun;
  142.     Name: string;
  143.     BeforeRun,
  144.     AfterRun: TProcedure;
  145.   end;
  146.  
  147.   TRound = record
  148.     ElCount,
  149.     TryCount: Integer;
  150.     JsonSize: string;
  151.   end;
  152.  
  153. const
  154.   MemberList: array of TMember = (
  155.     (Fun: @JsonStreamer;  Name: 'FpJsonStreamer......'; BeforeRun: nil; AfterRun: nil),
  156.     (Fun: @PdoCollection; Name: 'PdoCollection.......'; BeforeRun: nil; AfterRun: nil),
  157.     (Fun: @PdoMap;        Name: 'PdoRecordFieldMap...'; BeforeRun: @RegisterFields; AfterRun: @UnRegister),
  158.     (Fun: @PdoProc;       Name: 'PdoRecordCallback...'; BeforeRun: @RegisterCallback; AfterRun: @UnRegister));
  159.  
  160.   Rounds: array of TRound = (
  161.     (ElCount: 220;    TryCount: 100; JsonSize: '20 KB'),
  162.     (ElCount: 1091;   TryCount:  50; JsonSize: '100 KB'),
  163.     (ElCount: 10830;  TryCount:  10; JsonSize: '1 MB'),
  164.     (ElCount: 108400; TryCount:   5; JsonSize: '10 MB'),
  165.     (ElCount: 542000; TryCount:   1; JsonSize: '50 MB')
  166.   );
  167.  
  168. procedure Run;
  169. var
  170.   CurrRound: TRound;
  171.   Member: TMember;
  172.   I: Integer;
  173.   Start, Fin, Freq, Best: Int64;
  174.   Elapsed, Rate: Double;
  175.   s: string = '';
  176. const
  177.   Million = 1000000;
  178. begin
  179.   if not QueryPerformanceFrequency(Freq) or (Freq = 0) then
  180.     begin
  181.       WriteLn('Oops, something went wrong with QueryPerformanceFrequency()');
  182.       WriteLn('Error: ', GetLastOsError);
  183.       exit;
  184.     end;
  185.   for CurrRound in Rounds do begin
  186.     UpdateData(CurrRound.ElCount);
  187.     WriteLn('-------<JSON size ', CurrRound.JsonSize, '>-------');
  188.     for Member in MemberList do begin
  189.       if Member.BeforeRun <> nil then
  190.         Member.BeforeRun();
  191.       Best := High(Int64);
  192.       for I := 1 to CurrRound.TryCount do begin
  193.         QueryPerformanceCounter(Start);
  194.         s := Member.Fun();
  195.         QueryPerformanceCounter(Fin);
  196.         if Fin - Start < Best then
  197.           Best := Fin - Start;
  198.       end;
  199.       if Member.AfterRun <> nil then
  200.         Member.AfterRun();
  201.       Elapsed := Double(Best)/Freq;
  202.       WriteLn('  ', Member.Name, Double2Str(Double(Round(Elapsed*Million))/1000), ' ms.');
  203.       Rate := (Double(Length(s))/Million)/Elapsed;
  204.       WriteLn('  Rating:             ', Double2Str(Double(Round(Rate*100))/100), ' MB/s.');
  205.     end;
  206.   end;
  207. end;
  208.  
  209. begin
  210.   Run;
  211.   TestColl.Free;
  212. end.
  213.  

Results on my machine:
Code: Text  [Select][+][-]
  1. -------<JSON size 20 KB>-------
  2.   FpJsonStreamer......1.139 ms.
  3.   Rating:             20.41 MB/s.
  4.   PdoCollection.......0.135 ms.
  5.   Rating:             149.47 MB/s.
  6.   PdoRecordFieldMap...0.095 ms.
  7.   Rating:             212.01 MB/s.
  8.   PdoRecordCallback...0.06 ms.
  9.   Rating:             336.12 MB/s.
  10. -------<JSON size 100 KB>-------
  11.   FpJsonStreamer......6.171 ms.
  12.   Rating:             18.76 MB/s.
  13.   PdoCollection.......0.675 ms.
  14.   Rating:             148.94 MB/s.
  15.   PdoRecordFieldMap...0.475 ms.
  16.   Rating:             211.85 MB/s.
  17.   PdoRecordCallback...0.304 ms.
  18.   Rating:             331.03 MB/s.
  19. -------<JSON size 1 MB>-------
  20.   FpJsonStreamer......88.315 ms.
  21.   Rating:             13.04 MB/s.
  22.   PdoCollection.......7.062 ms.
  23.   Rating:             141.6 MB/s.
  24.   PdoRecordFieldMap...5.018 ms.
  25.   Rating:             199.27 MB/s.
  26.   PdoRecordCallback...3.317 ms.
  27.   Rating:             301.47 MB/s.
  28. -------<JSON size 10 MB>-------
  29.   FpJsonStreamer......3736.737 ms.
  30.   Rating:             3.08 MB/s.
  31.   PdoCollection.......76.899 ms.
  32.   Rating:             130.16 MB/s.
  33.   PdoRecordFieldMap...56.989 ms.
  34.   Rating:             175.63 MB/s.
  35.   PdoRecordCallback...40.309 ms.
  36.   Rating:             248.3 MB/s.
  37. -------<JSON size 50 MB>-------
  38.   FpJsonStreamer......86500.116 ms.
  39.   Rating:             0.67 MB/s.
  40.   PdoCollection.......379.533 ms.
  41.   Rating:             131.94 MB/s.
  42.   PdoRecordFieldMap...278.097 ms.
  43.   Rating:             180.07 MB/s.
  44.   PdoRecordCallback...192.341 ms.
  45.   Rating:             260.35 MB/s.
  46.  

It seems to work pretty fast.

AlexTP

  • Hero Member
  • *****
  • Posts: 2457
    • UVviewsoft
Re: LGenerics: yet another generics collection
« Reply #108 on: December 25, 2022, 12:18:19 pm »
« Last Edit: December 25, 2022, 12:22:51 pm by AlexTP »

avk

  • Hero Member
  • *****
  • Posts: 756
Re: LGenerics: yet another generics collection
« Reply #109 on: December 25, 2022, 01:02:21 pm »
Honestly, it would be much more interesting to see these numbers on other devices.

avk

  • Hero Member
  • *****
  • Posts: 756
Re: LGenerics: yet another generics collection
« Reply #110 on: December 25, 2022, 05:01:38 pm »
I guess we can end this pointless thread.

 

TinyPortal © 2005-2018