Recent

Author Topic: Programming language comparison by implementing the same transit data app  (Read 6542 times)

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #45 on: November 26, 2022, 09:21:37 am »
@Leledumbo
Thanks for the numbers!  ;D

During load test, there are in fact a lot of memory allocated during the response computation - so no GC, just the response array and the JSON cooked in memory. A ** request with a static response consumes no more memory (just some minimal information per connection).

About the static files, you are right, there are none needed for darwin-aarch64 IIRC.
To be fair this target was not well tested yet, since I don't have the HW. ;) Nor optimized yet: it uses pascal code everywhere.
Only linux-aarch64 has been validated, with some optimized asm for AES and hashing - my guess is that those binary is likely to work on darwin-aarch64 also, but I could not test it. And aarch64 is supported good enough by FPC - **s://blog.synopse.info/?post/2021/08/17/mORMot-2-on-Ampere-AARM64-CPU  8-)

It could be good to add memory consumption numbers to your readme file, for reference.
Memory consumption is of great interest for real server/business work, even if GB of RAM is somewhat cheap these days - but not for Apple M1  O:-)

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #46 on: November 26, 2022, 10:01:44 am »
@FredvS
I never tested fpc-llvm to be fair.
I usually use fpcupdeluxe to setup my environment, and llvm is not supported by it, IIRC.

Thaddy

  • Hero Member
  • *****
  • Posts: 12573
Re: Programming language comparison by implementing the same transit data app
« Reply #47 on: November 26, 2022, 10:11:26 am »
Freepascal has always tested good values for memory consumption. That is nothing new.
Writing code in an overly complex way is usually merely a complete misunderstanding of the goal you are trying to achieve.

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #48 on: November 26, 2022, 10:50:28 am »
Freepascal has always tested good values for memory consumption. That is nothing new.

The original FreePascal version by Leledumbo consumed much more memory.

Here good memory values comes not only from pascal, but the algorithms we used, and the mORMot way of handling the data.
We are dealing about 10 times less memory for storing the data in respect to Go, and 5 times less memory when serving the requests.https://forum.lazarus.freepascal.org/index.php/topic,61276.msg461380.html#msg461380

Fred vS

  • Hero Member
  • *****
  • Posts: 2898
    • StrumPract is the musicians best friend
Re: Programming language comparison by implementing the same transit data app
« Reply #49 on: November 26, 2022, 01:01:44 pm »
@FredvS
I never tested fpc-llvm to be fair.
I usually use fpcupdeluxe to setup my environment, and llvm is not supported by it, IIRC.

Hello.https://forum.lazarus.freepascal.org/index.php/topic,61069.msg459333.html#msg459333
I use Lazarus 2.2.0 32/64 and FPC 3.2.2 32/64 on Debian 11 64 bit, Windows 10, Windows 7 32/64, Windows XP 32,  FreeBSD 64.
Widgetset: fpGUI, MSEgui, Win32, GTK2, Qt.
https://github.com/fredvshttps://gitlab.com/fredvs
**s://codeberg.org/fredvs

Leledumbo

  • Hero Member
  • *****
  • Posts: 8523
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Programming language comparison by implementing the same transit data app
« Reply #50 on: November 28, 2022, 02:54:33 am »
The original FreePascal version by Leledumbo consumed much more memory.
True. A good example of how the convenience of object orientation, reference counted strings and generics can massively contribute to memory consumption as opposed to plain dynamic arrays (with clever growth), pchars and manual memory (de)allocation.
Follow this if you want me to answer: **://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

**://pascalgeek.blogspot.com
**s://bitbucket.org/leledumbohttps://github.com/leledumbo
Code first, think later - Natural programmer B)

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #51 on: November 28, 2022, 11:52:34 am »
@Leledumbo
Perhaps you may add my last blog article in the ReadMe:
**s://blog.synopse.info/?post/2022/11/26/Modern-Pascal-is-Still-in-the-Race

PierceNg

  • Sr. Member
  • ****
  • Posts: 287
    • SamadhiWeb
Re: Programming language comparison by implementing the same transit data app
« Reply #52 on: November 28, 2022, 12:57:03 pm »
Kudos to @Leledumbo and @abouchez for working on this.

For others who want to run this stuff on your machines, you need the k6 tool from https://github.com/grafana/k6.

Repos:

    https://github.com/losvedir/transit-lang-cmp[/li]https://github.com/leledumbo/transit-lang-cmp[/li]https://github.com/synopse/mORMot2/[/li]
    [/list]

    Steps:

    Code: Text  [Select][+][-]
    1. % git clone original-repo-or-fork
    2. % cd transit-lang-cmp
    3. % curl -o MBTA_GTFS.zip **s://cdn.mbta.com/MBTA_GTFS.zip
    4. % unzip -d MBTA_GTFS MBTA_GTFS.zip

    Build and run the implementation you want to try. In another terminal window, run
    Code: Text  [Select][+][-]
    1. k6 run loadTest.js
    and/or
    Code: Text  [Select][+][-]
    1. k6 run loadTestSmallResponses.js

    For abouchez's repo, the code is in the subdirectory ex/lang-cmp/.

    Edit: Rust's compilation time is terrible.
« Last Edit: November 28, 2022, 01:05:38 pm by PierceNg »
Blog: **s://samadhiweb.com/
Github: https://github.com/PierceNg

Pascal:
- **s://samadhiweb.com/tags/Pascal.html
- https://github.com/PierceNg/fpwebview
- https://github.com/PierceNg/wasm-demo

PierceNg

  • Sr. Member
  • ****
  • Posts: 287
    • SamadhiWeb
Re: Programming language comparison by implementing the same transit data app
« Reply #53 on: November 28, 2022, 04:04:57 pm »
Numbers on my laptop, using whatever versions of compilers I happened to have installed, not the latest but not very outdated either, no attempt to 'control the environment':
  • Rust ran the fastest
  • Pascal mORMot version and Go comparable
  • .NET 6.0 C# came fourth
  • Deno TypeScript
Don't have Scala.

As for Leledumbo's version, fp**app is the bottleneck.
Blog: **s://samadhiweb.com/
Github: https://github.com/PierceNg

Pascal:
- **s://samadhiweb.com/tags/Pascal.html
- https://github.com/PierceNg/fpwebview
- https://github.com/PierceNg/wasm-demo

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #54 on: November 28, 2022, 05:27:04 pm »
@PierceNg

Nice to read!

Perhaps you may try to get raw numbers also with wrk as tool:
Code: [Select]
wrk -c 50 -d 15s -t 4 **://localhost:4000/schedules/354Then try with -c 500 and perhaps -c 5000 (with proper ulimit set if needed). To see how they scale with a high number of connections.
I don't know very much k6, but wrk has been proven to be very low-resource consuming.

Perhaps also comparing top memory use by each language.

Thanks a lot for the feedback!

fcu

  • Jr. Member
  • **
  • Posts: 80
Re: Programming language comparison by implementing the same transit data app
« Reply #55 on: November 28, 2022, 09:12:30 pm »
good job Arnaud
i remember watching this video ( **s://youtu.be/Z5hxDviE_DM ) which benchmarking mormot with other library , it shows that mormot is the fastest . 

Leledumbo

  • Hero Member
  • *****
  • Posts: 8523
  • Programming + Glam Metal + Tae Kwon Do = Me
Re: Programming language comparison by implementing the same transit data app
« Reply #56 on: November 29, 2022, 05:21:26 am »
@Leledumbo
Perhaps you may add my last blog article in the ReadMe:
**s://blog.synopse.info/?post/2022/11/26/Modern-Pascal-is-Still-in-the-Race
OK
Numbers on my laptop, using whatever versions of compilers I happened to have installed, not the latest but not very outdated either, no attempt to 'control the environment':
Rust ran the fastest
LLVM magic is hard to beat, indeed.
As for Leledumbo's version, fp**app is the bottleneck.
Acknowledged, but I can't figure out how to make it faster. Michael patched it with exception handling heavily to solve stability issue with it way back then, I suppose it will need a proper rewrite if performance has become a target.
Follow this if you want me to answer: **://wiki.lazarus.freepascal.org/Lazarus_Faq#What_is_the_correct_way_to_ask_questions_in_the_forum.3F

**://pascalgeek.blogspot.com
**s://bitbucket.org/leledumbohttps://github.com/leledumbo
Code first, think later - Natural programmer B)

avk

  • Hero Member
  • *****
  • Posts: 686
Re: Programming language comparison by implementing the same transit data app
« Reply #57 on: December 11, 2022, 03:14:03 pm »
<...>
As for Leledumbo's version, fp**app is the bottleneck.
Acknowledged, but I can't figure out how to make it faster. Michael patched it with exception handling heavily to solve stability issue with it way back then, I suppose it will need a proper rewrite if performance has become a target.

I wondered what contribution response building and serialization made to the overall problem.
So I replaced all these things with something like this:
Code: Pascal  [Select][+][-]
  1. program app1;
  2.  
  3. {$mode objfpc}{$H+}
  4. {$modeswitch advancedrecords}
  5. {$modeswitch typehelpers}
  6.  
  7. uses
  8.   Classes,
  9.   SysUtils,
  10.   DateUtils,
  11.   lgUtils,
  12.   lgHashMap,
  13.   lgVector,
  14.   lgJson,
  15.   lgCsvUtils,
  16.   fp**app,
  17.   **defs,
  18.   **route;
  19.  
  20. type
  21.  
  22.   TStopTime = record
  23.     stop_id,
  24.     arrival_time,
  25.     departure_time: string;
  26.     constructor Make(const aStopID, aArrival, aDeparture: string);
  27.     procedure WriteJson(aWriter: TJsonWriter);
  28.   end;
  29.  
  30.   TStopTimeList = specialize TGVector<TStopTime>;
  31.   TStopTimeMap  = specialize TGObjHashMapLP<string, TStopTimeList>;
  32.  
  33.   TTrip = record
  34.     trip_id,
  35.     service_id,
  36.     route_id: string;
  37.     constructor Make(const aTripID, aRouteID, aServiceID: string);
  38.   end;
  39.  
  40.   TTripList = specialize TGVector<TTrip>;
  41.   TTripMap  = specialize TGObjHashMapLP<string, TTripList>;
  42.  
  43.   TTripResponse = record
  44.     trip_id,
  45.     service_id,
  46.     route_id: string;
  47.     schedules: TStopTimeList;
  48.     constructor Make(const aTrip: TTrip);
  49.     procedure WriteJson(aWriter: TJsonWriter);
  50.   end;
  51.   TTripResponses = array of TTripResponse;
  52.  
  53.   TTrHelper = type helper for TTripResponses
  54.     procedure WriteJson(aWriter: TJsonWriter);
  55.   end;
  56.  
  57. { TStopTime }
  58.  
  59. constructor TStopTime.Make(const aStopID, aArrival, aDeparture: string);
  60. begin
  61.   stop_id := aStopID;
  62.   arrival_time := aArrival;
  63.   departure_time := aDeparture;
  64. end;
  65.  
  66. procedure TStopTime.WriteJson(aWriter: TJsonWriter);
  67. begin
  68.   aWriter
  69.     .BeginObject
  70.       .Add('stop_id', stop_id)
  71.       .Add('arrival_time', arrival_time)
  72.       .Add('departure_time', departure_time)
  73.     .EndObject;
  74. end;
  75.  
  76. { TTrip }
  77.  
  78. constructor TTrip.Make(const aTripID, aRouteID, aServiceID: string);
  79. begin
  80.   trip_id := aTripID;
  81.   route_id := aRouteID;
  82.   service_id := aServiceID;
  83. end;
  84.  
  85. { TTripResponse }
  86.  
  87. constructor TTripResponse.Make(const aTrip: TTrip);
  88. begin
  89.   trip_id := aTrip.trip_id;
  90.   service_id := aTrip.service_id;
  91.   route_id := aTrip.route_id;
  92.   schedules := nil;
  93. end;
  94.  
  95. procedure TTripResponse.WriteJson(aWriter: TJsonWriter);
  96. var
  97.   I: SizeInt;
  98. begin
  99.   aWriter.BeginObject
  100.     .Add('trip_id', trip_id)
  101.     .Add('service_id', service_id)
  102.     .Add('route_id', route_id)
  103.     .AddName('schedules')
  104.     .BeginArray;
  105.     if schedules <> nil then
  106.       for I := 0 to Pred(schedules.Count) do
  107.         schedules.UncMutable[I]^.WriteJson(aWriter);
  108.     aWriter.EndArray;
  109.   aWriter.EndObject;
  110. end;
  111.  
  112. { TTrHelper }
  113.  
  114. procedure TTrHelper.WriteJson(aWriter: TJsonWriter);
  115. var
  116.   I: SizeInt;
  117. begin
  118.   aWriter.BeginArray;
  119.   for I := 0 to High(Self) do
  120.     Self[I].WriteJson(aWriter);
  121.   aWriter.EndArray;
  122. end;
  123.  
  124. function BuildTripResponse(const ARoute: string; const AStopTimesIxByTrip: TStopTimeMap;
  125.   const ATripsIxByRoute: TTripMap): TTripResponses;
  126. var
  127.   LTrips: TTripList;
  128.   LStopTimes: TStopTimeList;
  129.   LTripIx: Integer;
  130.   LTripResponse: TTripResponse;
  131. begin
  132.   Result := nil;
  133.   if ATripsIxByRoute.TryGetValue(ARoute, LTrips) then begin
  134.     SetLength(Result, LTrips.Count);
  135.     for LTripIx := 0 to Pred(LTrips.Count) do begin
  136.       LTripResponse := TTripResponse.Make(LTrips.UncMutable[LTripIx]^);
  137.       if AStopTimesIxByTrip.TryGetValue(LTripResponse.trip_id, LStopTimes) then
  138.         LTripResponse.schedules := LStopTimes;
  139.       Result[LTripIx] := LTripResponse;
  140.     end;
  141.   end;
  142. end;
  143.  
  144. procedure GetStopTimes(var AStopTimesIxByTrip: TStopTimeMap);
  145. var
  146.   CsvRef: specialize TGAutoRef<TCsvDoc>;
  147.   LCSV: TCsvDoc;
  148.   LStart,LEnd: TDateTime;
  149.   i: Integer;
  150.   LStopTimesIx: ^TStopTimeList;
  151.   p: TCsvDoc.PRow;
  152. begin
  153.   LStart := Now;
  154.   LCSV := CsvRef.Instance.LoadFromFile('../MBTA_GTFS/stop_times.txt', [true, true, true, true]);
  155.   if (LCSV[0, 0] <> 'trip_id') or (LCSV[0, 3] <> 'stop_id') or (LCSV[0, 1] <> 'arrival_time') or (LCSV[0, 2] <> 'departure_time') then begin
  156.     WriteLn('stop_times.txt not in expected format:');
  157.     for i := 0 to LCSV.ColCount[0] - 1 do begin
  158.       WriteLn(i, ' ' + LCSV[0, i]);
  159.     end;
  160.     Halt(1);
  161.   end;
  162.   AStopTimesIxByTrip := TStopTimeMap.Create([moOwnsValues]);
  163.   for i := 1 to LCSV.RowCount - 1 do begin
  164.     p := LCSV.Rows[i];
  165.     if not AStopTimesIxByTrip.FindOrAddMutValue(p^[0], LStopTimesIx) then
  166.       LStopTimesIx^ := TStopTimeList.Create;
  167.     LStopTimesIx^.Add(TStopTime.Make(p^[3], p^[1], p^[2]));
  168.   end;
  169.   LEnd := Now;
  170.   WriteLn('parsed ', LCSV.RowCount, ' stop times in ', SecondSpan(LStart, LEnd):1:3,' seconds');
  171. end;
  172.  
  173. procedure GetTrips(var ATripsIxByRoute: TTripMap);
  174. var
  175.   CsvRef: specialize TGAutoRef<TCsvDoc>;
  176.   LCSV: TCsvDoc;
  177.   LStart,LEnd: TDateTime;
  178.   i: Integer;
  179.   LTripsIx: ^TTripList;
  180.   LRoute: string;
  181.   p: TCsvDoc.PRow;
  182. begin
  183.   LStart := Now;
  184.   LCSV := CsvRef.Instance.LoadFromFile('../MBTA_GTFS/trips.txt', [true, true, true]);
  185.   if (LCSV[0, 2] <> 'trip_id') or (LCSV[0, 0] <> 'route_id') or (LCSV[0, 1] <> 'service_id') then begin
  186.     WriteLn('trips.txt not in expected format:');
  187.     for i := 0 to LCSV.ColCount[0] - 1 do begin
  188.       WriteLn(i, ' ' + LCSV[0, 1]);
  189.     end;
  190.     Halt(1);
  191.   end;
  192.   ATripsIxByRoute := TTripMap.Create([moOwnsValues]);
  193.   for i := 1 to LCSV.RowCount - 1 do begin
  194.     p := LCSV.Rows[i];
  195.     LRoute := p^[0];
  196.     if not ATripsIxByRoute.FindOrAddMutValue(LRoute, LTripsIx) then
  197.       LTripsIx^ := TTripList.Create;
  198.     LTripsIx^.Add(TTrip.Make(p^[2], LRoute, p^[1]));
  199.   end;
  200.   LEnd := Now;
  201.   WriteLn('parsed ', LCSV.RowCount, ' trips in ', SecondSpan(LStart, LEnd):1:3,' seconds');
  202. end;
  203.  
  204. var
  205.   GStopTimesIxByTrip: TStopTimeMap = nil;
  206.   GTripsIxByRoute: TTripMap = nil;
  207.  
  208. procedure SchedulesHandler(ARequest: TRequest; AResponse: TResponse);
  209. var
  210.   LTripResponses: TTripResponses;
  211.   LStringStream: specialize TGAutoRef<TStringStream>;
  212.   LWriter: specialize TGUniqRef<TJsonWriter>;
  213. begin
  214.   LTripResponses := BuildTripResponse(ARequest.RouteParams['route'], GStopTimesIxByTrip, GTripsIxByRoute);
  215.   {%H-}LWriter.Instance := TJsonWriter.New(LStringStream.Instance);
  216.   LTripResponses.WriteJson(LWriter.Instance);
  217.   LWriter.Clear;
  218.   AResponse.ContentType := 'application/json';
  219.   AResponse.Content := LStringStream.Instance.DataString;
  220. end;
  221.  
  222. procedure StopHandler(ARequest: TRequest; AResponse: TResponse);
  223. begin
  224.   GStopTimesIxByTrip.Free;
  225.   GTripsIxByRoute.Free;
  226.   Application.Terminate;
  227. end;
  228.  
  229. begin
  230.   GetStopTimes(GStopTimesIxByTrip);
  231.   GetTrips(GTripsIxByRoute);
  232.  
  233.   Application.Port := 4000;
  234.   Application.Threaded := True;
  235.   **Router.RegisterRoute('/schedules/:route',@SchedulesHandler);
  236.   **Router.RegisterRoute('/stop', @StopHandler);
  237.   Application.Run;
  238. end.
  239.  

Results on my machine, the left column is the results of the current version of app the right column is the results of the modified version:
Code: Text  [Select][+][-]
  1. running (1m28.0s), 00/25 VUs, 25 complete and 0 inter | running (1m08.2s), 00/25 VUs, 125 complete and 0 interrupted iterations
  2. default ✓ [ 100% ] 25 VUs  1m0s                         default ✓ [ 100% ] 25 VUs  1m0s
  3.                                                       |
  4. data_received..................: 2.3 GB 26 MB/s       | data_received..................: 11 GB  155 MB/s
  5. data_sent......................: 285 kB 3.2 kB/s      | data_sent......................: 1.6 MB 23 kB/s
  6. **_req_blocked...............: avg=28.28ms  min=0s  | **_req_blocked...............: avg=2.57ms   min=0s
  7. **_req_connecting............: avg=17.96ms  min=0s  | **_req_connecting............: avg=1.71ms   min=0s
  8. **_req_duration..............: avg=839.07ms min=1ms | **_req_duration..............: avg=134.26ms min=0s
  9.   { expected_response:true }...: avg=839.07ms min=1ms |   { expected_response:true }...: avg=134.26ms min=0s
  10. **_req_failed................: 0.00%  ✓ 0             **_req_failed................: 0.00%  ✓ 0
  11. **_req_receiving.............: avg=133.48ms min=0s  | **_req_receiving.............: avg=8.76ms   min=0s
  12. **_req_sending...............: avg=26.31ms  min=0s  | **_req_sending...............: avg=1.35ms   min=0s
  13. **_req_tls_handshaking.......: avg=0s       min=0s  | **_req_tls_handshaking.......: avg=0s       min=0s
  14. **_req_waiting...............: avg=679.28ms min=0s  | **_req_waiting...............: avg=124.14ms min=0s
  15. **_reqs......................: 2475   28.13618/s    | **_reqs......................: 12375  181.409315/s
  16. iteration_duration.............: avg=1m25s    min=1m22| iteration_duration.............: avg=13.5s    min=11.62
  17. iterations.....................: 25     0.284204/s    | iterations.....................: 125    1.832417/s
  18. vus............................: 9      min=9      max| vus............................: 8      min=8
  19. vus_max........................: 25     min=25     max| vus_max........................: 25     min=25
  20.  

It looks like the problem is not just with fp**app?

abouchez

  • Jr. Member
  • **
  • Posts: 78
    • Synopse
Re: Programming language comparison by implementing the same transit data app
« Reply #58 on: December 11, 2022, 07:42:10 pm »
It looks like the problem is not just with fp**app?
No, you are right, I don't think the web server is the issue with the Leledumbo's version.
Switching to mORMot Web Server did not make the code much better. Whereas the same mORMot Web Server with the mORMot JSON serialization is way faster - as fast as Go, i.e. reaching 1GB/s.

IMHO the JSON serialization code is clearly unoptimized and not scaling.
For instance, there are some manual concatenations, and a lot of objects creation, which makes it very slow.

avk

  • Hero Member
  • *****
  • Posts: 686
Re: Programming language comparison by implementing the same transit data app
« Reply #59 on: December 13, 2022, 01:13:21 pm »
<...>
Switching to mORMot Web Server did not make the code much better. Whereas the same mORMot Web Server with the mORMot JSON serialization is way faster - as fast as Go, i.e. reaching 1GB/s.
<...>

Oh, I have a pretty old budget computer, all the results look noticeably more modest on it.

I improved the performance of TJsonWriter a bit and at the same time tried to get rid of redundant entities in the application code, which now looks like this:
Code: Pascal  [Select][+][-]
  1. program app2;
  2.  
  3. {$mode objfpc}{$h+}{$modeswitch advancedrecords}{$modeswitch typehelpers}
  4.  
  5. uses
  6. {$ifdef UNIX}
  7.   cthreads,
  8. {$endif}
  9.   Classes, SysUtils, DateUtils, Math, Fp**App, **Defs, **Route,
  10.   lgUtils, lgHashMap, lgVector, lgJson, lgCsvUtils;
  11.  
  12. type
  13.   TStopTimeCols   = (stcTripId, stcArrival, stcDeparture, stcStopId);
  14.   TStopTimeFields = stcArrival..stcStopId;
  15.   TTripCols       = (tcRouteId, tcService, tcTripId);
  16.  
  17. {$push}{$J-}
  18. const
  19.   StopNames:    array[TStopTimeCols] of string = ('trip_id', 'arrival_time', 'departure_time', 'stop_id');
  20.   TripNames:    array[TTripCols] of string = ('route_id', 'service_id', 'trip_id');
  21.   StopTimeMask: array[TStopTimeCols] of Boolean = (True, True, True, True);
  22.   TripMask:     array[TTripCols] of Boolean = (True, True, True);
  23. {$pop}
  24.   SchedName = 'schedules';
  25.  
  26. type
  27.   PRow        = TCsvDoc.PRow;
  28.   TRowList    = specialize TGLiteVector<PRow>;
  29.   TRowHashMap = specialize TGLiteChainHashMap<string, TRowList, string>;
  30.   TRowMap     = TRowHashMap.TMap;
  31.   PRowList    = ^TRowList;
  32.  
  33. function BuildTripResponse(const ARoute: string; const aStopTimeRowMap, aTripRowMap: TRowMap): string;
  34. var
  35.   Trips, StopTimes: PRowList;
  36.   TripIdx, StopIdx: Integer;
  37.   WriteRef: specialize TGUniqRef<TJsonStrWriter>;
  38.   Writer: TJsonStrWriter;
  39.   p: PRow;
  40.   TripCol: TTripCols;
  41.   StopField: TStopTimeFields;
  42. begin
  43.   {%H-}WriteRef.Instance := TJsonStrWriter.Create;
  44.   Writer := WriteRef;
  45.   Writer.BeginArray;
  46.   if aTripRowMap.TryGetMutValue(ARoute, Trips) then
  47.     for TripIdx := 0 to Pred(Trips^.Count) do begin
  48.       p := Trips^.UncMutable[TripIdx]^;
  49.       Writer.BeginObject;
  50.       for TripCol in TTripCols do
  51.         Writer.Add(TripNames[TripCol], p^[Ord(TripCol)]);
  52.       Writer.AddName(SchedName).BeginArray;
  53.       if aStopTimeRowMap.TryGetMutValue(p^[Ord(tcTripId)], StopTimes) then
  54.         for StopIdx := 0 to Pred(StopTimes^.Count) do begin
  55.           p := StopTimes^.UncMutable[StopIdx]^;
  56.           Writer.BeginObject;
  57.           for StopField in TStopTimeFields do
  58.             Writer.Add(StopNames[StopField], p^[Ord(StopField)]);
  59.           Writer.EndObject;
  60.        end;
  61.       Writer.EndArray;
  62.       Writer.EndObject;
  63.     end;
  64.   Writer.EndArray;
  65.   Result := Writer.DataString;
  66. end;
  67.  
  68. procedure GetStopTimes(var aStopTimeRowMap: TRowMap; aDoc: TCsvDoc);
  69. var
  70.   Start, Stop: TDateTime;
  71.   I: Integer;
  72.   pStopList: PRowList;
  73.   p: PRow;
  74. begin
  75.   Start := Now;
  76.   aDoc.LoadFromFileMT('../MBTA_GTFS/stop_times.txt', StopTimeMask, Math.Min(TThread.ProcessorCount, 8)){%H-};
  77.   //aDoc.LoadFromFile('../MBTA_GTFS/stop_times.txt', StopTimeMask);
  78.   p := aDoc.Rows[0];
  79.   if(p^[Ord(stcTripId)] <> StopNames[stcTripId])or(p^[Ord(stcArrival)] <> StopNames[stcArrival])or
  80.     (p^[Ord(stcDeparture)] <> StopNames[stcDeparture])or(p^[Ord(stcStopId)] <> StopNames[stcStopId]) then begin
  81.     WriteLn('stop_times.txt not in expected format:');
  82.     for I := 0 to aDoc.ColCount[0] - 1 do
  83.       WriteLn(I, ' ' + p^[I]);
  84.     Halt(1);
  85.   end;
  86.   for I := 1 to Pred(aDoc.RowCount) do begin
  87.     p := aDoc.Rows[I];
  88.     aStopTimeRowMap.FindOrAddMutValue(p^[Ord(stcTripId)], pStopList);
  89.     pStopList^.Add(p);
  90.   end;
  91.   Stop := Now;
  92.   WriteLn('parsed ', aDoc.RowCount, ' stop times in ', SecondSpan(Start, Stop):1:3,' seconds');
  93. end;
  94.  
  95. procedure GetTrips(var aTripRowMap: TRowMap; aDoc: TCsvDoc);
  96. var
  97.   Start, Stop: TDateTime;
  98.   I: Integer;
  99.   pTripList: PRowList;
  100.   p: PRow;
  101. begin
  102.   Start := Now;
  103.   aDoc.LoadFromFile('../MBTA_GTFS/trips.txt', TripMask);
  104.   p := aDoc.Rows[0];
  105.   if(p^[Ord(tcRouteId)]<>TripNames[tcRouteId])or(p^[Ord(tcService)]<>TripNames[tcService])or(p^[Ord(tcTripId)]<>TripNames[tcTripId])then begin
  106.     WriteLn('trips.txt not in expected format:');
  107.     for I := 0 to aDoc.ColCount[0] - 1 do
  108.       WriteLn(I, ' ' + p^[I]);
  109.     Halt(1);
  110.   end;
  111.   for I := 1 to Pred(aDoc.RowCount) do begin
  112.     p := aDoc.Rows[I];
  113.     aTripRowMap.FindOrAddMutValue(p^[Ord(tcRouteId)], pTripList);
  114.     pTripList^.Add(p);
  115.   end;
  116.   Stop := Now;
  117.   WriteLn('parsed ', aDoc.RowCount, ' trips in ', SecondSpan(Start, Stop):1:3,' seconds');
  118. end;
  119.  
  120. var
  121.   StopTimeRowMap, TripRowMap: TRowMap;
  122.   StopTimes, Trips: specialize TGAutoRef<TCsvDoc>;
  123.  
  124. procedure SchedulesHandler(aRequest: TRequest; aResponse: TResponse);
  125. begin
  126.   aResponse.ContentType := 'application/json';
  127.   aResponse.Content := BuildTripResponse(aRequest.RouteParams['route'], StopTimeRowMap, TripRowMap);
  128. end;
  129.  
  130. procedure StopHandler(aRequest: TRequest; aResponse: TResponse);
  131. begin
  132.   Application.Terminate;
  133. end;
  134.  
  135. begin
  136.   GetStopTimes(StopTimeRowMap, StopTimes.Instance);
  137.   GetTrips(TripRowMap, Trips.Instance);
  138.  
  139.   Application.Port := 4000;
  140.   Application.Threaded := True;
  141.   **Router.RegisterRoute('/schedules/:route',@SchedulesHandler);
  142.   **Router.RegisterRoute('/stop', @StopHandler);
  143.   Application.Run;
  144. end.
  145.  

Results, the right column shows the results of the application, which is named in Leledumbo's repository as alt:
Code: [Select]
parsed 1739279 stop times in 0.611 seconds            | parsed 1739278 stop times in 748.66ms
parsed 69754 trips in 0.042 seconds                   | parsed 69753 trips in 36.05ms
                                                      |
running (0m42.2s), 00/50 VUs, 100 complete and 0 inter| running (0m35.6s), 00/50 VUs, 100 complete and 0 interra
default ✓ [ 100% ] 50 VUs  30s                          default ✓ [ 100% ] 50 VUs  30ss
                                                      |
 data_received..................: 8.5 GB 201 MB/s     | data_received..................: 8.5 GB 239 MB/s
 data_sent......................: 1.3 MB 30 kB/s      | data_sent......................: 930 kB 26 kB/s
 **_req_blocked...............: avg=2.75ms   min=0s | **_req_blocked...............: avg=107.98µs min=0s
 **_req_connecting............: avg=1.67ms   min=0s | **_req_connecting............: avg=37.17µs  min=0s
 **_req_duration..............: avg=206.97ms min=0s | **_req_duration..............: avg=171.51ms min=0s
   { expected_response:true }...: avg=206.97ms min=0s |   { expected_response:true }...: avg=171.51ms min=0s
 **_req_failed................: 0.00%  ✓ 0            **_req_failed................: 0.00%  ✓ 0
 **_req_receiving.............: avg=8.23ms   min=0s | **_req_receiving.............: avg=123.33ms min=0s
 **_req_sending...............: avg=1.35ms   min=0s | **_req_sending...............: avg=192.63µs min=0s
 **_req_tls_handshaking.......: avg=0s       min=0s | **_req_tls_handshaking.......: avg=0s       min=0s
 **_req_waiting...............: avg=197.38ms min=0s | **_req_waiting...............: avg=47.99ms  min=0s
 **_reqs......................: 9900   234.367143/s | **_reqs......................: 9900   278.394605/s
 iteration_duration.............: avg=20.72s   min=18.| iteration_duration.............: avg=17.04s   min=10.44s
 iterations.....................: 100    2.367345/s   | iterations.....................: 100    2.812067/s
 vus............................: 16     min=16       | vus............................: 12     min=12
 vus_max........................: 50     min=50       | vus_max........................: 50     min=50

Everything was compiled with FPC-3.3.1-12168-g14466ee9d9.

It would be very interesting to see how fp**app behaves on a modern and advanced device.

 

TinyPortal © 2005-2018