Programming language comparison by implementing the same transit data app

[abouchez]

@avk
Great!
Manual JSON serialization is clearly a way to have good performance. But it is clearly cheating against other languages, which use RTTI for this, over a dedicated transient list/array of results.
IIRC the fpjsonrtti unit may be able to serialize a collection of TObject?

As far as I can tell, fphttpserver.pp maintains one thread per request in threaded mode, not one thread per connection?
Weird. Would be slow on any modern CPUs, in comparison to thread-polled HTTP servers, which maintains the connection.
Edit: I was looking at an older version of fphttpserver source from last year. Now there is a thread maintained per connection. Not good for a lot of connections at the same time, but much better than the previous implementation.

Where from are the numbers on the left side?

[avk]

Thank you. This is undoubtedly cheating, but it's not like we're going to submit it as an example.

As for the numbers, the left side shows app2(fphttpapp) results and the right shows alt(Mormot2 server, i.e. yours).

[abouchez]

Great!
Just to be sure, you are measuring it on Linux?

As expected, for 50 connections, fphttpapp is just fine with HTTP/1.1 connections. It is similar to the old mORMot 1 socket server from 10 years ago for HTTP/1.1: one thread per connection. For HTTP1.0 connection, it seems to create a thread per request, whereas mORMot 1 used a thread-poll for such requests.
It also miss some features like "100 continue" or body chunking requests, which are pretty common in the wild.

My guess is that if we try with 5000 connections, or with HTTP/1.0 short-living connections (e.g. over a reverse proxy) fphttpapp would weep.

[avk]

No, I ran it on a Windows system, could that cause any negative impacts?

[abouchez]

On Windows, our async HTTP server is in "compatibility mode" with regular socket access. Not very powerful. You may switch to our http.sys server instead for a pure Windows access, by switching to the THttpApiServer class instead of THttpAsyncServer.

On Linux, our async HTTP server uses the very efficient epoll API.
Also ensure you use x86_64 as CPU, since it is a typical server CPU, and we optimized mORMot for it.

For my laptop, I have magnitude order better results with Linux x86_64 than with Win32/Win64.

[avk]

Thank you very much!

[Leledumbo]

Results, the right column shows the results of the application, which is named in Leledumbo's repository as alt:

I think in your solution fphttpapp is now the bottleneck (look at http_req_blocked, http_req_connecting, http_req_sending and http_req_waiting, although interestingly http_req_receiving is the other way around but that's the only one that's significantly better), I know mine isn't yet, it's the too plain translation from Go with random inefficient replacements that I did was the actual bottleneck.

Btw, do you might want to incorporate this solution to my repo?

[abouchez]

I also noticed that you are using PRow data, not high-level classes/records to store the data.
Another trick which is not fair, in respect to the language comparison goal.

@Leledumbo
About fphttpserver, the numbers are very good, for global throughput. But creating the per-connection thread makes connection slower.

What I don't understand is why the "sent" data is lower on the left. It should be the same for both sides.
Did you verify the JSON content?

[avk]

<...>
Btw, do you might want to incorporate this solution to my repo?

Doesn't it bother you that manual serialization in this case looks like an obvious cheating?

I also noticed that you are using PRow data, not high-level classes/records to store the data.
Another trick which is not fair, in respect to the language comparison goal.

Yes, of course, but don't get me wrong, I had a slightly different goal: to find out approximately
which part of the responsibility for delays in the current test lies with the server and which part
lies with the code that generates the response.

What I don't understand is why the "sent" data is lower on the left. It should be the same for both sides.
Did you verify the JSON content?

Roughly and selectively checked what BuildTripResponse() returns, something like this:

Code: Pascal  [Select][+][-]

1. procedure Test;
2. var
3.   I, Row, Range: SizeInt;
4.   Rout, Resp: string;
5.   Start: QWord;
6. begin
7.   Randomize;
8.   Range := Trips.Instance.RowCount - 1;
9.   Start := GetTickCount64;
10.   for I := 1 to 10000 do
11.     begin
12.       Row := Succ(Random(Range));
13.       Rout := Trips.Instance[Row, Ord(tcRouteId)];
14.       Resp := BuildTripResponse(Rout, StopTimeRowMap, TripRowMap);
15.       if (Resp = '[]') or not TJsonNode.ValidJson(Resp) then
16.         WriteLn('invalid response');
17.     end;
18.   WriteLn('Elapsed: ', GetTickCount64 - Start);
19. end;
20.  

[abouchez]

Perhaps compare the responses of the 3 solutions you have.
They should return the very same JSON for the same URI, I suppose.

The fphttpapp server seems stable enough. Congrats. A lot of good work done since the last months.
It won't perform badly with a few client connections.
It is in pair with mORMot 1 HTTP server from 10 years ago.
But I am afraid it won't scale with a lot of concurrent connections, with its one-thread-per-client pattern.

IMHO my previous remarks should better be fixed to be usable on production:
- proper HTTP/1.0 process (very common behind a reverse proxy like nginx, especially if you want to use the proxy to maintain a lot of concurrent connections) without a new thread per request
- chunked input body support (very common with Web clients)
- 100 CONTINUE support (still happens)
- and also perhaps explicit HEAD support (I didn't remember anything about HEAD requests in fphttpserver source)

I discovered stability issues and bottlenecks in the mORMot server from aggressive tests with the wrk tool for instance.
This wrk tool is much better than the ab tool from my tests.
It could be very good for you to test and validate fphttpserver with wrk, and see how many concurrent connections it sustains.

[avk]

Just checked, it looks like the responses from app2 and alt are the same, but app generates something completely different.

[abouchez]

So mORMot results are not the same?

[avk]

I seem to have created a fair amount of confusion with the versions?
Let me fix it, so:

• app - app.pas in Leledumbo's repository, fphttpapp application
• alt   - alt.pas in Leledumbo's repository, mormot2 application
• app2 - post #59 in this thread, modified version of app

So mORMot results seem to be okay .

[avk]

I guess I found the reason why responses of app are different from others.
line 181 of app.pas:

Code: Pascal  [Select][+][-]

1.       //LTrip := ATrips[LTripIx];
2.       LTrip := ATrips[LTripIxs[LTripIx]];
3.  

and line 187:

Code: Pascal  [Select][+][-]

1.           //LStopTime := AStopTimes[LStopTimeIx];
2.           LStopTime := AStopTimes[LStopTimeIxs[LStopTimeIx]];
3.  

[PierceNg]

I just built FPC LLVM from source with LLVM 14. It doesn't compile the lgenerics library:

Code: Text  [Select][+][-]

1. % fpc -Clv14.0 -Fulgenerics app.pas
2. Free Pascal Compiler version 3.3.1 [2022/12/10] for x86_64
3. Copyright (c) 1993-2022 by Florian Klaempfl and others
4. Target OS: Linux for x86-64
5. Compiling app.pas
6. Compiling ./lgenerics/lgutils.pas
7. Compiling ./lgenerics/lgstrconst.pas
8. Writing Resource String Table file: lgstrconst.rsj
9. Assembling lgstrconst
10. lgutils.pas(3321,26) Warning: Function result variable of a managed type does not seem to be initialized
11. Assembling lgutils
12. Compiling ./lgenerics/lghashmap.pas
13. Compiling ./lgenerics/lghelpers.pas
14. Compiling ./lgenerics/lghash.pas
15. lghash.pas(442,35) Fatal: Internal error 200611011
16. Fatal: Compilation aborted
17. Error: /home/pierce/pkg/fpcllvm/lib/fpc/3.3.1/ppcx64 returned an error exitcode

 

This is the second program I'm building with FPC LLVM. The first one, hello world , builds and runs.