Recent

Author Topic: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)  (Read 3726 times)

iLya2IK

  • New Member
  • *
  • Posts: 22
The fpweb library is certainly good, but it clearly lacks modern features - namely, full-fledged http/1.1 with persistent connections, and also http/2. I completed my search for the desired analog by creating a server with all the necessary. I would like to present to your attention my project - WCHTTPServer.

The project builds on the fcl-web library and extends it to increase functionality:

Client management using cookies (saving and maintaining sessions).
  • Saving information about clients and sessions in SQLite database.
  • Saving information about the latest requests and saving logs in SQLite database.
  • Multithreading preparation and execution of requests based on thread pools e.g. helpful classes to work with EventSources.
  • Client rankings based on the frequency of client requests.
  • Built-in support for gzip and deflate compression methods including decompression of client-side requests.
  • The WebSocket (RFC 6455) protocol is supported with the "permessage-deflate" extension (RFC 7692).
  • Ability to start the server both in HTTP/2 (RFC 7540) mode and in HTTP/1.1 mode.
  • Ability to start the server both in HTTP/2 mode and in HTTP 1.1 mode.
  • Modified OpenSSL modules (added necessary TLS extensions) in order to create and maintain HTTP/2 connections.
  • Added the ability to save the master key and a random set of client data on the server-side (necessary for debugging TLS dumps using WireShark).
  • Works under both Windows and Linux

Link to project on GitHub : https://github.com/iLya2IK/wchttpserver

Now the project is at its final stage, but it requires a careful look from the outside. On the forum, I would like to see suggestions for improving the functionality and bug reports.

Thanks.
« Last Edit: June 03, 2021, 08:58:24 am by iLya2IK »

PierceNg

  • Full Member
  • ***
  • Posts: 118
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #1 on: February 25, 2021, 02:42:26 pm »
Great stuff! Thanks for making this available.

I only started looking at HTTP/2 and know next to nothing about it. I did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please.

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #2 on: February 25, 2021, 05:03:40 pm »
Great stuff! Thanks for making this available.
Thanks :D

I did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please.
Nice advice I'll try it. I used before h2load. But I think h2spec models a lot of additional non-standard situations.

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #3 on: February 26, 2021, 05:02:41 pm »
I checked the server with the h2spec utility: 41 passed from 94 tests.
I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads.
The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server.

ASBzone

  • Hero Member
  • *****
  • Posts: 614
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #4 on: February 26, 2021, 05:12:34 pm »
Thanks for this great work.

You may want to post this in the "Third Party" board of the forum, though:

https://forum.lazarus.freepascal.org/index.php/board,19.0.html
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.0.13 r64843 / FPC v3.2.1-r49055 (via FpcUpDeluxe) -- Windows 64-bit install w/Win32 and Linux/Arm cross-compiles
Primary System: Windows 10 Pro x64, Version 2009 (Build 19042)
Other Systems: Windows 10 Pro x64, Version 2009 (Build 19042) or greater

PierceNg

  • Full Member
  • ***
  • Posts: 118
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #5 on: February 27, 2021, 01:14:46 am »
I checked the server with the h2spec utility: 41 passed from 94 tests.
I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads.
The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server.

This is great - both your implementation and h2spec will improve through this interaction.

Natural generative adversarial neural networks at work.  :D

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #6 on: February 28, 2021, 07:25:39 pm »
Testing status now using h2spec: 94 tests, 84 passed, 0 skipped, 10 failed
Failed tests
Code: XML  [Select][+][-]
  1.   4. HTTP Frames
  2.     4.2. Frame Size      
  3.       × 1: Sends a DATA frame with 2^14 octets in length
  4.         -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.
  5.            Expected: HEADERS Frame (stream_id:1)
  6.              Actual: Connection closed
  7.             Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.
  8.  
  9.   5. Streams and Multiplexing
  10.     5.1. Stream States
  11.       5.1.2. Stream Concurrency        
  12.         × 1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded
  13.           -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR or REFUSED_STREAM.
  14.              Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  15.                        RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  16.                        GOAWAY Frame (Error Code: REFUSED_STREAM)
  17.                        RST_STREAM Frame (Error Code: REFUSED_STREAM)
  18.                        Connection closed
  19.                Actual: DATA Frame (length:845, flags:0x01, stream_id:201)
  20.               Comment: Ambiguous testing algorithm in h2spec. Streams quickly snatched up by worker threads
  21.  
  22.     5.5. Extending HTTP/2      
  23.       × 2: Sends an unknown extension frame in the middle of a header block
  24.         -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.
  25.            Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  26.                      Connection closed
  27.              Actual: Timeout
  28.             Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle
  29.                      of a header block should cause a connection error
  30.  
  31.   6. Frame Definitions
  32.     6.5. SETTINGS
  33.       6.5.3. Settings Synchronization        
  34.         × 1: Sends multiple values of SETTINGS_INITIAL_WINDOW_SIZE
  35.           -> The endpoint MUST process the values in the settings in the order they apper.
  36.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  37.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  38.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  39.  
  40.     6.9. WINDOW_UPDATE
  41.       6.9.1. The Flow-Control Window
  42.         × 1: Sends SETTINGS frame to set the initial window size to 1 and sends HEADERS frame
  43.           -> The endpoint MUST NOT send a flow-controlled frame with a length that exceeds the space available.
  44.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  45.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)        
  46.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  47.         × 2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1
  48.           -> The endpoint MUST sends a GOAWAY frame with a FLOW_CONTROL_ERROR code.
  49.              Expected: GOAWAY Frame (Error Code: FLOW_CONTROL_ERROR)
  50.                Actual: Timeout        
  51.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  52.         × 3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream
  53.           -> The endpoint MUST sends a RST_STREAM frame with a FLOW_CONTROL_ERROR code.
  54.              Expected: RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR)
  55.                Actual: Timeout
  56.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  57.  
  58.       6.9.2. Initial Flow-Control Window Size        
  59.         × 1: Changes SETTINGS_INITIAL_WINDOW_SIZE after sending HEADERS frame
  60.           -> The endpoint MUST adjust the size of all stream flow-control windows.
  61.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  62.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)        
  63.                Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  64.         × 2: Sends a SETTINGS frame for window size to be negative
  65.           -> The endpoint MUST track the negative flow-control window.
  66.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  67.                Actual: Timeout
  68.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  69.  
  70.   8. HTTP Message Exchanges
  71.     8.1. HTTP Request/Response Exchange
  72.       8.1.2. HTTP Header Fields
  73.         8.1.2.2. Connection-Specific Header Fields          
  74.           × 1: Sends a HEADERS frame that contains the connection-specific header field
  75.             -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.
  76.                Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  77.                          RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  78.                          Connection closed
  79.                  Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  80.                 Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them
  81.                          redundant and the type of reaction to the header value should remain on the server side
  82.  

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1584
  • Former Delphi 1-7, 10.2 user
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #7 on: February 28, 2021, 11:33:30 pm »
@iLya2IK: Great work! I've added a link to the Wiki Web Development Portal.
Lazarus 2.3 2021-08-10 FPC 3.3.1 r49223 macOS 10.14.6 Xcode 11.3.1
Lazarus 2.3 r65500 FPC 3.3.1 r49634 macOS 11.5.2 aarch64 Xcode 12.5.1
Lazarus 2.3 2021-08-11 FPC 3.2.2 FreeBSD 13.0 amd64 VMware VM
Lazarus 2.1 r61574 FPC 3.0.4 Ubuntu 20.04 Parallels VM
Lazarus 2.0.10 FPC 3.2.0 Win10 Parallels VM

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #8 on: March 08, 2021, 02:47:29 pm »
The project has undergone major changes related to debugging of flow control (according to sec.5.2 RFC 7540). Here are the actual results of testing with utilities.
Code: XML  [Select][+][-]
  1. # h2spec -p 8080 http2 -t -k -o 20
  2. 94 tests, 91 passed, 0 skipped, 3 failed
  3. Hypertext Transfer Protocol Version 2 (HTTP/2)
  4.   4. HTTP Frames
  5.     4.2. Frame Size      
  6.       × 1: Sends a DATA frame with 2^14 octets in length
  7.         -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.
  8.            Expected: HEADERS Frame (stream_id:1)
  9.              Actual: Connection closed
  10.             Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.  
  11.  
  12.   5. Streams and Multiplexing
  13.     5.5. Extending HTTP/2      
  14.       × 2: Sends an unknown extension frame in the middle of a header block
  15.         -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.
  16.            Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  17.                      Connection closed
  18.              Actual: Timeout
  19.             Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle
  20.                      of a header block should cause a connection error
  21.  
  22.   8. HTTP Message Exchanges
  23.     8.1. HTTP Request/Response Exchange
  24.       8.1.2. HTTP Header Fields
  25.         8.1.2.2. Connection-Specific Header Fields          
  26.           × 1: Sends a HEADERS frame that contains the connection-specific header field
  27.             -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.
  28.                Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  29.                          RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  30.                          Connection closed
  31.                  Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  32.                 Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them
  33.                          redundant and the type of reaction to the header value should remain on the server side      

And here are the results of testing with h2load
Code: XML  [Select][+][-]
  1. # h2load -n32000 -c32 -m10 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080        
  2. starting benchmark...
  3. spawning thread #0: 32 total client(s). 32000 total requests
  4. TLS Protocol: TLSv1.2
  5. Cipher: ECDHE-RSA-AES256-GCM-SHA384
  6. Server Temp Key: X25519 253 bits
  7. Application protocol: h2
  8. progress: 10% done
  9. progress: 20% done
  10. progress: 30% done
  11. progress: 40% done
  12. progress: 50% done
  13. progress: 60% done
  14. progress: 70% done
  15. progress: 80% done
  16. progress: 90% done
  17. progress: 100% done
  18.  
  19. finished in 17.22s, 1858.68 req/s, 1.54MB/s
  20. requests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeout
  21. status codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xx
  22. traffic: 26.46MB (27746560) total, 126.38KB (129408) headers (space savings 95.01%), 25.79MB (27040000) data
  23.                      min         max         mean         sd        +/- sd
  24. time for request:    16.59ms    327.96ms    150.22ms     35.42ms    77.80%
  25. time for connect:     7.17ms       1.96s       1.15s    618.49ms    62.50%
  26. time to 1st byte:    59.55ms       2.00s       1.20s    629.72ms    62.50%
  27. req/s           :      58.09       72.88       61.87        4.42    81.25%
  28.  
  29. # h2load -n32000 -c32 -t8 -m1 --h1 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080
  30. -t: warning: the number of threads is greater than hardware cores.
  31. starting benchmark...
  32. spawning thread #0: 4 total client(s). 4000 total requests
  33. spawning thread #1: 4 total client(s). 4000 total requests
  34. spawning thread #2: 4 total client(s). 4000 total requests
  35. spawning thread #3: 4 total client(s). 4000 total requests
  36. spawning thread #4: 4 total client(s). 4000 total requests
  37. spawning thread #5: 4 total client(s). 4000 total requests
  38. spawning thread #6: 4 total client(s). 4000 total requests
  39. spawning thread #7: 4 total client(s). 4000 total requests
  40. TLS Protocol: TLSv1.2
  41. Cipher: ECDHE-RSA-AES256-GCM-SHA384
  42. Server Temp Key: X25519 253 bits
  43. Application protocol: http/1.1
  44. progress: 10% done
  45. progress: 20% done
  46. progress: 30% done
  47. progress: 40% done
  48. progress: 50% done
  49. progress: 60% done
  50. progress: 70% done
  51. progress: 80% done
  52. progress: 90% done
  53. progress: 100% done
  54.  
  55. finished in 29.06s, 1101.26 req/s, 1.01MB/s
  56. requests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeout
  57. status codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xx
  58. traffic: 29.39MB (30816000) total, 2.53MB (2656000) headers (space savings 0.00%), 25.79MB (27040000) data
  59.                      min         max         mean         sd        +/- sd
  60. time for request:     7.99ms    208.16ms     26.31ms      5.94ms    93.71%
  61. time for connect:     8.22ms       3.08s       1.25s    943.18ms    62.50%
  62. time to 1st byte:    99.48ms       3.12s       1.31s    913.25ms    62.50%
  63. req/s           :      34.42       38.74       36.31        1.25    62.50%

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #9 on: April 22, 2021, 09:16:54 am »
Improved support for TLS v1.3, optimized the speed of the network code (read latency increased when the server was idle).

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #10 on: April 26, 2021, 03:38:49 pm »
Decompression of client-side requests added. Only the deflate unpacking method is currently supported on the server-side
Look at main.js and server.cnf.
To implement compression on client-side used zlib.js https://github.com/imaya/zlib.js
Code: Javascript  [Select][+][-]
  1. if (content.length > 200) {
  2.     var plain = new Uint8Array(toUTF8Array(content));
  3.     content = new Zlib.Deflate(plain, {compressionType: Zlib.Deflate.CompressionType.DYNAMIC}).compress();            
  4.     xhr.setRequestHeader('Content-Encoding', 'deflate');        
  5. }

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #11 on: May 05, 2021, 05:12:39 pm »
The latest update adds support for the websocket protocol, and therefore the demo site has been changed. As an example of interaction over the websocket protocol, the JSON-RPC application level protocol (well, or some of its similarity  ::) - you can change it to your own) is used. Now I am working on adding the permessage-deflate extension (RFC 7692)

mr-highball

  • Full Member
  • ***
  • Posts: 228
    • Highball Github
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #12 on: May 06, 2021, 06:09:43 am »
Cool. Apparently I already starred your repo, but this comment is here for an extra ⭐

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #13 on: May 06, 2021, 08:13:31 am »
Cool. Apparently I already starred your repo, but this comment is here for an extra ⭐

Thank you for your support  :D

iLya2IK

  • New Member
  • *
  • Posts: 22
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #14 on: May 15, 2021, 03:37:28 pm »
The structure of the project has been put in order. Added lpk file. The "permessage-deflate" extension for websocket has been fully implemented.

 

TinyPortal © 2005-2018