Recent

Author Topic: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)  (Read 3352 times)

iLya2IK

  • New Member
  • *
  • Posts: 17
The fpweb library is certainly good, but it clearly lacks modern features - namely, full-fledged http/1.1 with persistent connections, and also http/2. I completed my search for the desired analog by creating a server with all the necessary. I would like to present to your attention my project - WCHTTPServer.

The project builds on the fcl-web library and extends it to increase functionality:

Client management using cookies (saving and maintaining sessions).
  • Saving information about clients and sessions in SQLite database.
  • Saving information about the latest requests and saving logs in SQLite database.
  • Multithreading preparation and execution of requests based on thread pools e.g. helpful classes to work with EventSources.
  • Client rankings based on the frequency of client requests.
  • Built-in support for gzip and deflate compression methods including decompression of client-side requests.
  • The WebSocket (RFC 6455) protocol is supported with the "permessage-deflate" extension (RFC 7692).
  • Ability to start the server both in HTTP/2 (RFC 7540) mode and in HTTP/1.1 mode.
  • Ability to start the server both in HTTP/2 mode and in HTTP 1.1 mode.
  • Modified OpenSSL modules (added necessary TLS extensions) in order to create and maintain HTTP/2 connections.
  • Added the ability to save the master key and a random set of client data on the server-side (necessary for debugging TLS dumps using WireShark).
  • Works under both Windows and Linux

Link to project on GitHub : https://github.com/iLya2IK/wchttpserver

Now the project is at its final stage, but it requires a careful look from the outside. On the forum, I would like to see suggestions for improving the functionality and bug reports.

Thanks.
« Last Edit: June 03, 2021, 08:58:24 am by iLya2IK »

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #1 on: February 25, 2021, 02:42:26 pm »
Great stuff! Thanks for making this available.

I only started looking at HTTP/2 and know next to nothing about it. I did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please.

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #2 on: February 25, 2021, 05:03:40 pm »
Great stuff! Thanks for making this available.
Thanks :D

I did find https://github.com/summerwind/h2spec. Can you try it on your implementation and share your views on the output please.
Nice advice I'll try it. I used before h2load. But I think h2spec models a lot of additional non-standard situations.

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #3 on: February 26, 2021, 05:02:41 pm »
I checked the server with the h2spec utility: 41 passed from 94 tests.
I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads.
The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server.

ASBzone

  • Hero Member
  • *****
  • Posts: 613
  • Automation leads to relaxation...
    • Free Console Utilities for Windows (and a few for Linux) from BrainWaveCC
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #4 on: February 26, 2021, 05:12:34 pm »
Thanks for this great work.

You may want to post this in the "Third Party" board of the forum, though:

https://forum.lazarus.freepascal.org/index.php/board,19.0.html
-ASB: https://www.BrainWaveCC.com/

Lazarus v2.0.13 r64843 / FPC v3.2.1-r49055 (via FpcUpDeluxe) -- Windows 64-bit install w/Win32 and Linux/Arm cross-compiles
Primary System: Windows 10 Pro x64, Version 2009 (Build 19042)
Other Systems: Windows 10 Pro x64, Version 2009 (Build 19042) or greater

PierceNg

  • Full Member
  • ***
  • Posts: 110
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #5 on: February 27, 2021, 01:14:46 am »
I checked the server with the h2spec utility: 41 passed from 94 tests.
I found some bugs in my code caused by a misunderstanding of the http2 protocol (mostly in the part of informing clients about malformed requests and protocol errors). I also found that tests have their own bugs and some ambiguous testing algorithms. For example - Test 5.1/1 waits for the GOAWAY frame with PROTOCOL_ERROR, but section 6.1 in RFC 7540 requires GOAWAY frame with STREAM_CLOSED error. Another example - Test to emulate maximum concurrent streams. On my server, the test program was unable to create so many concurrent streams - no matter how I increase or decrease the MAX_CONCURRENT_STREAMS value - they seem to be quickly snatched up by worker threads.
The author has yet to work out his testing program better, but now it is a powerful tool and I intend to use it further to improve the reliability and adequacy of my server.

This is great - both your implementation and h2spec will improve through this interaction.

Natural generative adversarial neural networks at work.  :D

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #6 on: February 28, 2021, 07:25:39 pm »
Testing status now using h2spec: 94 tests, 84 passed, 0 skipped, 10 failed
Failed tests
Code: XML  [Select][+][-]
  1.   4. HTTP Frames
  2.     4.2. Frame Size      
  3.       × 1: Sends a DATA frame with 2^14 octets in length
  4.         -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.
  5.            Expected: HEADERS Frame (stream_id:1)
  6.              Actual: Connection closed
  7.             Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.
  8.  
  9.   5. Streams and Multiplexing
  10.     5.1. Stream States
  11.       5.1.2. Stream Concurrency        
  12.         × 1: Sends HEADERS frames that causes their advertised concurrent stream limit to be exceeded
  13.           -> The endpoint MUST treat this as a stream error of type PROTOCOL_ERROR or REFUSED_STREAM.
  14.              Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  15.                        RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  16.                        GOAWAY Frame (Error Code: REFUSED_STREAM)
  17.                        RST_STREAM Frame (Error Code: REFUSED_STREAM)
  18.                        Connection closed
  19.                Actual: DATA Frame (length:845, flags:0x01, stream_id:201)
  20.               Comment: Ambiguous testing algorithm in h2spec. Streams quickly snatched up by worker threads
  21.  
  22.     5.5. Extending HTTP/2      
  23.       × 2: Sends an unknown extension frame in the middle of a header block
  24.         -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.
  25.            Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  26.                      Connection closed
  27.              Actual: Timeout
  28.             Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle
  29.                      of a header block should cause a connection error
  30.  
  31.   6. Frame Definitions
  32.     6.5. SETTINGS
  33.       6.5.3. Settings Synchronization        
  34.         × 1: Sends multiple values of SETTINGS_INITIAL_WINDOW_SIZE
  35.           -> The endpoint MUST process the values in the settings in the order they apper.
  36.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  37.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  38.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  39.  
  40.     6.9. WINDOW_UPDATE
  41.       6.9.1. The Flow-Control Window
  42.         × 1: Sends SETTINGS frame to set the initial window size to 1 and sends HEADERS frame
  43.           -> The endpoint MUST NOT send a flow-controlled frame with a length that exceeds the space available.
  44.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  45.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)        
  46.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  47.         × 2: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1
  48.           -> The endpoint MUST sends a GOAWAY frame with a FLOW_CONTROL_ERROR code.
  49.              Expected: GOAWAY Frame (Error Code: FLOW_CONTROL_ERROR)
  50.                Actual: Timeout        
  51.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  52.         × 3: Sends multiple WINDOW_UPDATE frames increasing the flow control window to above 2^31-1 on a stream
  53.           -> The endpoint MUST sends a RST_STREAM frame with a FLOW_CONTROL_ERROR code.
  54.              Expected: RST_STREAM Frame (Error Code: FLOW_CONTROL_ERROR)
  55.                Actual: Timeout
  56.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  57.  
  58.       6.9.2. Initial Flow-Control Window Size        
  59.         × 1: Changes SETTINGS_INITIAL_WINDOW_SIZE after sending HEADERS frame
  60.           -> The endpoint MUST adjust the size of all stream flow-control windows.
  61.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  62.                Actual: DATA Frame (length:845, flags:0x01, stream_id:1)        
  63.                Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  64.         × 2: Sends a SETTINGS frame for window size to be negative
  65.           -> The endpoint MUST track the negative flow-control window.
  66.              Expected: DATA Frame (length:1, flags:0x00, stream_id:1)
  67.                Actual: Timeout
  68.               Comment: Agreed. I working to fix this. My problem - i don't clearly understand the WINDOW_UPDATE mechanism.
  69.  
  70.   8. HTTP Message Exchanges
  71.     8.1. HTTP Request/Response Exchange
  72.       8.1.2. HTTP Header Fields
  73.         8.1.2.2. Connection-Specific Header Fields          
  74.           × 1: Sends a HEADERS frame that contains the connection-specific header field
  75.             -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.
  76.                Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  77.                          RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  78.                          Connection closed
  79.                  Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  80.                 Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them
  81.                          redundant and the type of reaction to the header value should remain on the server side
  82.  

trev

  • Global Moderator
  • Hero Member
  • *****
  • Posts: 1474
  • Former Delphi 1-7, 10.2 user
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #7 on: February 28, 2021, 11:33:30 pm »
@iLya2IK: Great work! I've added a link to the Wiki Web Development Portal.
Lazarus 2.1 r65061 FPC 3.3.1 r49223 macOS 10.14.6 Xcode 11.3.1
Lazarus 2.1 r65182 FPC 3.3.1 r49223 macOS 11.4 aarch64 Xcode 12.4
Lazarus 2.1 r61574 FPC 3.3.1 r42318 FreeBSD 12.1 amd64 VMware VM
Lazarus 2.1 r61574 FPC 3.0.4 Ubuntu 20.04 Parallels VM
Lazarus 2.0.10 FPC 3.2.0 Win10 Parallels VM

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server on Lazarus (Free Pascal)
« Reply #8 on: March 08, 2021, 02:47:29 pm »
The project has undergone major changes related to debugging of flow control (according to sec.5.2 RFC 7540). Here are the actual results of testing with utilities.
Code: XML  [Select][+][-]
  1. # h2spec -p 8080 http2 -t -k -o 20
  2. 94 tests, 91 passed, 0 skipped, 3 failed
  3. Hypertext Transfer Protocol Version 2 (HTTP/2)
  4.   4. HTTP Frames
  5.     4.2. Frame Size      
  6.       × 1: Sends a DATA frame with 2^14 octets in length
  7.         -> The endpoint MUST be capable of receiving and minimally processing frames up to 2^14 octets in length.
  8.            Expected: HEADERS Frame (stream_id:1)
  9.              Actual: Connection closed
  10.             Comment: Error in h2spec - DATA payload size didn't equal the value of "content-length" header.  
  11.  
  12.   5. Streams and Multiplexing
  13.     5.5. Extending HTTP/2      
  14.       × 2: Sends an unknown extension frame in the middle of a header block
  15.         -> The endpoint MUST treat as a connection error of type PROTOCOL_ERROR.
  16.            Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  17.                      Connection closed
  18.              Actual: Timeout
  19.             Comment: I didn't find any references in RFC 7540 that unknown extension frame in the middle
  20.                      of a header block should cause a connection error
  21.  
  22.   8. HTTP Message Exchanges
  23.     8.1. HTTP Request/Response Exchange
  24.       8.1.2. HTTP Header Fields
  25.         8.1.2.2. Connection-Specific Header Fields          
  26.           × 1: Sends a HEADERS frame that contains the connection-specific header field
  27.             -> The endpoint MUST respond with a stream error of type PROTOCOL_ERROR.
  28.                Expected: GOAWAY Frame (Error Code: PROTOCOL_ERROR)
  29.                          RST_STREAM Frame (Error Code: PROTOCOL_ERROR)
  30.                          Connection closed
  31.                  Actual: DATA Frame (length:845, flags:0x01, stream_id:1)
  32.                 Comment: I know that this violates the requirements of RFC 7540, but I sincerely consider them
  33.                          redundant and the type of reaction to the header value should remain on the server side      

And here are the results of testing with h2load
Code: XML  [Select][+][-]
  1. # h2load -n32000 -c32 -m10 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080        
  2. starting benchmark...
  3. spawning thread #0: 32 total client(s). 32000 total requests
  4. TLS Protocol: TLSv1.2
  5. Cipher: ECDHE-RSA-AES256-GCM-SHA384
  6. Server Temp Key: X25519 253 bits
  7. Application protocol: h2
  8. progress: 10% done
  9. progress: 20% done
  10. progress: 30% done
  11. progress: 40% done
  12. progress: 50% done
  13. progress: 60% done
  14. progress: 70% done
  15. progress: 80% done
  16. progress: 90% done
  17. progress: 100% done
  18.  
  19. finished in 17.22s, 1858.68 req/s, 1.54MB/s
  20. requests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeout
  21. status codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xx
  22. traffic: 26.46MB (27746560) total, 126.38KB (129408) headers (space savings 95.01%), 25.79MB (27040000) data
  23.                      min         max         mean         sd        +/- sd
  24. time for request:    16.59ms    327.96ms    150.22ms     35.42ms    77.80%
  25. time for connect:     7.17ms       1.96s       1.15s    618.49ms    62.50%
  26. time to 1st byte:    59.55ms       2.00s       1.20s    629.72ms    62.50%
  27. req/s           :      58.09       72.88       61.87        4.42    81.25%
  28.  
  29. # h2load -n32000 -c32 -t8 -m1 --h1 --header=connection:keep-alive --header=cookie:cid=11  https://localhost:8080
  30. -t: warning: the number of threads is greater than hardware cores.
  31. starting benchmark...
  32. spawning thread #0: 4 total client(s). 4000 total requests
  33. spawning thread #1: 4 total client(s). 4000 total requests
  34. spawning thread #2: 4 total client(s). 4000 total requests
  35. spawning thread #3: 4 total client(s). 4000 total requests
  36. spawning thread #4: 4 total client(s). 4000 total requests
  37. spawning thread #5: 4 total client(s). 4000 total requests
  38. spawning thread #6: 4 total client(s). 4000 total requests
  39. spawning thread #7: 4 total client(s). 4000 total requests
  40. TLS Protocol: TLSv1.2
  41. Cipher: ECDHE-RSA-AES256-GCM-SHA384
  42. Server Temp Key: X25519 253 bits
  43. Application protocol: http/1.1
  44. progress: 10% done
  45. progress: 20% done
  46. progress: 30% done
  47. progress: 40% done
  48. progress: 50% done
  49. progress: 60% done
  50. progress: 70% done
  51. progress: 80% done
  52. progress: 90% done
  53. progress: 100% done
  54.  
  55. finished in 29.06s, 1101.26 req/s, 1.01MB/s
  56. requests: 32000 total, 32000 started, 32000 done, 32000 succeeded, 0 failed, 0 errored, 0 timeout
  57. status codes: 32000 2xx, 0 3xx, 0 4xx, 0 5xx
  58. traffic: 29.39MB (30816000) total, 2.53MB (2656000) headers (space savings 0.00%), 25.79MB (27040000) data
  59.                      min         max         mean         sd        +/- sd
  60. time for request:     7.99ms    208.16ms     26.31ms      5.94ms    93.71%
  61. time for connect:     8.22ms       3.08s       1.25s    943.18ms    62.50%
  62. time to 1st byte:    99.48ms       3.12s       1.31s    913.25ms    62.50%
  63. req/s           :      34.42       38.74       36.31        1.25    62.50%

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #9 on: April 22, 2021, 09:16:54 am »
Improved support for TLS v1.3, optimized the speed of the network code (read latency increased when the server was idle).

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #10 on: April 26, 2021, 03:38:49 pm »
Decompression of client-side requests added. Only the deflate unpacking method is currently supported on the server-side
Look at main.js and server.cnf.
To implement compression on client-side used zlib.js https://github.com/imaya/zlib.js
Code: Javascript  [Select][+][-]
  1. if (content.length > 200) {
  2.     var plain = new Uint8Array(toUTF8Array(content));
  3.     content = new Zlib.Deflate(plain, {compressionType: Zlib.Deflate.CompressionType.DYNAMIC}).compress();            
  4.     xhr.setRequestHeader('Content-Encoding', 'deflate');        
  5. }

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+1.1 server written with Lazarus (Free Pascal)
« Reply #11 on: May 05, 2021, 05:12:39 pm »
The latest update adds support for the websocket protocol, and therefore the demo site has been changed. As an example of interaction over the websocket protocol, the JSON-RPC application level protocol (well, or some of its similarity  ::) - you can change it to your own) is used. Now I am working on adding the permessage-deflate extension (RFC 7692)

mr-highball

  • Full Member
  • ***
  • Posts: 225
    • Highball Github
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #12 on: May 06, 2021, 06:09:43 am »
Cool. Apparently I already starred your repo, but this comment is here for an extra ⭐

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #13 on: May 06, 2021, 08:13:31 am »
Cool. Apparently I already starred your repo, but this comment is here for an extra ⭐

Thank you for your support  :D

iLya2IK

  • New Member
  • *
  • Posts: 17
Re: HTTP/2+HTTP/1.1+WebSocket server written with Lazarus (Free Pascal)
« Reply #14 on: May 15, 2021, 03:37:28 pm »
The structure of the project has been put in order. Added lpk file. The "permessage-deflate" extension for websocket has been fully implemented.

 

TinyPortal © 2005-2018