Quiche (and BoringSSL) bindings

[Fred vS]

Bigger buffer,not PChar did the trick.The error is still the same.

And the debugger seems not very cooperative.

I'm on Gentoo Linux, I tried to rebuild Quiche with the debug flag but it fails and it seems to be a known issue. I also dowloaded the source and compiled it manualjy but it's giving me some linking error when I try to build my program.It looks like Quiche objects to debugging

Yes, I understand you totally, I just ended a hard battle to make libfdk-aac-2.so decode aac webstream and it takes me lot of white nights and hair.
For dealing with C libraries, I prefer to use dynamic loading using loadlibrary() and then assign and test each methods.
But it needs more work than using "external" but gives more control too.

[Fred vS]

...and messing with Quiche_Recv_Info doesn't change anything when it comes to the error but...

I do

Code: Pascal  [Select][+][-]

1. writeln('SizeOf(recv_info)=',SizeOf(recv_info));

right before calling quiche_conn_recv and the result reflects changes in the definition of Quiche_Recv_Info;
when it's (supposedly) correct, it returns 21, after I modified the record, it returned 24 but when I leave it "as it is",without deifning fields, it returns 0.I'm not sure if this information may help in anything.

Sorry, not sure to understand, can you access recv_info after calling quiche_conn_recv() ?
I was thinking that after quiche_conn_recv there was a crash. 

[TRon]

don't mess with packed record, use packrecords c instead (though it is rust so do not know if that does the same layout as c. I assumed it does)

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$h+}
2. {$packrecords c}
3.  
4. uses
5.   unixtype, sockets;
6.  
7. type
8.   // A QUIC connection.
9.   // typedef struct quiche_conn quiche_conn;
10.   Tquiche_conn = record end;
11.   Pquiche_conn = ^Tquiche_conn;
12.  
13.   Pquiche_recv_info = ^Tquiche_recv_info;
14.   Tquiche_recv_info = record
15.     // The remote address the packet was received from.
16.     from     : psockaddr;
17.     from_len : socklen_t;
18.  
19.     // The local address the packet was received on.
20.     &to      : psockaddr;
21.     to_len   : socklen_t;
22.   end;
23.  
24.  
25.   function quiche_conn_recv(conn: pquiche_conn; buf: pcuint8; buf_len: size_t; const info: pquiche_recv_info): ssize_t; cdecl; blah blah and blah blah
26.  

Is what I am able to come up with.

Do you perhaps need to change the struct to specifically use ip4 vs ip6 ?

right before calling quiche_conn_recv and the result reflects changes in the definition of Quiche_Recv_Info;
when it's (supposedly) correct, it returns 21, after I modified the record, it returned 24 but when I leave it "as it is",without deifning fields, it returns 0.I'm not sure if this information may help in anything.

32 bit or 64 bit os ? on 64-bit it should read 32 (on 32-bit 8 bytes less).

[r.lukasiak]

I was thinking that after quiche_conn_recv there was a crash. 

yes, it crashed AFTER quiche_conn_recv but I do  SizeOf(recv_info) right BEFORE and it returns different values, depending on its definition.

[r.lukasiak]

thanks for joining, TRon!

don't mess with packed record, use packrecords c instead

ok, I'm back to just record with {$packrecords c}

but

function quiche_conn_recv(conn: pquiche_conn; buf: pcuint8; buf_len: size_t; const info: pquiche_recv_info): ssize_t; cdecl;

is giving Access violation as it was in the very beginning of this post :|

though it is rust so do not know if that does the same layout as c

this was actually my concern right when I started facing errors but till now I was able to somehow solve them, conn_recv seems to be exceptionally stubborn tho.

However... There is this:

Code: Pascal  [Select][+][-]

1. type
2. PQuiche_stats = ^Quiche_stats;
3.   Quiche_stats = record
4.     recv: QWord;
5.     sent: QWord;
6.     lost: QWord;
7.     retrans: QWord;
8.     delivered: QWord;
9.     delivered_time: QWord;
10.     smoothed_rtt: QWord;
11.     rttvar: QWord;
12.     min_rtt: QWord;
13.     pkt_tx_count: QWord;
14.     pkt_rx_count: QWord;
15.   end;  
16.  
17. var
18.   Stats: Quiche_stats;
19. .
20. .
21. quiche_conn_stats(conn, Stats);
22.     WriteLn('Connection Statistics:');
23.     WriteLn('Packets received: ', stats.recv);
24.     WriteLn('Packets sent: ', stats.sent);
25.     WriteLn('Packets lost: ', stats.lost);
26.     WriteLn('Retransmissions: ', stats.retrans);
27.     WriteLn('Delivered: ', stats.delivered);
28.     WriteLn('Delivery time: ', stats.delivered_time);
29.     WriteLn('Smoothed RTT: ', stats.smoothed_rtt);
30.     WriteLn('RTT variance: ', stats.rttvar);
31.     WriteLn('Minimum RTT: ', stats.min_rtt);
32.     WriteLn('Packet TX count: ', stats.pkt_tx_count);
33.     WriteLn('Packet RX count: ', stats.pkt_rx_count);
34.  
35.  

and it works like a charm, regardless {$packrecords c}. I just converted it the way I just learnt and I didn't have a single error here.

[TRon]

thanks for joining, TRon!

You're most welcome.

and it works like a charm, regardless {$packrecords c}. I just converted it the way I just learnt and I didn't have a single error here.

Ok, hold on a moment. That is not what I see at https://github.com/cloudflare/quiche/blob/master/quiche/include/quiche.h

What version of the header are you using/converting ? That last structure you showed should be 112 bytes not 88 :-/

is giving Access violation as it was in the very beginning of this post :|

See previous question. I converted the function as expressed in the header in the provided link. If that still crashes for you then something else must be off. I am already a bit scared about the 21 and 24 bits you mentioned earlier. Could you share the order of which you include the units in your uses clause (both interface and implementation) ?

Code: Pascal  [Select][+][-]

1. {$mode objfpc}{$h+}
2. {$packrecords c}
3.  
4. uses
5.   unixtype, sockets;  
6.  
7. type
8.   Tquiche_stats = record
9.     // The number of QUIC packets received on this connection.
10.     recv : size_t;
11.  
12.     // The number of QUIC packets sent on this connection.
13.     sent : size_t;
14.  
15.     // The number of QUIC packets that were lost.
16.     lost : size_t;
17.  
18.     // The number of sent QUIC packets with retransmitted data.
19.     retrans : size_t;
20.  
21.     // The number of sent bytes.
22.     sent_bytes : uint64;
23.  
24.     // The number of received bytes.
25.     recv_bytes : uint64;
26.  
27.     // The number of bytes acked.
28.     acked_bytes : uint64;
29.  
30.     // The number of bytes lost.
31.     lost_bytes : uint64;
32.  
33.     // The number of stream bytes retransmitted.
34.     stream_retrans_bytes : uint64;
35.  
36.     // The number of known paths for the connection.
37.     paths_count : size_t;
38.  
39.     // The number of streams reset by local.
40.     reset_stream_count_local : uint64;
41.  
42.     // The number of streams stopped by local.
43.     stopped_stream_count_local: uint64;
44.  
45.     // The number of streams reset by remote.
46.     reset_stream_count_remote : uint64;
47.  
48.     // The number of streams stopped by remote.
49.     stopped_stream_count_remote : uint64;
50.   end;
51.  

[TRon]

My apologies I need to correct myself about some earlier statements about size:
32-bit:
sizeof(Tquiche_recv_info) = 16
sizeof(Tquiche_stats)     = 92

64-bit:
sizeof(Tquiche_recv_info) = 32
sizeof(Tquiche_stats)     = 112

[tetrastes]

It would be useful to write simple C program showing these sizeofs and compare them.

[Fred vS]

However... There is this:

Code: Pascal  [Select][+][-]

1. type
2. PQuiche_stats = ^Quiche_stats;
3.   Quiche_stats = record
4.     recv: QWord;
5.     sent: QWord;
6.     lost: QWord;
7.     retrans: QWord;
8.     delivered: QWord;
9.     delivered_time: QWord;
10.     smoothed_rtt: QWord;
11.     rttvar: QWord;
12.     min_rtt: QWord;
13.     pkt_tx_count: QWord;
14.     pkt_rx_count: QWord;
15.   end;  
16.  
17. var
18.   Stats: Quiche_stats;
19. .
20. .
21. quiche_conn_stats(conn, Stats);
22.     WriteLn('Connection Statistics:');
23.     WriteLn('Packets received: ', stats.recv);
24.     WriteLn('Packets sent: ', stats.sent);
25.     WriteLn('Packets lost: ', stats.lost);
26.     WriteLn('Retransmissions: ', stats.retrans);
27.     WriteLn('Delivered: ', stats.delivered);
28.     WriteLn('Delivery time: ', stats.delivered_time);
29.     WriteLn('Smoothed RTT: ', stats.smoothed_rtt);
30.     WriteLn('RTT variance: ', stats.rttvar);
31.     WriteLn('Minimum RTT: ', stats.min_rtt);
32.     WriteLn('Packet TX count: ', stats.pkt_tx_count);
33.     WriteLn('Packet RX count: ', stats.pkt_rx_count);
34.  
35.  

and it works like a charm, regardless {$packrecords c}. I just converted it the way I just learnt and I didn't have a single error here.

Hum, in the C header, the method is defined like this:

Code: C  [Select][+][-]

1. void quiche_conn_stats(const quiche_conn *conn, quiche_stats *out);

Where conn and out are pointers.
But in your code:

Code: Pascal  [Select][+][-]

1. quiche_conn_stats(conn, Stats);

only conn is a pointer, Stats is a Quiche_stats = record.
And it works like charm! 
May I ask you how you defined quiche_conn_stats() in your Pascal header?

So, maybe do the same for quiche_conn_recv()?

Code: Pascal  [Select][+][-]

1.     quiche_conn_recv(conn: PQuiche_Conn; buf: PByte; buf_len: CSize_t;
2.                    var info: Quiche_Recv_Info): CSsize_t; cdecl; external QUICHE_LIB;
3. var
4.       ...
5.       buf: PByte;
6.       recv_info: Quiche_recv_info;
7.       ...
8.      
9.       GetMem(buf, 8192*8);  // big buffer
10.       ...
11.       bytes_received := quiche_conn_recv(conn, @buf, 8192, recv_info);

Yes it should be strange if it works but all is strange those days.

[r.lukasiak]

Ok, hold on a moment. That is not what I see at https://github.com/cloudflare/quiche/blob/master/quiche/include/quiche.h

indeed... now I'm wondering what I was looking at when I was converting this... and what makes me think even more, why this even worked? I was able to get the stats.... But I think it just proves that the name of the field doesn't matter, as long as the size is ok. I had fewer fields so it didn't exceed the real size. Anyway,now I corrected it and it's 112 bytes.

Could you share the order of which you include the units in your uses clause (both interface and implementation) ?

I actually have uses only in the interface section:

Code: Pascal  [Select][+][-]

1. interface
2.  
3. uses
4. CTypes, Sockets, BaseUnix;
5.  

I am already a bit scared about the 21 and 24 bits you mentioned earlier

now I checked and sizeof(Tquiche_recv_info) = 32, it was 24 when I was trying packed records and 21 was when I was trying @Fred's suggestion to enter random data into the record to see if there is any reaction.

now it's:

Code: Pascal  [Select][+][-]

1. PQuiche_Recv_Info = ^TQuiche_Recv_Info;
2.   TQuiche_Recv_Info = record
3.       from : Psockaddr;
4.       from_len : socklen_t;
5.       &to : Psockaddr;
6.       to_len : socklen_t;
7.   end;

and it weights 32 bytes

[r.lukasiak]

@Fred vS

May I ask you how you defined quiche_conn_stats() in your Pascal header?

now I see your point...

Code: Pascal  [Select][+][-]

1.  
2. procedure quiche_conn_stats(var conn:PQuiche_conn; var _out:TQuiche_stats); cdecl; external QUICHE_LIB;
3.  

but how the heck it didn't cause any error? o.O

now I corrected it:

Code: Pascal  [Select][+][-]

1. procedure quiche_conn_stats(var conn:PQuiche_conn; var &out:PQuiche_stats); cdecl; external QUICHE_LIB; //edit: "_out" -> "&out"
2.  

Code: Pascal  [Select][+][-]

1. var
2.   Stats: TQuiche_stats;
3.   PStats: PQuiche_stats;  
4.  
5.  quiche_conn_stats(conn, PStats);
6.     WriteLn('Connection Statistics:');
7.     Writeln('SIZE='+SizeOf(Stats).ToString);
8.     WriteLn('Packets received: ', stats.recv);
9.     WriteLn('Packets sent: ', stats.sent);
10.     WriteLn('Packets lost: ', stats.lost);
11.     WriteLn('Retransmissions: ', stats.retrans);  
12.  

and it still "works like a charm" and returns the same values 

So, maybe do the same for quiche_conn_recv()?

Now I made changes and it's like you suggested but we're back to

An unhandled exception occurred at $00007F4392F4F9B1:
                                                     EAccessViolation: Access violation
                                                                                         $00007F4392F4F9B1

          --------------------------------------------------

[Fred vS]

Ok, so the good news is that using recv_info: PQuiche_recv_info makes the app less sad.

A other try with this?

Code: Pascal  [Select][+][-]

1.     quiche_conn_recv(var conn: PQuiche_Conn; var buf: PByte; var buf_len: CSize_t;
2.                    var info: PQuiche_Recv_Info): CSsize_t; cdecl; external QUICHE_LIB;

A other target to test is the result of the function and first test it alone ( without bytes_received := )
to see if the error is in CSsize_t result (should not but who knows)

Code: Pascal  [Select][+][-]

1. quiche_conn_recv(conn, @buf, 8192, recv_info);

[r.lukasiak]

A other target to test is the result of the function and first test it alone ( without bytes_received := )
to see if the error is in CSsize_t result (should not but who knows)

calling the function without bytes_received :=  unfortunately doesn't change anything.

quiche_conn_recv(var conn: PQuiche_Conn; var buf: PByte; var buf_len: CSize_t;
                   var info: PQuiche_Recv_Info): CSsize_t; cdecl; external QUICHE_LIB;

applied but it didn't change anything either :/

I need to go out in a moment and I'll be offline for almost 24h. When I'm back I'll clean the code because I have so many comments that I'm almost getting lost in the code. I'll clean it and post here entirely to have a better picture.

[TRon]

indeed... now I'm wondering what I was looking at when I was converting this...

Keep in mind that whatever you were using for the conversion might (still) be valid. I linked to trunk (current development version) while you might be basing your conversion of another version of the library. Hnece why I asked about which version of the library you are using.

and what makes me think even more, why this even worked? I was able to get the stats.... But I think it just proves that the name of the field doesn't matter, as long as the size is ok. I had fewer fields so it didn't exceed the real size. Anyway,now I corrected it and it's 112 bytes.

It does not have to be the correct size. When bigger you are left with some data that is undetermined when smaller you (the call to the library function) might be overwriting memory that is in use with your program (which is able to lead to all kinds of strange issues, from access violation to other variables declared in your program being overwritten).

The whole point of not using f.e. hard-coded Qword's is that most library headers (and the library itself) are made to work for different platforms. If you hard-code such definitions you make it fail when compiling for another target (not to mention that you never know if f.i. a record field is actually hard-coded by the library to be 32 bit or to change depending on the platform).

Thank you for sharing the order of your uses clause. I had the idea that perhaps the last included unit was overriding some of the earlier unit types (that does not appear to be the case).

BTW good that you are now able to show the same results that I have for the sizes of the records.

another BTW and it is imho. Pascalifying c-headers is all nice and dandy but for initial conversion it is a no-no. There are many c function calls that allow for a pointer to a record to be nil so that the function can act differently. Using var declarations prevent that from ever being possible. If you can't make it work in the basics using the c declarations then it will become a guessing game as of why things fail when using pascalified parameter declarations.

Either way, you would have to choose which path to follow because when I keep posting my suggestions (pure c translation) it will mess with your head when also trying Fred's suggestions (pascalified).

in any way, a pointer to a record structure (in c) is not the same as a var parameter to a pointer to a record (in pascal).

PS: is there any way I can install this library without the need to depend on rust (I really do not want that on my dev machine)

[Fred vS]

Either way, you would have to choose which path to follow because when I keep posting my suggestions (pure c translation) it will mess with your head when also trying Fred's suggestions (pascalified).

I'm just trying to help, from the most logical translation from C to Pascal.
It doesn't work so I suggested other ways but of course your path is better and I'll let you find the solution that you will surely find easily.
Good night TRon and sorry for the shadow.