Recent

Author Topic: [SOLVED] Indy > IdMappedPortTCP (and idHttpProxyServer)  (Read 8991 times)

RDL

  • Jr. Member
  • **
  • Posts: 71
[SOLVED] Indy > IdMappedPortTCP (and idHttpProxyServer)
« on: February 19, 2018, 03:18:21 pm »
Hi.
In the early versions of Indy it was like this:
TIdMappedPortContext (AContext) .NetData // (String)
Now:
TIdMappedPortContext (AContext) .NetData // (TidBytes)

Please help me. How do I get a response in string?

Sorry for my english, google translation!
« Last Edit: March 10, 2018, 03:15:45 am by RDL »
Sorry for my english, google translation!

dsiders

  • Hero Member
  • *****
  • Posts: 1045
Re: Indy > IdMappedPortTCP
« Reply #1 on: February 19, 2018, 04:41:50 pm »
Hi.
In the early versions of Indy it was like this:
TIdMappedPortContext (AContext) .NetData // (String)
Now:
TIdMappedPortContext (AContext) .NetData // (TidBytes)

Please help me. How do I get a response in string?

Sorry for my english, google translation!

There are convenience functions in IdGlobal.pas:

BytesToString()
BytesToStringRaw()

These should help.
Preview Lazarus 3.99 documentation at: https://dsiders.gitlab.io/lazdocsnext

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP
« Reply #2 on: February 20, 2018, 05:41:54 am »
Thank you. That helped!
But I can not understand why there is no response body, only headings.

Post request:

Code: Pascal  [Select][+][-]
  1. POST myURL HTTP/1.1
  2. Host: myURL
  3. User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:58.0) Gecko/20100101 Firefox/58.0
  4. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
  5. Accept-Language: ru-RU,ru;q=0.8,en-US;q=0.5,en;q=0.3
  6. Accept-Encoding: gzip, deflate
  7. Cookie: mycookies
  8. Connection: keep-alive
  9. Referer: http:/myurl
  10. Content-type: application/x-www-form-urlencoded
  11. Content-length: 1
  12.  

Post Response:

Code: Pascal  [Select][+][-]
  1. HTTP/1.1 200 OK
  2. Server: nginx/1.9.13
  3. Date: Tue, 20 Feb 2018 04:04:14 GMT
  4. Content-Type: application/octet-stream
  5. Content-Length: 697
  6. Connection: keep-alive
  7.  

I do this:

Code: Pascal  [Select][+][-]
  1. procedure TfrmMain.IdMappedPortTCPOutboundData(AContext: TIdContext);
  2. var
  3. S: string;
  4. begin
  5. S:=BytesToString(TIdMappedPortContext(AContext).NetData);
  6. frmMain.RichMemo_Log.Lines.Add(S);
  7. end;

In response, I get only headlines for all requests. Why?

Update:

Yes. If we got the data "application / octet-stream" then only the title will be in the NetData response. (Headers)
« Last Edit: February 20, 2018, 12:27:42 pm by RDL »
Sorry for my english, google translation!

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Indy > IdMappedPortTCP
« Reply #3 on: February 21, 2018, 02:56:13 am »
But I can not understand why there is no response body, only headings.

Because you are likely processing the NetData before the response body has actually been received.

You can't process the NetData as-is, let alone as strings.  It may be incomplete data on any given event.

TIdMappedPortTCP is just a passthrough of arbitrary raw bytes in both directions. It knows NOTHING about HTTP (or any other protocol that exists on top on TCP).  It simply runs a loop for the lifetime of the inbound and outbound connections, where each loop iteration checks if the 2 connections have any pending bytes waiting to be forwarded to the other connection, and if so then reads the bytes, fires the OnExecute (inbound connection) and/or OnOutboundData (outbound connection) events, in that order, and then forwards the bytes.  The events allow you to modify the bytes, but they are still just arbitrary bytes at any given moment.

If you need to process the bytes, you have to parse them yourself.  And since there is no 1:1 relationship between reads and sends in TCP, you have to buffer the NetData yourself and parse your buffer instead of the NetData itself.  That also means implementing a state machine to know what data you are parsing on any given event, as your buffer may contain a mix of partial messages and complete messages, so it may take multiple events to process any given message, or you may have to process multiple messages in a single event.

So, in this case, since you are trying to extract an HTTP server's response body, then in the OnOutboundData event, append the NetData as-is to the end of your own buffer.  EVERY TIME the OnOutBoundData event is fired, after copying the NetData into your buffer, scan your buffer for the '$0D $0A $0D $0A' byte sequence that separates HTTP headers from HTTP body, and if found then extract the HTTP headers from the buffer, take note of how the HTTP body (if present) is formatted and terminated (see RFC 2616 Section 4.4 for the rules), and then read the body data as needed, in a loop, updating your state machine on each iteration, until the buffer no longer holds a complete message.  It may potentially take multiple OnOutboundData events for you to reach the end of the headers, and/or the end of the body.  Then, and only then, can you process the headers and body as needed, and then start all over for the next HTTP response.

You had to do something to this in the old Indy version, too.  If you weren't, then you weren't using TIdMappedPortTCP correctly to begin with.

Things get a bit more complicated when you have to deal with things like SSL/TLS, HTTP pipelining, compression, etc.

If you are trying to create an HTTP proxy to capture traffic back and forth, you really should use TIdHTTPProxyServer instead of TIdMappedPortTCP.  TIdHTTPProxyServer has OnHTTPBeforeCommand, OnHTTPResponse, and OnHTTPDocument events.  Let TIdHTTPProxyServer handle the HTTP protocol for you, and provide processed data to you.
« Last Edit: February 21, 2018, 07:25:19 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP
« Reply #4 on: February 22, 2018, 10:25:35 am »
Quote
So, in this case, since you are trying to extract an HTTP server's response body, then in the OnOutboundData event, append the NetData as-is to the end of your own buffer.  EVERY TIME the OnOutBoundData event is fired, after copying the NetData into your buffer, scan your buffer for the '$0D $0A $0D $0A' byte sequence that separates HTTP headers from HTTP body, and if found then extract the HTTP headers from the buffer, take note of how the HTTP body (if present) is formatted and terminated (see RFC 2616 Section 4.4 for the rules), and then read the body data as needed, in a loop, updating your state machine on each iteration, until the buffer no longer holds a complete message.  It may potentially take multiple OnOutboundData events for you to reach the end of the headers, and/or the end of the body.  Then, and only then, can you process the headers and body as needed, and then start all over for the next HTTP response.

Can a small example of how to do this?
Please
« Last Edit: February 22, 2018, 12:52:30 pm by RDL »
Sorry for my english, google translation!

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Indy > IdMappedPortTCP
« Reply #5 on: February 22, 2018, 09:25:52 pm »
Can a small example of how to do this?

Unfortunately, an example would not be small.  I started writing one before I posted my previous reply, and it got quite long and complicated, so I decided not to post it.

Remember, TIdMappedPortTCP is giving you arbitrary amounts of data in each event.  But HTTP has structure to it, and you have to follow the structure.

For instance, to know what kind of request is being sent, or to know if the response says the request succeeded or failed, you have to read the first CRLF delimited line of the request/response.  That means receiving data until you encounter a CRLF.  The events don't guarantee that you will get a complete CRLF right away, or even that you will get the CR and LF bytes in the same event.  So, you have to continuously append event data to a separate buffer, and only when the buffer has a completed CRLF can you then extract a complete line from the buffer and process it as needed, leaving incomplete data in the buffer for the next event to append to.

Same goes with the HTTP headers following the first line.  They are also CRLF delimited.  So you have to receive, buffer, and extract completed lines until you encounter the terminating CRLF+CRLF sequence.  Only then can you process the headers, since they may be in any order, and several headers have to be analyzed before you can know how the rest of the request/response is formatted, and thus know how to extract it from your buffer (X number of bytes? formatted chunks? MIME? etc).

That kind of logic is not easy (but possible) to implement with TIdMappedPortTCP.  It requires a buffering mechanism, and a state machine that carries state information from one event to the next.  And then double this, since you have to do the same thing with both requests and responses, because part of HTTP response handling is knowing what kind of request is being responded to (in particular, if the request is HEAD or not, since HEAD responses don't have body data, even if the response headers suggest otherwise).

That is why I suggest you use TIdHTTPProxyServer instead.  Let it do all of the reading and parsing for you, and then give you events when it has complete HTTP requests/responses ready for you to process.
« Last Edit: February 22, 2018, 09:32:46 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP
« Reply #6 on: February 23, 2018, 01:43:39 am »
Made through IdHTTPPRoxyServer.

It turned out so:
Code: Pascal  [Select][+][-]
  1. procedure TfrmMain.IdHTTPProxyServerHTTPDocument(AContext: TIdHTTPProxyServerContext; var VStream: TStream);
  2. var
  3. BS: TBytesStream;
  4. S: String;
  5. begin
  6. BS:=TBytesStream.Create;
  7. BS.LoadFromStream(VStream);
  8. S:=BytesToHex(BS.Bytes);
  9. BS.Free;
  10. end;
The variable S contains the entire answer in Hex and with it I then work.
I do not know how much this is correct, but this method works and gets the whole answer.

Thank you!
Sorry for my english, google translation!

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Indy > IdMappedPortTCP
« Reply #7 on: February 23, 2018, 02:45:58 am »
Made through IdHTTPPRoxyServer.

FYI, the VStream provided by the OnHTTPDocument event is a TMemoryStream by default (the event allows you to substitute it with a different TStream object, if you want, before the data is forwarded to the next peer).  I suggest you operate on the VStream data as-is instead of making a separate copy in memory via TBytesStream.  For large documents, this makes a difference.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP
« Reply #8 on: February 23, 2018, 01:17:50 pm »
[Remy Lebeau]

Thank.  ::)

Code: Pascal  [Select][+][-]
  1. function StreamToHex(Buf: TStream): String;
  2. const
  3.   Convert: array[0..15] of Char = '0123456789ABCDEF';
  4. var
  5. i, p: Integer;
  6. B: Byte;
  7. begin
  8. SetLength(Result,Buf.Size*2);
  9. p:=Buf.Position;
  10. Buf.Position:=0;
  11. for i:=1 to Buf.Size do begin
  12.    Buf.Read(B,1);
  13.    Result[(i*2)-1]:=Convert[B shr $4];
  14.    Result[(i*2)]:=Convert[B and $F];
  15. end;
  16. Buf.Position:=p;
  17. end;
  18.  
  19. procedure TfrmMain.IdHTTPProxyServerHTTPDocument(AContext: TIdHTTPProxyServerContext; var VStream: TStream);
  20. var
  21. S: String;
  22. begin
  23. S:=StreamToHex(VStream);
  24. end;

As I understand it, is that better?

Update:

If you run online radio (streaming audio) then it does not play.
At the same time, the memory of the program starts to grow!
As I understand it because of tmFullDocument.
If I choose tmStreaming, then the HTTPDocument event does not work.
How can get out of this situation?
Sorry for my english, google translation!

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Indy > IdMappedPortTCP
« Reply #9 on: February 23, 2018, 07:17:14 pm »
As I understand it, is that better?

Almost, try this instead:

Code: Pascal  [Select][+][-]
  1. function StreamToHex(Buf: TStream): String;
  2. const
  3.   Convert: array[0..15] of Char = '0123456789ABCDEF';
  4. var
  5.   i, p, s: Int64;
  6.   B: Byte;
  7. begin
  8.   p := Buf.Position;
  9.   s := Buf.Size - p;
  10.   SetLength(Result, s * 2 * SizeOf(Char));
  11.   for i := 1 to s do begin
  12.     Buf.ReadBuffer(B, 1);
  13.     Result[(i*2)-1] := Convert[B shr $4];
  14.     Result[(i*2)] := Convert[B and $F];
  15.   end;
  16.   Buf.Position := p;
  17. end;
  18.  

Alternatively:

Code: Pascal  [Select][+][-]
  1. function StreamToHex(Buf: TStream): String;
  2. const
  3.   Convert: array[0..15] of Char = '0123456789ABCDEF';
  4. var
  5.   i, p, s: Int64;
  6.   B: Byte;
  7.   Ptr: PChar;
  8. begin
  9.   p := Buf.Position;
  10.   s := Buf.Size - p;
  11.   SetLength(Result, s * 2 * SizeOf(Char));
  12.   Ptr := PChar(Result);
  13.   while s > 0 do begin
  14.     Buf.ReadBuffer(B, 1);
  15.     Ptr^ := Convert[B shr $4]; Inc(Ptr);
  16.     Ptr^ := Convert[B and $F]; Inc(Ptr);
  17.     Dec(s);
  18.   end;
  19.   Buf.Position := p;
  20. end;
  21.  

But why do you need the data as a hex string to begin with?  What are you going to do with it that you can't do with the raw stream data?

If you run online radio (streaming audio) then it does not play.

No, you cannot use the OnHTTPDocument event with streaming media, since there is no end to the "document".  The TIdHTTPProxyServerContext.TransferMode property would have to be set to tmStreaming, which you can do dynamically in the OHTTPBeforeCommand event based on the type of request/response being sent.  But there is currently no data event fired while streaming media.

At the same time, the memory of the program starts to grow!

Right, because tmFullDocument is buffering data in memory until the end of the "document" is reached, then the OnHTTPDocument event is fired with the buffered data.  That does not work with streaming media.

As I understand it because of tmFullDocument.

Yes.

If I choose tmStreaming, then the HTTPDocument event does not work.

Correct.

How can get out of this situation?

Until a new data event is added to TIdHTTPProxyServer to handle streaming data, the only way I can think to handle this would be to assign an OnHTTPBeforeCommand event handler that looks at the AContext.Headers for the request/response, and if streaming media is detected then assign a TIdConnectionIntercept object to the AContext.Connection.Intercept (request) or AContext.OutboundClient.Intercept (response) property, and then use the TIdConnectionIntercept.OnReceive event to handle the streaming data.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP
« Reply #10 on: February 26, 2018, 03:33:11 am »
[Remy Lebeau]
Hey.
There was a new problem with idHttpProxyServer.
The example in the attachment works correctly on a Linux system, but does not work properly on Windows.
The problem is this:
Why is it that the page does not download, it seems that the program just hung and does not respond to requests. Again, this option works fine on Linux. After the program is closed, it remains hanging in the processes.
Is it an Indy error or mine?
Why can this be?

PS: Any HTTP site freezes at boot time.

Update 1:

I checked again on Linux:
On Firefox, pages are not always loaded.
On GoogleChrome works correctly.

I checked again on Windows:
IE, Firefox, GoogleChrome sometimes the pages are loaded, but most often not loaded or loaded incomplete.

Update 2:

If you wait about 15-20 minutes, then the page is loaded.
This is very long, why is this happening?
How can I speed up the work of idHttpProxyServer?

Dear, Remy Lebeau
Help please
« Last Edit: February 26, 2018, 04:42:49 pm by RDL »
Sorry for my english, google translation!

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP (and idHttpProxyServer)
« Reply #11 on: March 06, 2018, 05:34:23 pm »
Please tell me why slows down very much idHTTPProxyServer?  :'(

I have attached an example one message above
« Last Edit: March 06, 2018, 05:36:43 pm by RDL »
Sorry for my english, google translation!

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 1311
    • Lebeau Software
Re: Indy > IdMappedPortTCP (and idHttpProxyServer)
« Reply #12 on: March 06, 2018, 09:35:53 pm »
Please tell me why slows down very much idHTTPProxyServer?  :'(

I can't answer that.  You will just have to debug and profile it for yourself to find out what is going on.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP (and idHttpProxyServer)
« Reply #13 on: March 07, 2018, 01:44:42 pm »
[Remy Lebeau]
Try to open sites with .swf through idhttpproxyserver. Will it slow down?

On the Internet, there are people with the same problem ...
Is there a problem with idHttpProxyServer?
Sorry for my english, google translation!

RDL

  • Jr. Member
  • **
  • Posts: 71
Re: Indy > IdMappedPortTCP (and idHttpProxyServer)
« Reply #14 on: March 08, 2018, 10:01:10 am »
[Remy Lebeau]
Try using idHTTPProxyServer to visit http://gameswf1.weebly.com/
It will not be loaded until the end. If you go to a site without idHTTPProxyServer, then it boots up normally.

Why is this happening?  %)
« Last Edit: March 08, 2018, 10:25:58 am by RDL »
Sorry for my english, google translation!

 

TinyPortal © 2005-2018