Indy TIdHTTP & Private Proxies Issue....

[AaronCatolico1]

I'm doing a small test project that does a simple google search query through Indy and a private proxy that requires username & password. I keep getting the following error message while trying to get the HTML data back from the Google search:

"Connection closed gracefully".

Here's the code I've tried:

Code: Pascal  [Select][+][-]

1. uses
2.     Classes, SysUtils, Forms, Controls, Graphics, Dialogs, StdCtrls,
3.     IdHTTP, IdSSLOpenSSL, IdAuthentication, IdIOHandler, IdIOHandlerSocket, IdIOHandlerStack, IdServerIOHandlerStack;
4.                        
5.  
6. procedure TForm1.Button1Click(Sender: TObject);
7. var
8.   IdHTTP: TIdHTTP;
9.   SSLHandler: TIdSSLIOHandlerSocketOpenSSL;
10.   Response: TStringStream;
11.   SearchQuery, URL: string;
12. begin
13.   IdHTTP := TIdHTTP.Create(nil);
14.   SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
15.   Response := TStringStream.Create;
16.   try
17.     // Configure SSL handler
18.     SSLHandler.SSLOptions.Method := sslvTLSv1_2;
19.     IdHTTP.IOHandler := SSLHandler;
20.  
21.     // Set proxy parameters
22.     IdHTTP.ProxyParams.ProxyServer := edit1.text; //Proxy Host address (e.g. 168.1.9.169)
23.     IdHTTP.ProxyParams.ProxyPort := 50101; // Replace with your proxy port
24.     IdHTTP.ProxyParams.ProxyUsername := edit3.text;
25.     IdHTTP.ProxyParams.ProxyPassword := edit4.text;
26.  
27.     // Set user agent to mimic a browser
28.     IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';
29.  
30.     // Prepare the search query
31.     SearchQuery := 'lazarus pascal indy proxy example';
32.     URL := 'https://www.google.com/search?q=dog';
33.  
34.     // Perform the GET request
35.     IdHTTP.Get(URL, Response);
36.  
37.     // Display the response in Memo1
38.     Memo1.Lines.Text := Response.DataString;
39.   except
40.     on E: Exception do
41.       Memo1.Lines.Text := 'Error: ' + E.Message;
42.   end;
43.   // Clean up
44.   Response.Free;
45.   SSLHandler.Free;
46.   IdHTTP.Free;
47.  
48. end;        
49.  

[Remy Lebeau]

You are seeing a DEBUGGER message.  The debugger sees all exceptions before your app's code does. Just resume execution and TIdHTTP should catch that exception internally. It should not be escaping into your normal code at runtime.  If you don't want to see the exception in the debugger, you can configure the debugger to ignore it.

[AaronCatolico1]

Right, but how do I get the HTML from the IdHTTP into memo1? Why would there be any "connection closed gracefully" in the first place if my code was written correctly?

You are seeing a DEBUGGER message.  The debugger sees all exceptions before your app's code does.  TIdHTTP should catch that exception internally, it should not be escaping into your normal code at runtime.  If you don't want to see the exception in the debugger, you can configure the debugger to ignore it.

[CCRDude]

Your issue is a bit vague. Most simply answer would be: ignore the exception, and it probably should appear in Memo1.Lines .

[Remy Lebeau]

Right, but how do I get the HTML from the IdHTTP into memo1?

What you already have is fine for that. However, TIdHTTP.Get() has overloads which return a String, you don't need the Response stream, eg:

Code: Pascal  [Select][+][-]

1. Memo1.Text := IdHTTP.Get(URL);

Why would there be any "connection closed gracefully" in the first place if my code was written correctly?

Because HTTP allows for that. EIdConnClosedGracefully happens when Indy tries to read more data from the socket after the peer has closed the connection. For HTTP, that means the server has either closed the connection prematurely before it has sent the end of the response, or the server is using a socket closure as the end-of-response indicator rather than using the Content-Length or Transfer-Encoding response headers. You would have to analyze the traffic to see what is really happening.

[AaronCatolico1]

Well, I ignored the exception & still nothing comes back while trying to tunnel through a private proxy.
As soon as I get rid of the the following proxy lines, the html data actually appears in the memo1:

Code: Pascal  [Select][+][-]

1.     IdHTTP.ProxyParams.ProxyServer := edit1.text;
2.     IdHTTP.ProxyParams.ProxyPort := 50101; // Replace with your proxy port
3.     IdHTTP.ProxyParams.ProxyUsername := edit3.text;
4.     IdHTTP.ProxyParams.ProxyPassword := edit4.text;
5.  

It seems to only work when I turn off the proxy configurations. I'd like to keep this as an option while connecting to any websites. Any idea why I'm not able to connect through this private proxy? BTW, I know that this private proxy is working because I've made the configurations on my pc which shows that I am indeed able to connect to Google, YouTube, this forum, etc.. The problem has got to be with Indy or the way that I've written the code, just not sure exactly how it's supposed to be implemented.

Your issue is a bit vague. Most simply answer would be: ignore the exception, and it probably should appear in Memo1.Lines .

[AaronCatolico1]

I still keep getting the "connection closed gracefully" while trying to use the private proxy. I turn it off & then nothing comes back at all.
Here's the updated code I'm using now:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. var
3.   IdHTTP: TIdHTTP;
4.   SSLHandler: TIdSSLIOHandlerSocketOpenSSL;
5.   URL: string;
6. begin
7.   IdHTTP := TIdHTTP.Create(nil);
8.   SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(nil);
9.   try
10.     // Configure SSL handler
11.     SSLHandler.SSLOptions.Method := sslvTLSv1_2;
12.     IdHTTP.IOHandler := SSLHandler;
13.  
14.     // Set proxy parameters
15.     IdHTTP.ProxyParams.ProxyServer := edit1.text;
16.     IdHTTP.ProxyParams.ProxyPort := strToInt(edit2.text); // Replace with your proxy port
17.     IdHTTP.ProxyParams.ProxyUsername := edit3.text;
18.     IdHTTP.ProxyParams.ProxyPassword := edit4.text;
19.  
20.     // Set user agent to mimic a browser
21.     IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';
22.  
23.     // Prepare the search query
24.     URL := 'https://www.google.com/search?q=dog';
25.  
26.     //Display the html data from google
27.     Memo1.Text := IdHTTP.Get(URL);
28.  
29.   except
30.     on E: Exception do
31.       showmessage('Error: ' + E.Message);
32.   end;
33.   // Clean up
34.  
35.   SSLHandler.Free;
36.   IdHTTP.Free;
37.  
38. end;
39.  

How would I "analyze the traffic to see what is really happening"? You mean using a program like wireshark or fiddler? If so, what exactly am I looking for? Even when I find it, what changes would most likely need to be made to the code I'm using above? I've use plenty of programming languages in the past such as Python, PHP, Visual Basic, Javascript, etc., & have never had such difficulty trying to use any library with proxies. I'm not sure why it's so difficult to do in Pascal. It's usually just plug n play in all other languages & libraries. I'm not trying to be rude, I'm just being honest that I've never had this much difficulty to do something that is usually very easy to accomplish in other languages.

Right, but how do I get the HTML from the IdHTTP into memo1?

What you already have is fine for that. However, TIdHTTP.Get() has overloads which return a String, you don't need the Response stream, eg:

Code: Pascal  [Select][+][-]

1. Memo1.Text := IdHTTP.Get(URL);

Because HTTP allows for that. EIdConnClosedGracefully happens when Indy tries to read more data from the socket after the peer has closed the connection. For HTTP, that means the server has either closed the connection prematurely before it has sent the end of the response, or the server is using a socket closure as the end-of-response indicator rather than using the Content-Length or Transfer-Encoding response headers. You would have to analyze the traffic to see what is really happening.

[Remy Lebeau]

Well, I ignored the exception & still nothing comes back while trying to tunnel through a private proxy.

What kind of proxy are you using, exactly?  Is it an HTTP proxy, a SOCKS proxy, something else, etc?  The ProxyParams feature only works with HTTP proxies.  If you need to support other kinds of proxies, you have to use the IOHandler's TransparentProxy feature instead.

It seems to only work when I turn off the proxy configurations. I'd like to keep this as an option while connecting to any websites. Any idea why I'm not able to connect through this private proxy?

Not without more detailed information.  The ProxyParams feature has been available for a VERY LONG time, so I know it works under normal conditions.

Can you provide a capture of the raw socket traffic and the HTTP traffic?

BTW, I know that this private proxy is working because I've made the configurations on my pc which shows that I am indeed able to connect to Google, YouTube, this forum, etc..

That doesn't negate the possibility that the proxy may be something that is unsupported by the ProxyParams feature.  There are many different kinds of proxies.  I need more details to diagnose the issue.

The problem has got to be with Indy or the way that I've written the code, just not sure exactly how it's supposed to be implemented.

The code shown is fine, provided the proxy is a standard HTTP proxy.

I still keep getting the "connection closed gracefully" while trying to use the private proxy.

Without seeing the HTTP/socket traffic, there is no way to diagnose that.

I turn it off & then nothing comes back at all.

Then that is a completely different issue.

How would I "analyze the traffic to see what is really happening"? You mean using a program like wireshark or fiddler?

Yes, if the problem turns out to be at the socket/proxy level.  If the problem is at the HTTP level, you can capture the HTTP requests/responses by assigning one of Indy's TIdLog... components to the TIdHTTP.Intercept property.

If so, what exactly am I looking for?

At what point exactly the socket closure is actually occurring in relation to the rest of the traffic.  For instance, is the socket closure intentional at the HTTP level (ie, there is a Connection: close response header and no Connect-Length or Transfer-Encoding header)? Or, is it happening without any HTTP response being sent at all? etc.  Maybe the proxy doesn't like the request and is forcing the connection closed.  OR maybe there is a firewall blocking the connection.  There is just not enough information yet to know what is really going on.

Even when I find it, what changes would most likely need to be made to the code I'm using above?

I can't answer that without first identifying what the root cause actually is.

I've use plenty of programming languages in the past such as Python, PHP, Visual Basic, Javascript, etc., & have never had such difficulty trying to use any library with proxies. I'm not sure why it's so difficult to do in Pascal. It's usually just plug n play in all other languages & libraries. I'm not trying to be rude, I'm just being honest that I've never had this much difficulty to do something that is usually very easy to accomplish in other languages.

Are you really complaining about Pascal in general, or just about Indy specifically?

Those other languages you mention are more higher-level, whereas Pascal in general is more lower-level.  High-level languages are designed for simplicity, so they tend to provide more wrappers and less complexity.  Pascal tends to be more flexible, and sometimes that means you need to piece multiple isolated components together to get the same tasks done.

[AaronCatolico1]

What kind of proxy are you using, exactly?  Is it an HTTP proxy, a SOCKS proxy, something else, etc?  The ProxyParams feature only works with HTTP proxies.  If you need to support other kinds of proxies, you have to use the IOHandler's TransparentProxy feature instead.

- Gambit

The private proxy I ordered is in IPV4 format which requires username & password authentication. It supports both SOCKS5 and HTTP protocols depending on the port at which I try to access it. I've tried both ports to no avail.

If you'd like (and I personally don't mind because I know that you're a trustworthy guy), I could send you the login credentials for you to do a quick test with the code I used above to see if you can quickly find where the issue is on the network. I could send you a quick PM with the login credentials, if you don't mind testing it. It shouldn't take long at all. I was given permission by proxy-seller . com to access this proxy for up to 3 devices.

PM Sent!

[AaronCatolico1]

@Gambit

Here's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy:

Code: Pascal  [Select][+][-]

1. CONNECT www.google.com:443 HTTP/1.1
2. Pragma: no-cache
3. Proxy-Connection: keep-alive
4. Host: www.google.com
5. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
6. User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3
7.  
8.  
9. HTTP/1.1 407 Proxy Authentication Required
10. Date: Mon, 16 Dec 2024 19:15:52 GMT
11. Proxy-Authenticate: Basic realm="proxy"
12. Connection: close
13. Content-Length: 0
14.  
15.  
16. GET /search?q=dog HTTP/1.1
17. Host: www.google.com
18. Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
19. User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3
20. Proxy-Authorization: Basic *HIDDEN==
21.  
22.  
23.  

[Remy Lebeau]

The private proxy I ordered is in IPV4 format which requires username & password authentication. It supports both SOCKS5 and HTTP protocols depending on the port at which I try to access it. I've tried both ports to no avail.

The TIdHTTP.ProxyParams does not support SOCKS.  For that, you must use a separate TIdSocksInfo component and assign it to the IOHandler's TransparentProxy property when you want the connection to go through a SOCKS proxy, eg:

Code: Pascal  [Select][+][-]

1. procedure TForm1.Button1Click(Sender: TObject);
2. var
3.   IdHTTP: TIdHTTP;
4.   SSLHandler: TIdSSLIOHandlerSocketOpenSSL;
5.   IdSocks: TIdSocksInfo;
6.   URL: string;
7. begin
8.   try
9.     IdHTTP := TIdHTTP.Create(nil);
10.     try
11.       // Configure SSL handler
12.       SSLHandler := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);
13.       SSLHandler.SSLOptions.Method := sslvTLSv1_2;
14.       IdHTTP.IOHandler := SSLHandler;
15.      
16.       // Set proxy parameters
17.       if UseSOCKSProxy then // <-- add this option to your UI!
18.       begin
19.         IdSocks := TIdSocksInfo.Create(SSLHandler);
20.         IdSocks.Authentication := saUsernamePassword;
21.         IdSocks.Host := Edit1.Text;
22.         IdSocks.Port := StrToInt(Edit2.Text);
23.         IdSocks.Username := Edit3.Text;
24.         IdSocks.Password := Edit4.Text;
25.         IdSocks.Version := svSocks5;
26.         SSLHandler.TransparentProxy := IdSocks;
27.       end
28.       else if UseHTTPProxy then // <-- add this option to your UI!
29.       begin
30.         IdHTTP.ProxyParams.ProxyServer := Edit1.Text;
31.         IdHTTP.ProxyParams.ProxyPort := StrToInt(Edit2.Text);
32.         IdHTTP.ProxyParams.ProxyUsername := Edit3.Text;
33.         IdHTTP.ProxyParams.ProxyPassword := Edit4.Text;
34.       end;
35.      
36.       // Set user agent to mimic a browser
37.       IdHTTP.Request.UserAgent := 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3';
38.      
39.       // Prepare the search query
40.       URL := 'https://www.google.com/search?q=dog';
41.      
42.       //Display the html data from google
43.       Memo1.Text := IdHTTP.Get(URL);    
44.     finally
45.       // Clean up    
46.       IdHTTP.Free;
47.     end;
48.   except
49.     on E: Exception do
50.       ShowMessage('Error: ' + E.Message);
51.   end;
52. end;

Here's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy:

That does not look right.  After the 407 reply, TIdHTTP should have send another CONNECT request with the Proxy-Authorization header added to it. It should not be sending the GET request (without the Proxy-Authorization header) until after CONNECT has returned a 200 reply first.  I will have to dig into the code to see what's happening.

[Remy Lebeau]

Here's what I've come up with from WireShark as far as the TCP Stream goes while trying to connect through the private proxy:

That does not look right.  After the 407 reply, TIdHTTP should have send another CONNECT request with the Proxy-Authorization header added to it. It should not be sending the GET request (without the Proxy-Authorization header) until after CONNECT has returned a 200 reply first.  I will have to dig into the code to see what's happening.

Nevermind, I see the problem.  You don't have the hoInProcessAuth flag enabled in the TIdHTTP.HTTPOptions property, so when connecting to the HTTP proxy TIdHTTP ignores the 407 reply and moves on with sending the GET request.  If you enable the flag, then TIdHTTP will process the 407 to login to the proxy before sending the GET.

[AaronCatolico1]

@Remy Lebeau
Thanks again Remy! My apologies for such a delay in response time. I was out of state, then came back with the flu. I've been out for a while now. It seems like you've solved the issues here. Just wanted to thank you again for your usual awesome support. Happy holidays to you as well!