[Solved] - Retrieving XML file with Indy + SSL

[qoxop]

Hello,

I can establish the SSL connection through HTTPS:// with indy 10.  I can retrieve normal HTML. The server support as a response page  XML file. e.g. when in URL is additional parameter (https://someurl.somedomain/?format=XML). It is working ok in web browser - you see XML. When I try it with indy, server response is 500 - Internal server error.
When I played a little bit with idHTTP.pas, it looks like there is a problems with header, which is sent to server.... Have you any experience with similar problems?

[Phil]

Have you any experience with similar problems?

Yes, I've seen this kind of problem, where a URL works fine in all browsers, but not via a programmed HTTP request, returning a server error.

What OS(s) are you working with?

Try the same request with Synapse and FPC's HTTP client. If those work (or don't work), you'll know something more about the problem.

You could also try curl (or nscurl on Mac) to get more data points about the problem.

[qoxop]

I'm working on windows 7/64bit and windows 7/32bit. But program is compiled allways as 32 bit.
I tried download it with wget under windows, and it worked OK. So I can bypass this problem with wget. (And I'm forced to bypass it ) But I like to know why has indy a problem with it. It looks like the problem is only with HTTPS connections. I tried latest libeay32.dll ssleay32.dll. without succeed. Also tested on different network places (providers), so it is not a network problem.

[Phil]

But I like to know why has indy a problem with it.

Again, try Synapse and FPC's HTTP client. I've seen lots of sites, mostly https, which work with one or two HTTP clients, but not all of them.

[qoxop]

Thank you very much!
I tried synapse, and it works like a charm XML file is retrieved. I think I will use synapse.

Neverthless the question is still open.... I preffer rather Indy and it seems, there is some kind bug or somethig like that...

[Remy Lebeau]

I like to know why has indy a problem with it.

That is hard to diagnose without having the real URL you are requesting so it can be tested with.  The code you have shown is fine in general, so it is likely to be a server-side issue instead.  For instance, some web servers are UserAgent-aware, and have problems with TIdHTTP's default UserAgent value.  Try changing the TIdHTTP.Request.UserAgent property (or global GIdDefaultUserAgent variable) to mimic a real web browser's UserAgent.

It looks like the problem is only with HTTPS connections. I tried latest libeay32.dll ssleay32.dll. without succeed. Also tested on different network places (providers), so it is not a network problem.

HTTP vs HTTPS has no effect on how TIdHTTP create its HTTP requests.  The fact that you are getting a 500 response at all means that HTTPS is working fine.  The problem is simply in how the server is processing the HTTP request.

[Remy Lebeau]

I tried synapse, and it works like a charm XML file is retrieved. I think I will use synapse.

Neverthless the question is still open.... I preffer rather Indy and it seems, there is some kind bug or somethig like that...

I suggest you look at the actual HTTP requests being sent, see what differences there are between what Synapse sends vs what Indy sends vs what a web browser sends.  Most modern web browsers have a developer mode for capturing HTTP traffic, even over HTTPS.  Indy has Intercept classes that can be used for logging HTTP/S traffic as well.  Not sure if Synapse provides anything, you might have to resort to using a debugging proxy like Fiddler.

[qoxop]

I used latest svn Indy snapshot from https://indy.fulgan.com/ instead of official release of Indy 10.
It seems I had outdated version of Indy.
Everything works perfectly now. Without any problems.
Thank you very much Remy.

[Remy Lebeau]

I used latest svn Indy snapshot from https://indy.fulgan.com/ instead of official release of Indy 10.
It seems I had outdated version of Indy.

What "official release" are you referring to?  10.0.52?  Indy hasn't had an "official release" in years, only daily SVN snapshots (not counting the revisions that shipped with each Delphi release - Indy's SVN has tagged which revision went into each Delphi release).

[qoxop]

What "official release" are you referring to?  10.0.52?

I took it from
http://www.indyproject.org/Sockets/fpc/index.EN.aspx
There is a link - indy-10.2.0.3.zip - I thought it is an official release. Yes, it seems to be outdated, therefore I thought it's abandoned - what will be very pity, because indy is a great project.
I considered SVN only as a non stable - intermediate development version. But now I see it is worth to have the latest version.
BTW, it will be good to create from time to time some "offical" release on http://www.indyproject.org to avoid confusion.

[Remy Lebeau]

I took it from
http://www.indyproject.org/Sockets/fpc/index.EN.aspx
There is a link - indy-10.2.0.3.zip - I thought it is an official release.

No, not even close.  That is an extremely old page and version.  The page even says so:

An older code from the version control system that was organized into a sensible distribution.  This was updated on 11/13/2007.

The current version is 10.6.2 (SVN rev 5428 at the time of this writing).  FPC support was rolled into the main Indy codebase many years ago.

Yes, it seems to be outdated, therefore I thought it's abandoned

That old version, yes.  Not the latest version.

BTW, it will be good to create from time to time some "offical" release on http://www.indyproject.org to avoid confusion.

Really, these old and outdated pages need to be removed from the site.  People keep finding them and think they are up-to-date when they are really not.  The whole site is outdated in general.  Work has started to bring it up-to-date, but it is going to take awhile.