Recent

Author Topic: Indy10 TIdHTTP gzip issue  (Read 1504 times)

asdf121

  • New member
  • *
  • Posts: 23
Indy10 TIdHTTP gzip issue
« on: November 14, 2018, 09:03:50 pm »
Hello,
recently I noticed that gzip does not work with TIdHTTP anymore. It just returns garbage.

Code: Pascal  [Select]
  1. program IndyGZipIssue;
  2.  
  3. //{$DEFINE STATICLOAD_ZLIB}
  4. //{$UNDEF STATICLOAD_ZLIB}
  5.  
  6. {$IFDEF FPC}
  7.         {$MODE Delphi}
  8. {$ENDIF}
  9.  
  10. uses
  11.     {$IFDEF MSWINDOWS} Windows, {$ENDIF}
  12.     {$IFNDEF MSWINDOWS}cthreads, cmem,{$ENDIF} SysUtils, Classes,
  13.                 IdHTTP, IdURI, IdSSLOpenSSL, IdCompressorZLib, IdExceptionCore, StrUtils, IdSSLOpenSSLHeaders,
  14.                 IdGlobal, IdZLibHeaders, IdZLibCompressorBase;
  15.  
  16. Var
  17.   IdHTTP: TIdHTTP;
  18.   IdSSLIOHandlerSocketOpenSSL: TIdSSLIOHandlerSocketOpenSSL;
  19.   EncURL: String;
  20.   Str: String;
  21.   MyText: TStringlist;
  22.  
  23. begin
  24.     writeln('Start!');
  25.        
  26.         EncURL := TIdURI.URLEncode('https://indy.fulgan.com/');
  27.  
  28.         writeln(EncURL);
  29.        
  30.   try
  31.     IdSSLOpenSSLHeaders.Load;
  32.   except
  33.     on e: EIdOSSLCouldNotLoadSSLLibrary do
  34.     begin
  35.       writeln(Format('Loading OpenSSL failed: %s', [IdSSLOpenSSLHeaders.WhichFailedToLoad]));
  36.       exit;
  37.     end;
  38.     on e: Exception do
  39.     begin
  40.       writeln(Format('[EXCEPTION] OpenSSL: %s %s', [e.ClassName, e.Message]));
  41.       exit;
  42.     end;
  43.   end;
  44.   writeln('OpenSSL loaded!');
  45.  
  46.        
  47.         try
  48.     IdHTTP := TIdHTTP.Create(nil);
  49.     try
  50.       with IdHTTP do
  51.       begin
  52.         HandleRedirects := True;
  53.         RedirectMaximum := 5;
  54.         ConnectTimeout := 5000;
  55.         ReadTimeout := 5000;
  56.  
  57.         // compression
  58.                 try
  59.                         Compressor := TIdCompressorZLib.Create(IdHTTP);
  60.                 except
  61.                         on e: Exception do
  62.                         begin
  63.                           writeln(Format('[EXCEPTION] TIdCompressorZLib Create: %s', [e.Message]));
  64.                         end;
  65.                   end;
  66.                  
  67.                 if Compressor = nil then
  68.                         writeln(Format('Compressor is nil!', []));
  69.                  
  70.         //Request.AcceptEncoding := 'gzip';
  71.                 Request.AcceptEncoding := 'deflate';
  72.       end;
  73.  
  74.       // secure connection
  75.       {$IFNDEF UNICODE}
  76.         if AnsiStartsText('https', EncURL) then
  77.       {$ELSE}
  78.          if StartsText('https', EncURL) then
  79.       {$ENDIF}
  80.       begin
  81.                 writeln(Format('Using secure connection!', []));
  82.                  try
  83.           IdSSLIOHandlerSocketOpenSSL := TIdSSLIOHandlerSocketOpenSSL.Create(IdHTTP);
  84.           IdHTTP.IOHandler := IdSSLIOHandlerSocketOpenSSL;
  85.                  except
  86.                         on e: Exception do
  87.                         begin
  88.                           writeln(Format('[EXCEPTION] HTTP GET %s : %s', [EncURL, e.Message]));
  89.                         end;
  90.                   end;
  91.       end;
  92.  
  93.         with IdHTTP do
  94.         begin
  95.           try
  96.             Str := Get(EncURL);
  97.           except
  98.             on e: Exception do
  99.             begin
  100.               writeln(Format('HTTP GET: %s failed: %d ec --- %s.', [EncURL, ResponseCode, ResponseText]));
  101.               writeln(Format('ClassName: %s <--> Exception: %s', [e.ClassName, e.Message]));
  102.             end;
  103.           end;
  104.                 end;
  105.     finally
  106.       IdHTTP.Free;
  107.     end;
  108.   except
  109.     on e: Exception do
  110.     begin
  111.       writeln(Format('[EXCEPTION] HTTP GET %s : %s', [EncURL, e.Message]));
  112.     end;
  113.   end;
  114.  
  115.   MyText:= TStringlist.Create;
  116.   try
  117.     MyText.Text := Str;
  118.     MyText.SaveToFile('http.txt');
  119.   finally
  120.     MyText.Free
  121.   end; {try}
  122.  
  123.     writeln('End!');
  124. end.

Compiled with:
Code: [Select]
Free Pascal Compiler version 3.2.0-beta-r40097 [2018/11/10] for x86_64

fpc -MDelphi -O3 -FuC:\Users\asdf\Downloads\Indy10_5485\Lib\* C:\Users\asdf\Downloads\http.pas

You can easily test with changing
Code: Pascal  [Select]
  1. //Request.AcceptEncoding := 'gzip'; <-- does not work
  2. Request.AcceptEncoding := 'deflate'; <-- works fine

I also tested to define one of
Code: Pascal  [Select]
  1. //{$DEFINE STATICLOAD_ZLIB}
  2. //{$UNDEF STATICLOAD_ZLIB}
but it also has no effect at all.

Program output is
Code: [Select]
Start!
https://indy.fulgan.com/
OpenSSL loaded!
Using secure connection!
End!

As far as I can say it appears also on Linux.

Regards.

PS: Will Indy support brotli sooner or later?
« Last Edit: November 15, 2018, 09:58:56 am by asdf121 »

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #1 on: November 17, 2018, 12:15:47 pm »
Seems I've fixed it at least on Windows - got the 32-bit .dll...  :-[
Does the static linking also work with FPC? If yes, how do I do that? Static linking only works on Windows, no? How can I see if it's static or dynamic linked?

But the issue on linux still exist, maybe it's related to the change of SSLDLLVers, Bug fix for HackLoad() and Updating various DLL related functions where the last one says
Quote
// TODO: setup this array more like the SSLDLLVers arrays in the IdSSLOpenSSLHeaders unit...
As far as I can say: with an older Indy rev 5448 it's still working fine.

EDIT:
Why does
Quote
TIdCompressorZLib.Create
not raise an exception? It creates the object but when using gzip, the internal check does only check if Compressor is not nil which is also true if loading zlib failed -> returning garbage.
« Last Edit: November 17, 2018, 12:21:46 pm by asdf121 »

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #2 on: November 17, 2018, 09:21:51 pm »
The issue was introduced between Fix for TIdIMAP4Server.ProcessStore() not processing FLAGS arguments correctly and Updating various DLL related functions to use THandle instead of HMODULE. With the first one my old existing code does still works fine, with one of the newer versions it does not.
I haven't tested all commits between because the ones I tried weren't able to compile with FPC.

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy10 TIdHTTP gzip issue
« Reply #3 on: November 19, 2018, 10:01:14 pm »
recently I noticed that gzip does not work with TIdHTTP anymore. It just returns garbage.

Your test code works fine for me on Windows, at least (I'm not a Linux user).

//{$DEFINE STATICLOAD_ZLIB}
//{$UNDEF STATICLOAD_ZLIB}

(Un)defining that in your own code has no effect on Indy.  It needs to be (un)defined in Indy's IdCompilerDefines.inc file instead, and then recompile Indy.

//Request.AcceptEncoding := 'gzip';
Request.AcceptEncoding := 'deflate';

Note that you cannot force TIdHTTP to use only deflate when a Compressor is assigned.  If the Compressor is loaded successfully (its IsReady property is true), TIdHTTP will automatically add gzip and/or deflate as needed to the Request.AcceptEncoding if they are not already present, thus allowing the server to choose between them.

Because TIdHTTP auto-sets the Request.AcceptEncoding if the Compressor is ready, you should not set the AcceptEncoding manually at all, or at least not for gzip/deflate specifically.  You should let TIdHTTP enable gzip/deflate for you only if the Compressor is able to handle them.

That is likely the root cause of your problem.  The Compressor is not ready (for whatever reason), so TIdHTTP does not update the Request.AcceptEncoding, so your manual setting gets passed as-is to the server, which then sends back a compressed response (not garbage) that TIdHTTP does not auto-decompress for you.

So, either figure out why the Compressor is not loading correctly, or else leave the Compressor set to nil, set the Request.AcceptEncodings as needed, and then handle the decompression manually after the compressed response is returned to you.

PS: Will Indy support brotli sooner or later?

Probably not, at least not anytime soon.  But there is nothing stopping you from implementing your own Compressor class for it, and then use it with TIdHTTP.  And if you get something working, you can always submit it for review for a possible future inclusion.

Does the static linking also work with FPC?

I don't know.  It was designed for Delphi.

If yes, how do I do that? Static linking only works on Windows, no?

I suppose as long as FPC can consume Indy's ZLIB obj files, then it should work, at least on Windows.

How can I see if it's static or dynamic linked?

There is nothing readily exposed to check that at runtime.  But, at compile-time, you can probably use {$IF DECLARED(...)} to check if the IdZlibHeaders.pas unit declares the EIdZLibStubError class or not.  It is declared only when dynamic linking.

But the issue on linux still exist, maybe it's related to the change of SSLDLLVers, Bug fix for HackLoad() and Updating various DLL related functions

It should not be related, since those changes do not affect how Indy uses Zlib.

where the last one says
Quote
// TODO: setup this array more like the SSLDLLVers arrays in the IdSSLOpenSSLHeaders unit...

The changes I made for handling OpenSSL DLLs do not apply to ZLib DLLs.  So, if ZLib was loading correctly before, it should still load OK now, I did not change that logic.

Why does
Quote
TIdCompressorZLib.Create
not raise an exception?

Because it is not supposed to, in part because the constructor is not what loads ZLib at runtime, the TIdCompressorZLib.IsReady property getter procedure does.

Otherwise, you can pre-load ZLib by calling IdZLibHeaders.Load() manually before using TIdHTTP, similar to how you are calling IdSSLOpenSSLHeaders.Load().

However, note that both IdSSLOpenSSLHeaders.Load() and IdZLibHeaders.Load() do not raise an exception on load failure, they return Boolean values instead, eg:

Code: [Select]
if not IdSSLOpenSSLHeaders.Load then
begin
  writeln(Format('Loading OpenSSL failed: %s', [IdSSLOpenSSLHeaders.WhichFailedToLoad]));
  exit;
end;
writeln('OpenSSL loaded!');

if not IdZLibHeaders.Load then
begin
  writeln('Loading ZLib failed');
  // there is no WhichFailedToLoad() function for ZLib!  Individual ZLib functions will
  // raise an EIdZLibStubError if they cannot be loaded from the DLL when called...
  exit;
end;
writeln('ZLib loaded!');

when using gzip, the internal check does only check if Compressor is not nil which is also true if loading zlib failed -> returning garbage.

TIdHTTP checks the Compressor.IsReady property before updating the Request.AcceptEncoding, or decompressing a compressed response.
« Last Edit: November 19, 2018, 10:39:44 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #4 on: November 20, 2018, 09:38:41 pm »
recently I noticed that gzip does not work with TIdHTTP anymore. It just returns garbage.

Your test code works fine for me on Windows, at least (I'm not a Linux user).
Yes, on windows it's working fine for me too now - was just an issue with 32/64bit zlib.

//{$DEFINE STATICLOAD_ZLIB}
//{$UNDEF STATICLOAD_ZLIB}

(Un)defining that in your own code has no effect on Indy.  It needs to be (un)defined in Indy's IdCompilerDefines.inc file instead, and then recompile Indy.
It's only defined in IdZLibConst and IdZLibHeaders, seems it's does even not support FPC at all. First statement is always $UNDEF.
Btw, why are the included zlib linking files so ancient?

//Request.AcceptEncoding := 'gzip';
Request.AcceptEncoding := 'deflate';

Note that you cannot force TIdHTTP to use only deflate when a Compressor is assigned.  If the Compressor is loaded successfully (its IsReady property is true), TIdHTTP will automatically add gzip and/or deflate as needed to the Request.AcceptEncoding if they are not already present, thus allowing the server to choose between them.

Because TIdHTTP auto-sets the Request.AcceptEncoding if the Compressor is ready, you should not set the AcceptEncoding manually at all, or at least not for gzip/deflate specifically.  You should let TIdHTTP enable gzip/deflate for you only if the Compressor is able to handle them.

That is likely the root cause of your problem.  The Compressor is not ready (for whatever reason), so TIdHTTP does not update the Request.AcceptEncoding, so your manual setting gets passed as-is to the server, which then sends back a compressed response (not garbage) that TIdHTTP does not auto-decompress for you.

So, either figure out why the Compressor is not loading correctly, or else leave the Compressor set to nil, set the Request.AcceptEncodings as needed, and then handle the decompression manually after the compressed response is returned to you.
Okay, I'll remove my Request.AcceptEncodings line and let TIdHTTP decide.

But the issue on linux still exist, maybe it's related to the change of SSLDLLVers, Bug fix for HackLoad() and Updating various DLL related functions

It should not be related, since those changes do not affect how Indy uses Zlib.

where the last one says
Quote
// TODO: setup this array more like the SSLDLLVers arrays in the IdSSLOpenSSLHeaders unit...

The changes I made for handling OpenSSL DLLs do not apply to ZLib DLLs.  So, if ZLib was loading correctly before, it should still load OK now, I did not change that logic.
Well, you changed the function to load a library which is also used for zlib. I only can tell you what I noticed, with older Indy version it loads it correctly and with a newer one TIdHTTP fails to decompress gzip but uses gzip (own hardcoded Request.AcceptEncodings removed). I don't know where to search exactly but there must be an issue somewhere.

However, note that both IdSSLOpenSSLHeaders.Load() and IdZLibHeaders.Load() do not raise an exception on load failure, they return Boolean values instead, eg:
Thanks for the hint, didn't knew before. Only noticed that IdSSLOpenSSLHeaders.Load() raises an exception when OpenSSL library not found. Guessed IdZLibHeaders.Load() would do the same.

when using gzip, the internal check does only check if Compressor is not nil which is also true if loading zlib failed -> returning garbage.

TIdHTTP checks the Compressor.IsReady property before updating the Request.AcceptEncoding, or decompressing a compressed response.
I'll test again with checks for Compressor.IsReady which should be true, else it shouldn't ask for gzip at all, correct? And without gzip enabled and zlib not working, I shouldn't receive garbage...

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy10 TIdHTTP gzip issue
« Reply #5 on: November 24, 2018, 12:49:28 am »
It's only defined in IdZLibConst and IdZLibHeaders, seems it's does even not support FPC at all. First statement is always $UNDEF.

Well, then I guess that answers your earlier question about whether FPC is supported.  Static linking is apparently setup only for Delphi.  If you have tweaks that will work for FPC, feel free to submit them for review.

Btw, why are the included zlib linking files so ancient?

Because nobody has gotten around to updating them.  I can't compile the zlib obj files myself.  And Indy was never updated to take advantage of Delphi's or FPC's own zlib wrappers (it is still on the todo list).

Okay, I'll remove my Request.AcceptEncodings line and let TIdHTTP decide.

Please do.

Well, you changed the function to load a library which is also used for zlib.

I only can tell you what I noticed, with older Indy version it loads it correctly and with a newer one TIdHTTP fails to decompress gzip but uses gzip (own hardcoded Request.AcceptEncodings removed). I don't know where to search exactly but there must be an issue somewhere.

Again, the changes I made for how OpenSSL is loaded should not have any effect on how ZLib is loaded.  You are going to have to be more specific about how you think the loading logic has changed for ZLib between the older version to the newer version.  Step through the code with the debugger and see exactly how each version loads the DLL. I have no way of testing or debugging it myself, as I don't use Linux or FPC.

Thanks for the hint, didn't knew before. Only noticed that IdSSLOpenSSLHeaders.Load() raises an exception when OpenSSL library not found.

No, higher-level code (in TIdSSLIOHandlerSocketOpenSSL.SetPassThrough() and TIdSSLContext.Create()) is what raises the exception if IdSSLOpenSSLHeaders.Load() fails.  Load() itself does not raise the exception, and never has.  Same with IdZLibHeaders.Load().

I'll test again with checks for Compressor.IsReady which should be true, else it shouldn't ask for gzip at all, correct?

If you leave out gzip and deflate from the Request.AcceptEncoding property, then yes.  If you do not manually request compression, and the Compressor is not ready, then TIdHTTP will not request compression.

And without gzip enabled and zlib not working, I shouldn't receive garbage...

It is not garbage, it is compression.  There is a difference.
« Last Edit: November 24, 2018, 12:54:52 am by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

Mo0211

  • Newbie
  • Posts: 2
Re: Indy10 TIdHTTP gzip issue
« Reply #6 on: November 28, 2018, 10:10:52 pm »
Hey Remy,

i analyzed this behavior and got it working on FPC:

there are several issues in this case:

First one in hackload and LoadLibVer functions:

Code: Pascal  [Select]
  1. ZLib: TLibHandle = 0;

the return-type on linux FPC is a TLibHandle-Object.
So, the return kinda overflows in your code (value of -40XXXXXXXXXXXX).

The next issue is the Handling in IdZlibHeaders:

The loading of libz-library is dynamic implemented.
But it checks of Handle-Value of <> 0 - if the Value owerflowed then it has a high negative value.

Code: Pascal  [Select]
  1.   if High(ALibVersions) > -1 then begin
  2.     Result := NilHandle;
  3.     for i := Low(ALibVersions) to High(ALibVersions) do
  4.     begin
  5.       Result := LoadLibVer(ALibVersions[i]);
  6.       if Result > NilHandle then begin
  7.         Break;
  8.       end;
  9.     end;
  10.   end else begin
  11.     Result := LoadLibVer('');

i'm not sure if the dynamically loading works - i tried to change it in loading only ''.

the library is named libz per default in linux (not zlib).

I hope this is a help for you to review and fix it

Thanks and good evening!

edit: Forgot to mention i use ubuntu Linux 18.04 64 Bit and fpc version 3.3.1.
« Last Edit: November 28, 2018, 10:28:00 pm by Mo0211 »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy10 TIdHTTP gzip issue
« Reply #7 on: November 29, 2018, 12:54:52 am »
First one in hackload and LoadLibVer functions:

Code: Pascal  [Select]
  1. ZLib: TLibHandle = 0;

the return-type on linux FPC is a TLibHandle-Object.
So, the return kinda overflows in your code (value of -40XXXXXXXXXXXX).

That is not what the declaration looks like in the current version of IdZLibHeaders.pas:

Code: Pascal  [Select]
  1. hZLib: THandle = 0;

All uses of HMODULE (which FPC defines as an alias for TLibHandle) were replaced with THandle several weeks ago.  Make sure you are using the latest revision.

The next issue is the Handling in IdZlibHeaders:

The loading of libz-library is dynamic implemented.
But it checks of Handle-Value of <> 0 - if the Value owerflowed then it has a high negative value.

That is not a problem in the latest version.  Besides, even if it did overflow, it still would not be 0.

Code: Pascal  [Select]
  1.   if High(ALibVersions) > -1 then begin
  2.     Result := NilHandle;
  3.     for i := Low(ALibVersions) to High(ALibVersions) do
  4.     begin
  5.       Result := LoadLibVer(ALibVersions[i]);
  6.       if Result > NilHandle then begin
  7.         Break;
  8.       end;
  9.     end;
  10.   end else begin
  11.     Result := LoadLibVer('');

That is not what the latest loop code looks like.  Although yes, the code was still using NilHandle (I just now fixed that to use 0 instead), the comparison has never used > , it has always used <>.

the library is named libz per default in linux (not zlib).

'libz' is the base name that Indy uses to load ZLib dynamically on all UNIX platforms, including Linux:

Code: Pascal  [Select]
  1.   {$IFDEF UNIX}
  2. const
  3.   //The extensions will be resolved by IdGlobal.HackLoad
  4.   //This is a little messy because symbolic links to libraries may not always be the same
  5.   //in various Unix types.  Even then, there could possibly be differences.
  6.   libzlib = 'libz';
  7.   // TODO: setup this array more like the SSLDLLVers arrays in the IdSSLOpenSSLHeaders unit...
  8.   libvers : array [0..3] of string = ('.1','','.3','.2');
  9.   {$ENDIF}
  10.  
« Last Edit: November 29, 2018, 12:59:19 am by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

Mo0211

  • Newbie
  • Posts: 2
Re: Indy10 TIdHTTP gzip issue
« Reply #8 on: November 29, 2018, 07:49:46 am »
Hi Remy,

sorry for my misleading comments and first of all thank you for your fast reply.

All code i posted is already "corrected" by me.

you are right, the ZLib-Object is defined as THandle.
But issue is that when the library will we loaded and the return-value will be given back a conversion from TLibHandle to THandle does not work.
And this seems to be the main issue for me.

Thats my i changed the object back to TLibHandle which is working.

Also checking for the result with nilhandle was changed by me in the way that i compare with bigger then because when the object has overflowed then the number is a high negative value.
So, the value seems only be correct to me if a big positive number is assigned.

i have to check further if the dynamicaly loading works with this setup.
I will come back to you with this information.

But the loading of the lib does not work correctly in this time on fpc.

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #9 on: November 29, 2018, 10:28:57 am »
Good to see that I'm not the only one who has issues lately.  :D

All uses of HMODULE (which FPC defines as an alias for TLibHandle) were replaced with THandle several weeks ago.

After reading the documentation from FPC, it seems that THandle is just for backward compatibility and always a 32bit signed integer (which seems to overflow on 64bit platform as reported above).
Whereas TLibHandle is defined differently on various platforms. Used LoadLibrary function does also return a TLibHandle.

In the documentation for HMODULE (Windows compatibility type), you see that it's defined as unsigned 32-bits integer (when using SafeLoadLibrary function - might be problematic on 64bit but I guess it's Windows only).

I guess the following code in IdGlobal.pas is also problematic, even if NO_REDECLARE is defined on FPC as default but if you undef it, I assume it'll fail on 64bit too.
Code: Pascal  [Select]
  1.  {$IFNDEF NO_REDECLARE}
  2.     {$IFDEF LINUX}
  3.       {$IFNDEF VCL_6_OR_ABOVE}
  4.   THandle = UInt32; //D6.System <-- should be platform independent
  5.       {$ENDIF}
  6.     {$ENDIF}
  7. {$ENDIF}
  8.  
« Last Edit: November 29, 2018, 10:32:41 am by asdf121 »

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #10 on: November 29, 2018, 09:00:32 pm »
Small program to test bytes of each type:
Code: Pascal  [Select]
  1. program SizeOfTest;
  2.  
  3. uses
  4.     SysUtils, DynLibs;
  5.  
  6. begin
  7.     writeln(Format('TLibhandle bytes %d', [SizeOf(TLibhandle)]));
  8.     writeln(Format('THandle bytes %d', [SizeOf(THandle)]));
  9.     writeln(Format('HMODULE bytes %d', [SizeOf(HMODULE)]));
  10. end.
  11.  

Free Pascal Compiler version 3.0.4 [2018/04/10] for x86_64:
Code: [Select]
TLibhandle bytes 8
THandle bytes 4
HMODULE bytes 8

Free Pascal Compiler version 3.1.1 [2018/02/09] for x86_64:
Code: [Select]
TLibhandle bytes 8
THandle bytes 4
HMODULE bytes 8

Free Pascal Compiler version 3.2.0-beta-r40301 [2018/11/26] for x86_64:
Code: [Select]
TLibhandle bytes 8
THandle bytes 4
HMODULE bytes 8
« Last Edit: November 29, 2018, 09:05:47 pm by asdf121 »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy10 TIdHTTP gzip issue
« Reply #11 on: December 01, 2018, 09:28:33 pm »
After reading the documentation from FPC, it seems that THandle is just for backward compatibility and always a 32bit signed integer (which seems to overflow on 64bit platform as reported above).

Then that is an incompatibility with Delphi (and Indy uses FPC in Delphi mode), whose System.THandle is a NativeUInt, so it is 4 bytes on 32bit platforms and 8 bytes on 64bit platforms.

In the documentation for HMODULE (Windows compatibility type), you see that it's defined as unsigned 32-bits integer (when using SafeLoadLibrary function - might be problematic on 64bit but I guess it's Windows only).

FPC defines System.HMODULE as a PtrUInt, so it is 8 bytes on a 64bit platform.

I guess the following code in IdGlobal.pas is also problematic, even if NO_REDECLARE is defined on FPC as default but if you undef it, I assume it'll fail on 64bit too.

That code is intended for Delphi only, not FPC.

Indy was using HMODULE originally, but apparently there is an incompatibility with HMODULE caused by FPC's DynLibs unit (which a few of Indy's units do use), that is why I switched everything to THandle.  But I did not know that FPC limits THandle to 4 bytes even on 64bit platforms (Delphi doesn't).

Looks like I have some more fixes to make for FPC  :'(

Update: Actually, I just now looked at the latest FPC 3.0.4 RTL source, and there is no single place where THandle is defined.  Different platforms define their own THandle, where some define it as 4 bytes, and some define it as 8 bytes (Win64 does, for instance).  So, it is actually a platform-specific type.  But maybe some platforms are not defining it as 8 bytes for x64 correctly?  What platform are you compiling your SizeOfTest project on?  I see the RTL using THandle all over the place.  So, I'm a little lost whether I should keep Indy using THandle, or have it go back to using HMODULE/TLibHandle and just work around the DynLibs.HModule incompatibility.
« Last Edit: December 01, 2018, 10:12:28 pm by Remy Lebeau »
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #12 on: December 01, 2018, 11:13:48 pm »
Quote from: Remy Lebeau
What platform are you compiling your SizeOfTest project on?
For doing the test with 3.0.4 I used an online compiler.
The other ones were done on Linux Mint 64bit.

I'd use TLibHandle as that is also the return type for LoadLibrary.

I even don't know where THandle is defined. Strange :o

EDIT:
Quote from: Marco van de Voort
Thandle is mostly used for filehandles in the RTL (e.g. the THandleStream, and the COM streamwrappers), which are 32-bit.
The problem is simply that (really) all handles are THandle on Windows, but on *nix there many different types.
Changing thandle there, will only cause problems elsewhere.
Use an own defined handle type instead of the generic one.
In general, I would advise you to avoid the use of Windows compatibility types as much as possible.

https://bugs.freepascal.org/view.php?id=21669
« Last Edit: December 02, 2018, 12:02:24 am by asdf121 »

Remy Lebeau

  • Hero Member
  • *****
  • Posts: 515
    • Lebeau Software
Re: Indy10 TIdHTTP gzip issue
« Reply #13 on: December 02, 2018, 03:11:03 am »
I'd use TLibHandle as that is also the return type for LoadLibrary.

I have just now checked in an update to Indy's SVN so that Indy now uses TLibHandle on FreePascal again, and keeps using THandle on Delphi.

I even don't know where THandle is defined. Strange :o

It is defined in various platform-specific .inc files, usually named sysosh.inc.

Quote from: Marco van de Voort
Thandle is mostly used for filehandles in the RTL (e.g. the THandleStream, and the COM streamwrappers), which are 32-bit.
The problem is simply that (really) all handles are THandle on Windows, but on *nix there many different types.
Changing thandle there, will only cause problems elsewhere.
Use an own defined handle type instead of the generic one.
In general, I would advise you to avoid the use of Windows compatibility types as much as possible.

https://bugs.freepascal.org/view.php?id=21669

Hmm.  There is still quite a bit of THandle usage in Indy beyond just loading libraries.  I'll review them and see if they need to be adjusted.  I think the remaining usages are all on Windows, though, so I can probably just use HMODULE and/or other appropriate handle types.
Remy Lebeau
Lebeau Software - Owner, Developer
Internet Direct (Indy) - Admin, Developer (Support forum)

asdf121

  • New member
  • *
  • Posts: 23
Re: Indy10 TIdHTTP gzip issue
« Reply #14 on: December 03, 2018, 06:47:08 pm »
Thanks Remy, I've tested your changes and it's working flawless again :)