GetURLText

[VTwin]

For a user to check for updates I have been using a very simple method, where a text file containing the current version (e.g., '3.21.0') is checked against the user's version.

I have been using a function GetURLText posted by trev, for quite a while, but it seems to no longer work. A search for GetURLText does not come up with a hit.

What is the best way to download a simple text file from a url?

[KodeZwerg]

For what OS with what needs (plain old http or newer https) (username password needed?)

[VTwin]

Thanks KodeZwerg,

I need something for macOS, Windows, and Linux, as in my profile.

I just want to download a text file, plain old http, no password.

Edit: Actually the site is https.
Something seems to have changed in the past few months.

[KodeZwerg]

I am unsure about target platforms, please try for yourself.
(OPM -> Synapse -> Install)
Start a new project and try that:
(add to project requirement lazSynapse)

Code: Pascal  [Select][+][-]

1. unit Unit1;
2.  
3. {$mode objfpc}{$H+}
4.  
5. interface
6.  
7. uses
8.   Classes , SysUtils , Forms , Controls , Graphics , Dialogs , StdCtrls ,
9.   HTTPSend;
10.  
11. type
12.  
13.   { TForm1 }
14.  
15.   TForm1 = class(TForm)
16.     Button1: TButton;
17.     Memo1: TMemo;
18.     procedure Button1Click(Sender: TObject);
19.   strict private
20.   private
21.  
22.   public
23.  
24.   end;
25.  
26. var
27.   Form1: TForm1;
28.  
29. implementation
30.  
31. {$R *.lfm}
32.  
33. { TForm1 }
34.  
35. function GetURLText(const AURL: string; out AText: string): Boolean;
36. begin
37.   AText := '';
38.   Result := False;
39.   with THTTPSend.Create do
40.   begin
41.     if HTTPMethod('GET', AURL) then
42.     try
43.       if Document.Size > 0 then
44.       begin
45.         Document.Position := 0;
46.         SetLength(AText, Document.Size);
47.         Document.ReadBuffer(Pointer(AText)^, Document.Size);
48.         Result := AText <> '';
49.       end;
50.     except
51.       Application.MessageBox('Unknown','Error', 0);
52.     end;
53.     Free;
54.   end;
55. end;
56.  
57. procedure TForm1.Button1Click(Sender: TObject);
58. var
59.   s: string;
60. begin
61.   if GetURLText('http://your url with filename', s) then
62.     Memo1.Lines.Add(s)
63.     else
64.     Memo1.Lines.Add('Error');
65. end;
66.  
67. end.

[dsiders]

Thanks KodeZwerg,

I need something for macOS, Windows, and Linux, as in my profile.

I just want to download a text file, plain old http, no password.

Edit: Actually the site is https.
Something seems to have changed in the past few months.

You can use the TFPHttpClient class in the fphttpclient unit. It's cross-platform. Supports SSL when you use the appropriate optional units. Call the Get method, or SimpleGet to ignore protocol errors/exceptions.

Code: Pascal  [Select][+][-]

1. AString := TFPHTTPClient.SimpleGet(AUrl);

See: https://wiki.freepascal.org/fphttpclient

[Thaddy]

If simpleget won't work, do it like this (also simple, but slightly more involved):

Code: Pascal  [Select][+][-]

1. {$mode objfpc}
2. uses
3.   classes,sysutils,fphttpclient,opensslsockets,fpJson,jsonparser;
4.  
5. var
6.   Client:TFpHttpClient;
7.   List:TStringList;
8.   URL:String = 'https://www.example.com';
9. begin
10.   Client := TfpHttpClient.Create(nil);
11.   try
12.     // this is important
13.     Client.AllowRedirect := true; // I am not sure SimpleGet handles redirects.
14.     // optional browser impersonation. This is sometimes necessary, although there is a very old default.
15.     Client.RequestHeaders.Add('User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:12.0) Gecko/20100101 Firefox/12.0');
16.     List := TStringlist.Create;
17.     try
18.       List.Text := Client.Get(URL);
19.       writeln(List.Text);
20.     finally
21.       List.Free;
22.     end;
23.   finally
24.     Client.Free;
25.   end;
26. end.

I recommend using fcl-web like above, instead of Synapse. Better maintained and standard.

[Warfley]

Just use TFPHTTPClient with the openssl handler:

Code: Pascal  [Select][+][-]

1. uses
2.   opensslsockets, // Include per uses to enable OpenSSL handling
3.   fphttpclient;
4.  
5. var
6.   WebsiteText: String;
7. begin
8.   WebsiteText := TFPHTTPClient.SimpleGet('https://my.https.example.com');
9.   WriteLn(WebsiteText);
10. end.

OpenSSL is pre installed on windows since a few years, and should also be available on mac and pretty much all of the most common linux distros.

[Thaddy]

@Warfley
SimpleGet does not always work, hence my example.

[VTwin]

Thanks all. Trying Warfley and Thaddy's examples, I get (on Windows) exception  'EInOutError' "Could not initialize OpenSSL library" in both cases.

I did not try KodeZwerg's example as HTTPSend was not found.

In the distant past I was using Synapse, but switched to fphttpclient to minimize dependencies. It was working fine for a long time, but recently I got a compiler message to include opensslsockets. Now I am getting exceptions.

[Thaddy]

64 bit or 32 bit?
The openssl libraries must match exactly.
For now I think there is your problem.
If that is the case it is easy to solve.
(assuming windows64)
Wait for it:
- if it is 32 bit, download the 32 bit version and copy the dll's to \windows\syswow64 (yes 32 bit goes there!)(Many wtf's here)
- if it is 64 bit on 64 bit windows, download the 64 bit version and put the dll's in \windows\system32 (yes 64 bit goes there)(Many wtf's here)

I know this is counter-intuitive..... Let's call it "Windows Logic".

"Surprised?  You won't be, until the next episode of Soap"
https://www.youtube.com/watch?v=0BHQT3Omqtw

btw: WoW stands for Windows32 on Windows64. On Linux it would be multi-arch.

Microsoft's answer (ad verbatim):
"The Windows/SysWOW64 directory contains 32-bit libraries and executables while the Windows/System32 directory contains 64-bit libraries and executables. The SysWOW64 directory was created to facilitate the running of 32-bit legacy applications on 64-bit Windows systems."

Many people, even on this forum, get it wrong all the time...... 

But that is obvious, isn't it? Everyone understands that? 

[Warfley]

(assuming windows64)
Wait for it:
- if it is 32 bit, download the 32 bit version and copy the dll's to \windows\syswow64 (yes 32 bit goes there!)(Many wtf's here)
- if it is 64 bit on 64 bit windows, download the 64 bit version and put the dll's in \windows\system32 (yes 64 bit goes there)(Many wtf's here)

I know this is counter-intuitive..... Let's call it "Windows Logic".

There is actually reasoning behind it. System32 is named the same on 32bit and 64 bit, because of backwards compatibility, this is not pretty, but this backwards compatibility even to the lowest elevel is one of the reasons why windows is so dominant in the market.
The syswow64 is called that way because its the system folder for the Windows on Windows 64 (WoW64) subsystem, which basically is a 32 bit windows environment emulated within a windows 64 environment, therefore the name "windows" on "windows 64".

So it might not be intuitive, but it makes at least a bit sense when thinking about it. But I would not rule out that the naming of SysWoW64 was an april fools joke by some microsoft employee that got through QA and got rolled out to production

[VTwin]

Thanks. Wow, or is it SysWoW64?

I use 64 bit on Windows, Mac, and Linux.

Maybe that is the problem, but why doesn't fpc link in the required version? On Mac I go to

usr/local/lib/fpc/3.2.2/units/

where I find aarch64-darwin, i386-darwin, x86_64-darwin, each with an openssl directory.

[VTwin]

I also tried KodeZwerg's solution using Synapse. No luck yet.

[GetMem]

@VTwin

On windows you should check if the openssl libraries are available, if not download them, something like this:

Code: Pascal  [Select][+][-]

1. uses zipper, fphttpclient, opensslsockets;
2.  
3. function GetURL(URL: String; var URLRes: String): Boolean;
4. var
5.   SS: TStringStream;
6.   FPHTTPClient: TFPHTTPClient;
7. begin
8.   Result := False;
9.   FPHTTPClient := TFPHTTPClient.Create(nil);
10.   try
11.     FPHTTPClient.AllowRedirect := True;
12.     SS := TStringStream.Create('');
13.     try
14.       Screen.Cursor := crHourglass;
15.       try
16.         FPHTTPClient.Get(URL, SS);
17.       finally
18.         Screen.Cursor := crDefault;
19.       end;
20.       if FPHTTPClient.ResponseStatusCode = 200 then
21.       begin
22.         URLRes := SS.DataString;
23.         Result := True;
24.       end;
25.     finally
26.       SS.Free;
27.     end;
28.   finally
29.     FPHTTPClient.Free;
30.   end;
31. end;
32.  
33. function IsOpenSSLAvailable: Boolean;
34. const
35.   {$IFDEF WIN64}
36.     cOpenSSLURL = 'http://packages.lazarus-ide.org/openssl-1.1.1o-x64_86-win64.zip';
37.   {$ENDIF}
38.   {$IFDEF WIN32}
39.     cOpenSSLURL = 'http://packages.lazarus-ide.org/openssl-1.1.1o-i386-win32.zip';
40.   {$ENDIF}
41. var
42.   {$IFDEF MSWINDOWS}
43.   UnZipper: TUnZipper;
44.   FHTTPClient: TFPHTTPClient;
45.   ParamPath, LibCrypto, LibSSL, ZipFile: String;
46.   {$EndIf}
47. begin
48.   {$IFDEF MSWINDOWS}
49.   ParamPath := ExtractFilePath(ParamStr(0));
50.  
51.   {$IFDEF WIN64}
52.   LibCrypto := ParamPath + 'libcrypto-1_1-x64.dll';
53.   LibSSL := ParamPath + 'libssl-1_1-x64.dll';
54.   {$ENDIF}
55.   {$IFDEF WIN32}
56.   LibCrypto := ParamPath + 'libcrypto-1_1.dll';
57.   LibSSL := ParamPath + 'libssl-1_1.dll';
58.   {$ENDIF}
59.  
60.   if not Result then
61.   begin
62.     ZipFile := ParamPath + ExtractFileName(cOpenSSLURL);
63.     FHTTPClient := TFPHTTPClient.Create(nil);
64.     try
65.       try
66.         FHTTPClient.Get(cOpenSSLURL, ZipFile);
67.        except
68.        end;
69.     finally
70.       FHTTPClient.Free;
71.     end;
72.     if FileExists(ZipFile) then
73.     begin
74.       UnZipper := TUnZipper.Create;
75.       try
76.         try
77.           UnZipper.FileName := ZipFile;
78.           UnZipper.Examine;
79.           UnZipper.UnZipAllFiles;
80.         except
81.         end;
82.       finally
83.         UnZipper.Free;
84.       end;
85.       DeleteFile(ZipFile);
86.       Result := FileExists(LibCrypto) and FileExists(LibSSL);
87.     end;
88.   end;
89.   {$ELSE}
90.   FOpenSSLAvailable := True;
91.   {$ENDIF}
92. end;
93.  

See attached demo for more details.


Correction: The 64 bit library name was wrong. Thanks @KodeZwerg!

[VTwin]

Thanks GetMem.

I tried your example on Windows, it tells me that OpenSSL is not available. It does not compile on Mac though, and I need something cross-platform.

Maybe I should just give up. It is a very minor function of my software, just a nice convenience for my users. It is weird though, if I run my software from 2020, it works fine.