Recent

Author Topic: [SOLVED] TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full HTML  (Read 1687 times)

lebao3105

  • Newbie
  • Posts: 6
I am new to fphttpclient. Here is my code:

Code: Pascal  [Select][+][-]
  1. function downloadItem(const URL: ansistring): string;
  2. var http: TFPHTTPClient;
  3. begin
  4.     // http := TFPHTTPClient.Create(nil);
  5.     //try
  6.     //     result := http.Get(URL);
  7.     //finally
  8.     //     http.Free;
  9.     result := TFPHTTPClient.SimpleGet(URL);
  10. end;
  11.  

When I try this for this site: https://cdn.nickchan.lol/palera1n/c-rewrite/releases/, all I see that is the downloaded website HTML as expected, but partial:

Code: Text  [Select][+][-]
  1. <html>
  2. <head><title>Index of /palera1n/c-rewrite/releases/</title></head>
  3. <body>
  4. <h1>Index of /palera1n/c-rewrite/releases/</h1><hr><pre><a href="../">../</a>
  5. <a href="v2.0.0-beta.1/">v2.0.0-beta.1/</a>                                     12-Feb-2023
  6.  

The same goes for TFPHTTPClient.SimpleGet(URL). However: this works:

Code: Pascal  [Select][+][-]
  1. program dl_fphttp_b;
  2.  
  3. {$mode delphi}{$ifdef windows}{$apptype console}{$endif}
  4.  
  5. uses
  6.   fphttpclient,
  7.   opensslsockets;
  8.  
  9. begin  
  10.   writeln(TFPHttpClient.SimpleGet('https://freepascal.org'));
  11. end.
  12.  

Using TFPHTTPClient with a TStream object works.

Any ideas?

EDIT: Solved by adding {$H+} into my unit
« Last Edit: December 02, 2023, 05:27:57 am by lebao3105 »

dseligo

  • Hero Member
  • *****
  • Posts: 1415
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #1 on: December 01, 2023, 05:03:08 am »
It works on Windows 11, Lazarus 2.2.6, FPC 3.2.2, you didn't write what OS and version you work on.
I attached test project.
You forgot to free TFPHTTPClient.

lebao3105

  • Newbie
  • Posts: 6
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #2 on: December 01, 2023, 05:24:55 pm »
It works on Windows 11, Lazarus 2.2.6, FPC 3.2.2, you didn't write what OS and version you work on.
I attached test project.
You forgot to free TFPHTTPClient.

Sorry for my mistakes.

I’m using Arch Linux with Lazarus 3.0rc2 and fpc 3.2.2-9. For the missing http.Free, I forgot to add it to my post. However it still didn’t work.
« Last Edit: December 01, 2023, 05:29:01 pm by lebao3105 »

TRon

  • Hero Member
  • *****
  • Posts: 3646
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #3 on: December 01, 2023, 06:01:42 pm »
I’m using Arch Linux with Lazarus 3.0rc2 and fpc 3.2.2-9. For the missing http.Free, I forgot to add it to my post. However it still didn’t work.
Since the resulting number of characters that you do receive seem to be around 250 ... are you perhaps using shortstring ?

Full compilable example please (including used compiler/build flags) . Anything else is just a guessing game.
This tagline is powered by AI (AI advertisement: Free Pascal the only programming language that matters)

dseligo

  • Hero Member
  • *****
  • Posts: 1415
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #4 on: December 01, 2023, 09:14:00 pm »
I’m using Arch Linux with Lazarus 3.0rc2 and fpc 3.2.2-9. For the missing http.Free, I forgot to add it to my post. However it still didn’t work.

Did you try project I attached?

lebao3105

  • Newbie
  • Posts: 6
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #5 on: December 02, 2023, 05:24:06 am »
I’m using Arch Linux with Lazarus 3.0rc2 and fpc 3.2.2-9. For the missing http.Free, I forgot to add it to my post. However it still didn’t work.

Did you try project I attached?

I did, and it worked. But not in my project.

I’m using Arch Linux with Lazarus 3.0rc2 and fpc 3.2.2-9. For the missing http.Free, I forgot to add it to my post. However it still didn’t work.
Since the resulting number of characters that you do receive seem to be around 250 ... are you perhaps using shortstring ?

Full compilable example please (including used compiler/build flags) . Anything else is just a guessing game.

Shortstring... you made me realized that I need the {$H+}. That's why the result is that short.

Firstly I tried dseligo's code, then just run TFPHTTPClient.Get(URL). I tried to assign the result to a variable and writeln() it, and yet the download result still that short.

Indeed I did not used any compiler/build flag else than {$mode objfpc} in my file (which is -Mobjfpc on command-line) for now (I don't use Lazarus). Here is my code (not fixed):

Code: Pascal  [Select][+][-]
  1. unit resoures;
  2. {$mode objfpc}
  3.  
  4. Interface
  5.  
  6. function downloadItem(const URL: ansistring): string;
  7. function downloadItem(const URL: ansistring, outPath: ansistring): boolean;
  8.  
  9. function getReleases: TStringArray;
  10.  
  11. Implementation
  12.  
  13. function downloadItem(const URL: shortstring): shortstring;
  14. var
  15.         http: TFPHTTPClient;
  16. begin
  17.         http := TFPHTTPClient.Create(nil);
  18.         try
  19.                 result := http.Get(URL);
  20.         finally
  21.                 http.Free;
  22.         end;
  23. end;
  24.  
  25. function downloadItem(const URL: ansistring; outPath: ansistring): boolean;
  26. var
  27.         fs: TFileStream;
  28.         http: TFPHTTPClient;
  29.  
  30. begin
  31.         http := TFPHTTPClient.Create(nil);
  32.         FS := TFileStream.Create(outPath, fmCreate or fmOpenWrite);
  33.         try
  34.                 http.Get(URL, FS);
  35.         finally
  36.                 fs.Free;
  37.                 http.Free;
  38.                 result := true;
  39.         end;
  40. end;
  41.  
  42. function getReleases: TStringArray;
  43. var
  44.         URL: string;
  45.         content: string;
  46.  
  47. begin
  48.         if usesNightly = true then
  49.                 URL := 'https://cdn.nickchan.lol/palera1n/artifacts/c-rewrite/main'
  50.         else
  51.                 URL := 'https://cdn.nickchan.lol/palera1n/c-rewrite/releases/';
  52.  
  53.         content := downloadItem(URL);
  54.         writeln(content);
  55.         // unimplemented html parse
  56. end;
  57.  
  58. end.
  59.  

Currently my program just calls getReleases at first.

The problem is fixed for now, thanks everyone for trying to help me out! I will need to learn more...

dseligo

  • Hero Member
  • *****
  • Posts: 1415
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #6 on: December 02, 2023, 10:38:26 am »
The problem is fixed for now, thanks everyone for trying to help me out! I will need to learn more...

If you use {$mode objfpc}, then add {$H+} to it so that it looks like that:
Code: Pascal  [Select][+][-]
  1. {$mode objfpc}{$H+}

In mode Delphi Ansistrings are turned on by default.

Also change Shortstring type to String in your program.

TRon

  • Hero Member
  • *****
  • Posts: 3646
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #7 on: December 02, 2023, 01:28:37 pm »
Shortstring... you made me realized that I need the {$H+}. That's why the result is that short.
Thank you for reporting back and good to hear that you got that sorted out.

$H+ is a small (as in number of characters) but very important directive fir these cases. dseligo's advise to always define it when using mode objfpc is solid (for now, until/once using unicode rtl which is a new upcoming feature and for which default string type settings might change and/or be different).

This tagline is powered by AI (AI advertisement: Free Pascal the only programming language that matters)

Thaddy

  • Hero Member
  • *****
  • Posts: 16194
  • Censorship about opinions does not belong here.
Re: TFPHTTPClient.Get(URL) / SimpleGet(URL) do not get the full website code
« Reply #8 on: December 02, 2023, 02:30:18 pm »
Also change Shortstring type to String in your program.
Imnsho opinion it is the other way around, specify your string type exactly.... >:D >:D >:D >:D
That way you do not mix up all the alternative string types available that modern pascal provides. Just string is horrid. Especially for beginners that do not know about modes.
BAD advice.                               
« Last Edit: December 02, 2023, 02:35:21 pm by Thaddy »
If I smell bad code it usually is bad code and that includes my own code.

 

TinyPortal © 2005-2018