Recent

Author Topic: Use ShellExecute to send browser output to a file  (Read 13581 times)

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Use ShellExecute to send browser output to a file
« on: July 28, 2015, 02:47:44 am »
The following approach opens up the default browser and displays the indicated web page.

    s := 'http://www.google.com';
    ShellExecute(0, 'open', PChar(s), '', '',  SW_SHOW);   

How can I have the HTML output sent to a file (rather than displayed on the monitor) so I can subsequently open and parse it in my program?
Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: Use ShellExecute to send browser output to a file
« Reply #1 on: July 28, 2015, 04:04:16 am »
Usually one would open the file menu in the browser and select to save the web-page in whatever file-format the by the user custom selected default browser supports.

Meaning: to my knowledge there is no unified answer to that question.

You should be better off using something like wget or curl, or better yet retrieve the webpage yourself by using one of the network suites.

You could try httpget for instance, which is part of the fcl-web package.


jc99

  • Hero Member
  • *****
  • Posts: 553
    • My private Site
Re: Use ShellExecute to send browser output to a file
« Reply #2 on: July 28, 2015, 11:43:50 am »
How can I have the HTML output sent to a file (rather than displayed on the monitor) so I can subsequently open and parse it in my program?
Try a sneak peek to
http://wiki.freepascal.org/fphttpclient
OS: Win XP x64, Win 7, Win 7 x64, Win 10, Win 10 x64, Suse Linux 13.2
Laz: 1.4 - 1.8.4, 2.0
https://github.com/joecare99/public
'~|    /''
,_|oe \_,are
If you want to do something for the environment: Twitter: #reduceCO2 or
https://www.betterplace.me/klimawandel-stoppen-co-ueber-preis-reduzieren

balazsszekely

  • Guest
Re: Use ShellExecute to send browser output to a file
« Reply #3 on: July 28, 2015, 04:08:56 pm »
Try this(windows only):
Code: [Select]
uses wininet;

function URLToFile(const AURL, AFileName: String): Boolean;
var
  lpBuffer: array[0..1024 + 1] of Char;
  lpdwNumberOfBytesRead: DWORD;
  hSession: HINTERNET;
  hService: HINTERNET;
  fs: TFileStream;
  Data: String;
begin
  Result := False;

  //open internet connection
  hSession := InternetOpen('My application', INTERNET_OPEN_TYPE_PRECONFIG, nil, nil, 0);
  if hSession = nil then
  begin
    Exit;
  end;

  //open url
  hService := InternetOpenUrl(hSession, PChar(AURL), nil, 0, INTERNET_FLAG_DONT_CACHE or INTERNET_FLAG_PRAGMA_NOCACHE or INTERNET_FLAG_RELOAD, 0);
  if hService = nil then
  begin
    InternetCloseHandle(hSession);
    Exit;
  end;

  while (lpdwNumberOfBytesRead > 0) do
  begin
   // Read data from the hService handle
    if not InternetReadFile(hService, @lpBuffer, 1024, lpdwNumberOfBytesRead) then
    begin
      InternetCloseHandle(hService);
      InternetCloseHandle(hSession);
      Exit;
    end;
    lpBuffer[lpdwNumberOfBytesRead] := #0;
    Data := Data + lpBuffer;
  end;
  if Length(Data) > 0 then
  begin
    fs := TFileStream.Create(AFileName, fmCreate);
    try
      fs.WriteBuffer(Pointer(Data)^, Length(Data));
      Result := True;
    finally
      fs.Free
    end;
  end;
end;                   

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Use ShellExecute to send browser output to a file
« Reply #4 on: July 28, 2015, 06:23:00 pm »

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Re: Use ShellExecute to send browser output to a file
« Reply #5 on: July 29, 2015, 01:00:24 am »
Thank you all for your good ideas.

For my situation, it's imperative that I use the default browser because a cookie is stored there. I've tried most of the above ideas and they don't provide the complete web page (when testing with other URLs). For example, image files don't appear. The ShellExecute approach picks up the cookie and displays the complete web page.

It appears that I need to pipe the output to a file, using ShellExecute - if I can figure that out.
Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: Use ShellExecute to send browser output to a file
« Reply #6 on: July 29, 2015, 02:32:46 am »
Strange that you would have the need to store the cookie inside the default browser... Is there a particular reason you could not store the cookie yourself ?

I would like to make a minor correction to your statement "don't provide the complete web-page" + "image files don't appear":

Image files are never part of a web-page, rather a web-page (html) instructs the browser to 'load' data into it's memory layout so that the browser is able to put all 'loaded' files together and construct something to the window that you experience as being "a web-page".

If you save the pure received html then this will also not include images, sounds, video's and all other stuff that's possible to link to in a html file


So, unless the default browser is capable of providing a static representation of your web-page in whatever format (mht or pdf comes to mind) you are not able to extract anything useful from that information either. E.g. you would have to render the information provided by the html (web-page) yourself as well.

As said before, there is no unified way to do as you ask because it depends on which browser is the default browser (and assuming internet exploder would be the default browser would just be a wrong assumption).

As for an example on how to request/render html files and managing session, you could perhaps take a look at the thtmlviewer component.


Quote
It appears that I need to pipe the output to a file, using ShellExecute - if I can figure that out.
You are unable to figure that out because there simply isn't a web-browser capable of outputting complete rendered web-pages by using pipes (what would that be, a nice .png file anyway ?). Maybe one exception to that statement could be Lynx (but i haven't used that in a very long while).


PS: i still don't see what's wrong with using "wget --page-requisites url" and using one of the cookie options, other then you would have to render the web-page yourself, (although i have no idea what good that would do, but you haven't provided a reason as of why).
« Last Edit: July 29, 2015, 03:28:42 am by molly »

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Re: Use ShellExecute to send browser output to a file
« Reply #7 on: July 29, 2015, 05:05:15 pm »
My app doesn't need to store a cookie.
Using some browser, the user must first login with his username/password to store a cookie which persists for 30 days, and is renewed for 30 each time the site is accessed. It is assumed that the browser used to login initially is the default browser. To access the desired web page, the user must use the browser that has stored the cookie (PHPSESSID). For this reason, an approach that accesses the web via my Pascal code will not work since the cookie is not available in that "browser".

Retrieving that cookie from the default browser is a task I haven't contemplated as it seems a lot more difficult than just piping the results via a ShellExecute. ShellExecute is a good solution because I don't have to find out what the default browser is.

Molly - your point about "complete web-pages is correct. Of course those image files, etc. are not within the html file.


Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Re: Use ShellExecute to send browser output to a file
« Reply #8 on: July 29, 2015, 05:24:16 pm »
Perhaps I need to rephrase my question. When I use
    s := 'http://  any URL';
    ShellExecute(0, 'open', PChar(s), '', '',  SW_SHOW);
my default browser opens and displays the desired page. When I use "View Source" (in that browser) a file appears with the html rendering of that page. I want to be able to store that html rendering into a file.
Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

jc99

  • Hero Member
  • *****
  • Posts: 553
    • My private Site
Re: Use ShellExecute to send browser output to a file
« Reply #9 on: July 29, 2015, 06:06:57 pm »
Perhaps I need to rephrase my question. When I use
    s := 'http://  any URL';
    ShellExecute(0, 'open', PChar(s), '', '',  SW_SHOW);
my default browser opens and displays the desired page. When I use "View Source" (in that browser) a file appears with the html rendering of that page. I want to be able to store that html rendering into a file.
Are you using Windows or another OS ?
OS: Win XP x64, Win 7, Win 7 x64, Win 10, Win 10 x64, Suse Linux 13.2
Laz: 1.4 - 1.8.4, 2.0
https://github.com/joecare99/public
'~|    /''
,_|oe \_,are
If you want to do something for the environment: Twitter: #reduceCO2 or
https://www.betterplace.me/klimawandel-stoppen-co-ueber-preis-reduzieren

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Re: Use ShellExecute to send browser output to a file
« Reply #10 on: July 29, 2015, 06:07:58 pm »
Only Windows
Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Use ShellExecute to send browser output to a file
« Reply #11 on: July 29, 2015, 06:13:13 pm »
And whats the url you want to do this? any?

bobonwhidbey

  • Hero Member
  • *****
  • Posts: 590
    • Double Dummy Solver - free download
Re: Use ShellExecute to send browser output to a file
« Reply #12 on: July 29, 2015, 06:14:12 pm »
I'd like to be able to open any URL.
Lazarus 3.0RC2, FPC 3.2.2 x86_64-win64-win32/win64

molly

  • Hero Member
  • *****
  • Posts: 2330
Re: Use ShellExecute to send browser output to a file
« Reply #13 on: July 29, 2015, 06:20:46 pm »
Perhaps I need to rephrase my question. When I use
At least let us know what exactly you are trying to accomplish ;-)

Quote
I want to be able to store that html rendering into a file
And that is exactly what we have been telling you how you could accomplish that task, BUT using another solution as you had envisioned.

But, then you add an additional factor into the equation e.g. that you need to use the users default browser because it stores the user's session id inside a cookie ?

The latter might be necessary for you to be able to retrieve the webpage that you need the source-code from, but IMO is just nonsense as you can accomplish the same task using the methods already suggested by others. You seem to fail to understand that part (and yes, managing a session requires a little additional work).

If you are not capable of copy-paste existing examples source-code and implement it in your own program then i strongly suggest to use shellexecute in combination with wget (as suggested by me)

It does not really matter how many times you ask the same question, but it is simply not possible by using shell execute to let the default browser save the html file of the webpage that it currently displays or use pipes to accomplish that task.

That's why people suggested using another methods to solve the problem, but you haven't given those suggestions a real serious look have you ? (e.g. merely dismissed them)

lainz

  • Hero Member
  • *****
  • Posts: 4460
    • https://lainz.github.io/
Re: Use ShellExecute to send browser output to a file
« Reply #14 on: July 29, 2015, 06:30:08 pm »
this works, i get the same source as "view source" (except for an extra empty line at the end of file).

Code: [Select]
uses
  fphttpclient;

procedure TForm1.Button1Click(Sender: TObject);
var
  s: TStringList;
  client: TFPHTTPClient;
begin
  s := TStringList.Create;
  client := TFPHTTPClient.Create(nil);
  client.SimpleGet('http://wiki.lazarus.freepascal.org/', s);
  client.Cookies.SaveToFile('cookies.txt');
  s.SaveToFile('wiki.html');
  s.Free;
  client.Free;
end;         

If you want to download the images, css, js, everything else you must analyze the html to get all these.

If you don't want to do that better include chromium in your application, don't try to use shellexecute anymore.

 

TinyPortal © 2005-2018