Browser Automation with selenium

[RedOctober]

I run the .exe by itself, and the same error message appears.

[Jurassic Pork]

have a look in the file chromedriver.log in the project folder. When it fails i have this in the file :

[1630342063.048][INFO]: Launching chrome: "C:\Program Files (x86)\Google\Chrome\Application\chrome.exe" --allow-pre-commit-input --disable-background-networking --disable-backgrounding-occluded-windows --disable-client-side-phishing-detection --disable-default-apps --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --log-level=0 --no-first-run --no-service-autorun --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir="C:\Users\jp\AppData\Local\Temp\scoped_dir2892_1394031789" data:,

[1630342063.084][INFO]: [281292df0dfd4ae3205b7269a71a9341] RESPONSE InitSession ERROR unknown error: Failed to create Chrome process.

[RedOctober]

Hi JP.  Here is the contents of my log file.  I have to have IPV6 shut off on my server.  Would that cause this problem?  Or, maybe port 9515 is already busy?  Is there a place I can configure/reserve this port for chromedriver if it is needs this specific port/

[1630341068.643][INFO]: Starting ChromeDriver 92.0.4515.107 (87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-heads/4515@{#1634}) on port 9515
[1630341068.643][INFO]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[1630341068.646][SEVERE]: bind() returned an error: Only one usage of each socket address (protocol/network address/port) is normally permitted. (0x2740)
[1630341068.647][INFO]: listen on IPv6 failed with error ERR_ADDRESS_IN_USE

[Jurassic Pork]

Hi JP.  Here is the contents of my log file.  I have to have IPV6 shut off on my server.  Would that cause this problem?  Or, maybe port 9515 is already busy?  Is there a place I can configure/reserve this port for chromedriver if it is needs this specific port/

[1630341068.643][INFO]: Starting ChromeDriver 92.0.4515.107 (87a818b10553a07434ea9e2b6dccf3cbe7895134-refs/branch-heads/4515@{#1634}) on port 9515
[1630341068.643][INFO]: Please see https://chromedriver.chromium.org/security-considerations for suggestions on keeping ChromeDriver safe.
[1630341068.646][SEVERE]: bind() returned an error: Only one usage of each socket address (protocol/network address/port) is normally permitted. (0x2740)
[1630341068.647][INFO]: listen on IPv6 failed with error ERR_ADDRESS_IN_USE

When you have this problem it's because the webdriver is already running (see in the task manager). You must kill it before to  run your project (use the task manager).

An for the other problem on a faster computer i have no problem with fpdebug  :
I have found why you haven't the reason of the error but the message Unknown object member: "sessionId"
There is an error in the TChromeDriver.NewSession function of WebDriver4L.

With the new version of WebDriver4L (0.2) you can see why you have an error.

See  my next message for new version of WebDriver4L

[Jurassic Pork]

hello,
here is the new version of WebDriver4L :

WebDriver4L   Selenium Webdriver for Lazarus  V 0.2  // Jurassic Pork 08/2021
Derivative work from Ericwang1104/WebDriver4D  : https://github.com/Ericwang1104/WebDriver4D
Source code modified to use fphttpclient, fpjson,jsonparser.
The project is in beta stage.
V 0.1 improve demo project.
V 0.2 improve WebDriver4L and demo project.
A demo project is included :  get a screenshot of the lazarus forum, get all the recent topics and display then in a memo, get stats.
the demo project has been tested on windows 10 lazarus 2.0.10   with chrome browser and on Ubuntu 20.04 Lazarus 2.0.10 with firefox browser.
You must download the selenium Webdriver for your browser and put it in your project folder. You must use a compatible driver version of your browser :
links to download webdrivers are here : https://www.selenium.dev/documentation/en/webdriver/driver_requirements/ (Quick reference)
For Chrome the selenium drivers are here : https://chromedriver.chromium.org/downloads
  If you use Chrome 92.x  on Windows  download a 92 version of the driver.
For Firefox the selenium drivers are here : https://github.com/mozilla/geckodriver/releases
For Edge Chromium the selenium drivers are here : https://msedgewebdriverstorage.z22.web.core.windows.net/
 go near to the last page with the next button and choose the same version than your ms edge chromium browser.
For IE 11 the selenium drivers are here : https://selenium-release.storage.googleapis.com/index.html?path=3.9/   (IEDriverServer 3.9)
For Safari  the selenium drivers are here : Built-in

The Software is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders X be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the Software.

Ericwang1104/WebDriver4D has no license so this project will not be published on github. No activity since March 2019

To choose your automation browser comment uncomment this part of the code

Code: Pascal  [Select][+][-]

1.   Try
2.    {$IFDEF WINDOWS}
3.      Robot := TChromeDriver.Create(nil);
4.   //   Robot := TFireFoxDriver.Create(nil);
5.   //   Robot := TEdgeDriver.Create(nil);
6.   //   Robot := TIEDriver.Create(Nil);
7.  
8.      Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\chromedriver.exe');
9.   //    Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\geckodriver.exe');
10.   //   Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\msedgedriver.exe');
11.   //   Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '\IEDriverServer.exe');

the demo project has been tested on windows 10 lazarus 2.0.12   with chrome browser,  Edge Chromium and internet Explorer 11.

Friendly, J.P

[Fahmy Rofiq]

How to save login ?

[Jurassic Pork]

How to save login ?

What does it mean ?

[Fahmy Rofiq]

After login to a website, how to save its cookie?
Because after application restart, it need to login again...

[Jurassic Pork]

Ok ,
to save cookies , history, cache you must use a profile stored in a directory.
To do this with chrome change the beginning of function TChromeDriver.NewSession in WebDriver4L unit from :

Code: Pascal  [Select][+][-]

1. const
2.     Chrome_Param =
3.     '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome",'
4.     + ' "platformName": "any", "goog:chromeOptions": {"extensions": [],'
5.         + ' "args": []}}},'  // visible mode
6. //      + ' "args": ["--headless","--disable-gpu"]}}},' // hidden mode
7. //      + ' "args": ["--user-data-dir=d:/temp/lazSelenium"]}}},' // userdir
8.     + ' "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY",'
9.     + ' "goog:chromeOptions": {"extensions": [], "args": []}}}';  

with :

Code: Pascal  [Select][+][-]

1. const
2.     Chrome_Param =
3.     '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "chrome",'
4.     + ' "platformName": "any", "goog:chromeOptions": {"extensions": [],'
5. //      + ' "args": []}}},'  // visible mode
6. //      + ' "args": ["--headless","--disable-gpu"]}}},' // hidden mode
7.          + ' "args": ["--user-data-dir=d:/temp/lazSelenium"]}}},' // userdir
8.     + ' "desiredCapabilities": {"browserName": "chrome", "version": "", "platform": "ANY",'
9.     + ' "goog:chromeOptions": {"extensions": [], "args": []}}}';  

and adjust the user-data-dir to a read/write directory in your system.

[Fahmy Rofiq]

Thanks, thats work.

[lazzaro2023]

Hi Jurassic Pork,  thanks for WebDriver4L.  I have downloaded your demo to my platform, which is  Laz 2.0.10  FPC 3.2 Windows Server 2016.  I have downloaded chromedriver for 92.  (which matches my Chrome version).

When I compile and run the demo, the main window is displayed as expected.  However, when I click either of the buttons, I get the following error message:


Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865

To resolve the error, the version of chromedriver.exe must be the same as Google Chrome.

[TRon]

Thank you for yet another great project Jurassic Pork.

Small issue to mention: Unit WebDriver4L function TFireFoxDriver.NewSession const Firefox_Param currently reads:

Code: Pascal  [Select][+][-]

1. const
2.   Firefox_Param =
3.     '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", '
4.     + '"acceptInsecureCerts": false, "moz:firefoxOptions": ' +
5. //    '{"binary": "%s"}}},' +  // visible mode
6.     '{"binary": "%s",args:"-headless"}}},' +  // hidden mode
7.     '"desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": false, "moz:firefoxOptions": {"binary": "%s"}}}';
8.  

According to https://developer.mozilla.org/en-US/docs/Web/WebDriver/Capabilities/firefoxOptions that should read:

Code: Pascal  [Select][+][-]

1. const
2.   Firefox_Param =
3.     '{"capabilities": {"firstMatch": [{}], "alwaysMatch": {"browserName": "firefox", '
4.     + '"acceptInsecureCerts": false, "moz:firefoxOptions": ' +
5. //    '{"binary": "%s"}}},' +  // visible mode
6.     '{"binary": "%s","args":["-headless"]}}},' +  // hidden mode
7.     '"desiredCapabilities": {"browserName": "firefox", "acceptInsecureCerts": false, "moz:firefoxOptions": {"binary": "%s"}}}';
8.  
9.  

Note:

Code: [Select]

args:"-headless" vs "args":["-headless"]
When fix not applied the geckodriver will report back that the JSON request is invalid.


When using the function FindElementsByClassName the gecko driver/server reports:

Code: [Select]

1685771451493 webdriver::server DEBUG -> POST /session/91cf59aa-1503-4a17-9bbe-ffa5728fa2ee/elements { "value" : "page-title-text", "sessionid" : "91cf59aa-1503-4a17-9bbe-ffa5728fa2ee", "using" : "class name" }
1685771451493 webdriver::server DEBUG <- 400 Bad Request {"value":{"error":"invalid argument","message":"unknown variant `class name`, expected one of `css selector`, `link text`, `partial link text`, `tag name`, `xpath` at line 1 column 107","stacktrace":""}}

Apparently that is not supported.

[kapibara]

How to call Robot.ExecuteScript to click a button? So far I have used Python for web robots, but yesterday I got the WebDriver4L demo working under Linux and can't resist trying to switch! 

Here is my working Python code

Code: Python  [Select][+][-]

1. DownloadButton = WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[2]/div/button')))
2.  
3. browser.execute_script("arguments[0].click();", DownloadButton)
4.  

My attempt with WebDriver4L. I confess to not knowing exactly what I'm doing here..

Code: Pascal  [Select][+][-]

1. var
2.   DownloadButton: TWebElement;
3. begin
4.   DownloadButton:= Robot.FindElementByXPath('/html/body/div[2]/div/button');
5.   Robot.ExecuteScript('DownloadButton', 'arguments[0].click();')
6.  

And the error message:

Unexpected token (arguments) encountered

EDIT: Found an example that clicks the button and completes the download, but the downloaded textfile is garbled and unreadable for some reason?

Code: Pascal  [Select][+][-]

1.   Robot.ExecuteScript ( 'var xpath = "/html/body/div[2]/div/main/div[2]/article/div[3]/div[1]/div/div/div[3]/div[2]/div[2]/div/button";' +
2.                          'function getElementByXpath(path) {' +
3.                          'return document.evaluate(' +
4.                          '  path,' +
5.                          '  document,' +
6.                          '  null,' +
7.                          '  XPathResult.FIRST_ORDERED_NODE_TYPE,' +
8.                          '  null' +
9.                          ').singleNodeValue;' +
10.                          '}' +
11.                          'var element = getElementByXpath(xpath);' +
12.                          'if (!element) {' +
13.                          'throw new Error("Error: cannot find an element with XPath(" + xpath + ")");' +
14.                          '}' +
15.                          'element.click();' ) ;
16.  

code is borrowed from here: http://www.tauhidslab.my.id/2024/02/scrape-data-dari-website-kpu-dengan.html

EDIT2:

After checking the downloaded .CSV file more closely, I found that it was just in LibreOffice the content seemed garbled. CudaText, MousePad etc showed it just fine. It turned out to be US-ASCII encoding. I converted the file to UTF8 instead and LibreOffice read it fine too.

[paweld]

Clicking the button is relatively easy. You need to find the button on the page and simply click it programmatically - I usually search by ID or Name.

Code: Pascal  [Select][+][-]

1. uses
2.   Webdriver4L;
3.  
4. procedure TForm1.Button1Click(Sender: TObject);
5. var
6.   Robot: TWebDriver;
7.   downbtn: TWebElement;
8. begin
9.   try
10.     Robot := TFireFoxDriver.Create(nil);
11.     Robot.HideWindow := True;     //hide robot window
12.     Robot.StartDriver(ExtractFileDir(ParamStr(0)) + '\geckodriver.exe');  //path to firefox webdriver exe
13.     Robot.NewSession;
14.     Robot.GetURL('https://brudnopis.ovh/downloadtest.html');
15.     downbtn := Robot.FindElementByID('downloadBtn'); //button id (from page source)
16.     if not downbtn.IsEmpty then  //if find button
17.       downbtn.Click;  //click the button
18.     Robot.Quit;
19.   finally
20.     Robot.Free;
21.   end;
22. end;  

[af0815]

BTW: I have written an issue on the repro and ask, if it is allowed to fork ( https://github.com/Ericwang1104/WebDriver4D/issues/6 ) . I have seen, EricWang1104 is from time to time available, so it is possible to get an answer.

I think this is the fastest and easiest way to clear the situation (for Jurrasic Pork and all working on the project).