Browser Automation with selenium

[Jurassic Pork]

hello,
i have found this library to do that with pascal language : Webdriver For Delphi lazarus.
The problem is that the lazarus part isn't done.
1 - the main problem is the json library used  in the unit Webdriver4D.pas : JsonDataObjects  . How to replace it with fcl-json and jsonparser ?
for example how to replace the code in brackets with the equivalent  of the fcl-json library :

Code: Pascal  [Select][+][-]

1. function TWebDriver.ExecuteScript(const Script: string;
2.   const Args: string = '[]'): string;
3. var
4.   Command: string;
5.   Data: string;
6.   Resp: string;
7.   Json: TJsonObject;
8. begin
9.   Json := TJsonObject.Create;
10.   try
11.     Command := Host + '/session/' + FSessionID + '/execute/sync';
12. {   Json.S['script'] := Script;
13.     Json.A['args'].FromJSON(Args);
14.     Data := Json.ToJSON();    }
15.     Resp := ExecuteCommand(cPost, Command, Data);
16.     result := ProcResponse(Resp);
17.   Finally
18.     FreeAndNil(Json);
19.   end;
20. end;            

Friendly, J.P

[Leledumbo]

.S[] -> https://www.freepascal.org/docs-html/current/fcl/fpjson/tjsonobject.strings.html
.A[] -> https://www.freepascal.org/docs-html/current/fcl/fpjson/tjsonobject.arrays.html
.FromJSON -> https://www.freepascal.org/docs-html/current/fcl/fpjson/getjson.html (not a method, but same functionality to convert a JSON formatted string to a JSON data, probably needs a cast as well)
.ToJSON -> either https://www.freepascal.org/docs-html/current/fcl/fpjson/tjsondata.formatjson.html or https://www.freepascal.org/docs-html/current/fcl/fpjson/tjsondata.asjson.html, depending on whether you want to format it or not

[Jurassic Pork]

hello,
thanks Leledumbo, i have found how to do that with  fpjson :

Code: Pascal  [Select][+][-]

1. function TWebDriver.ExecuteScript(const Script: string;
2.   const Args: string = '[]'): string;
3. var
4.   Command: string;
5.   Data: string;
6.   Resp: string;
7.   Json: TJsonObject;
8. begin
9.   Json := TJsonObject.Create(['script', Script]);
10.   try
11.     Command := Host + '/session/' + FSessionID + '/execute/sync';
12.     Json.Add('args',GETJson(Args));
13.     Data := Json.AsJSON;
14.     Resp := ExecuteCommand(cPost, Command, Data);
15.     result := ProcResponse(Resp);
16.   Finally
17.     FreeAndNil(Json);
18.   end;
19. end;

I started a lazarus version of WebDriver4D ( name WebDriver4L) .
replace the delphi unit httpclient with the lazarus unit   fphttpclient
replace the delphi unit JsonDataObjects with the lazarus units fpson, jsonparser
version of WebDriver4L is alpha but it begins to work :
Example of code to open chrome and clear image and files cache  in  settings :

Code: Pascal  [Select][+][-]

1. implementation
2. uses  Webdriver4L;
3. {$R *.lfm}
4.  
5. { TForm1 }
6.  
7. procedure TForm1.Button2Click(Sender: TObject);
8. const CacheDlg = 'return document.querySelector(''settings-ui'')' +
9.                  '.shadowRoot.querySelector(''settings-main'')' +
10.                  '.shadowRoot.querySelector(''settings-basic-page'')' +
11.                  '.shadowRoot.querySelector(''settings-section > settings-privacy-page'')' +
12.                  '.shadowRoot.querySelector(''settings-clear-browsing-data-dialog'')';
13. const btClear  = '.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +
14.                  '.querySelector(''#clearBrowsingDataConfirm'')';
15. const chkboxH1  = '.shadowRoot.querySelector(''iron-pages'')' +
16.                   '.querySelector(''#browsingCheckboxBasic'')';
17. const chkboxH2  = '.shadowRoot.querySelector(''iron-pages'')' +
18.                   '.querySelector(''#cookiesCheckboxBasic'')';
19. const chkboxH3  = '.shadowRoot.querySelector(''iron-pages'')' +
20.                   '.querySelector(''#cacheCheckboxBasic'')';
21. var Robot2 : TWebDriver;
22.     Element : TWebElement;
23. begin
24.   Robot2 := TChromeDriver.Create(nil);
25.   Robot2.StartDriver('chromedriver.exe');
26.   Robot2.NewSession;
27.   Robot2.Implicitly_Wait(1000);
28.   Robot2.Set_Window_Size(640, 640);
29.   Sleep(5000);
30.   Robot2.GetURL('chrome://settings/clearBrowserData');
31.   Sleep(2000);
32.   Element := Robot2.FindElementByXPath('//settings-ui');
33.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH1 + ';');
34.   Memo1.Append('Etiquette case à cocher 1 : ' + Element.AttributeValue('label'));
35.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
36.   Sleep(1000);
37.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';');
38.   Memo1.Append('Etiquette case à cocher 2 : ' + Element.AttributeValue('label'));
39.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
40.   Sleep(1000);
41.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';');
42.   Memo1.Append('Etiquette case à cocher 3 : ' + Element.AttributeValue('label'));
43.   if Element.PropertyValue('checked') = 'False' then  Element.Click();
44.   Sleep(1000);
45.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + btClear + ';');
46.   Element.Click();
47.   Sleep(30000);
48.   Robot2.Clear;
49. end;

Friendly, J.P

[Jurassic Pork]

hello,
now my alpha version of webdriver4L is working on linux (centos 8 - lazarus 2.0.8 ) . Screenshots work.

Code: Pascal  [Select][+][-]

1. implementation
2. uses  Webdriver4L;
3. {$R *.lfm}
4. { TForm1 }
5. procedure TForm1.Button2Click(Sender: TObject);
6. const CacheDlg = 'return document.querySelector(''settings-ui'')' +
7.                  '.shadowRoot.querySelector(''settings-main'')' +
8.                  '.shadowRoot.querySelector(''settings-basic-page'')' +
9.                  '.shadowRoot.querySelector(''settings-section > settings-privacy-page'')' +
10.                  '.shadowRoot.querySelector(''settings-clear-browsing-data-dialog'')';
11. const btClear  = '.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +
12.                  '.querySelector(''#clearBrowsingDataConfirm'')';
13. const chkboxH1  = '.shadowRoot.querySelector(''iron-pages'')' +
14.                   '.querySelector(''#browsingCheckboxBasic'')';
15. const chkboxH2  = '.shadowRoot.querySelector(''iron-pages'')' +
16.                   '.querySelector(''#cookiesCheckboxBasic'')';
17. const chkboxH3  = '.shadowRoot.querySelector(''iron-pages'')' +
18.                   '.querySelector(''#cacheCheckboxBasic'')';
19. var Robot2 : TWebDriver;
20.     Element : TWebElement;
21.     png :  TPortableNetworkGraphic;
22. begin
23.   Robot2 := TChromeDriver.Create(nil);
24.   Robot2.StartDriver('chromedriver');
25.   Sleep(5000);
26.   Robot2.NewSession;
27.   Robot2.Implicitly_Wait(1000);
28.   Robot2.Set_Window_Size(640, 640);
29.   Robot2.GetURL('chrome://settings/clearBrowserData');
30.   Sleep(2000);
31.   Robot2.ScreenShot('clearBrowserData1.png');
32.   Element := Robot2.FindElementByXPath('//settings-ui');
33.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH1 + ';');
34.   Memo1.Append('Etiquette case à cocher 1 : ' + Element.AttributeValue('label'));
35.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
36.   Sleep(1000);
37.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';');
38.   Memo1.Append('Etiquette case à cocher 2 : ' + Element.AttributeValue('label'));
39.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
40.   Sleep(1000);
41.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';');
42.   Memo1.Append('Etiquette case à cocher 3 : ' + Element.AttributeValue('label'));
43.   if Element.PropertyValue('checked') = 'False' then  Element.Click();
44.   Sleep(1000);
45.   Robot2.ScreenShot('clearBrowserData2.png');
46.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + btClear + ';');
47.   Element.Click();
48.   Application.ProcessMessages;
49.   Sleep(10000);
50.   Robot2.Clear;
51.   png := TPortableNetworkGraphic.Create;
52.   png.Assign(Self.GetFormImage);
53.   png.SaveToFile('clearBrowserDataLazForm.png');
54.   png.free;
55. end;

Results in Attachments (click on the first attachment to see animation)

Friendly, J.P

[Blade]

This is very useful.  Thanks for sharing.

[devEric69]

Hello @Jurassic Pork,

It's interesting to know.

Just for information, Selenium's successor is called Katalon (HMI web automation). It can use globally (i haven't looked at how deep compatible it is exactly) the same scripts as Selenium, which look like the ones coded as parameters in your Robot.ExecuteScript method.

Nevertheless, what is interesting to know, is that such version of a plugin (Selenium or Katalon) generally imposes a compatibility constraint, with a minimal version of a browser. Your solution is a way to get rid of it, if need be.

[systemgvp]

Hi, I would like to try your library, can you post it?

[tobiastromm]

Hi, is Webdriver4L available to download? 

[Jurassic Pork]

hello,
the problem is  that WebDriver4L is builded from EricWang  WebDriver4D  Sources  and it seems that there is no licence on the code of this project. I don't know if i can publish my derivated work and how create a new project in github (not a fork) from the webdriver4D sources.
Friendly, J.P

[Jurassic Pork]

hello,
   Finally i put WebDriver4L source code on the forum. Hope to not go to jail 

WebDriver4L   Selenium Webdriver for Lazarus   // Jurassic Pork 01/2021
Derivative work from Ericwang1104/WebDriver4D  : https://github.com/Ericwang1104/WebDriver4D
Source code modified to use fphttpclient, fpjson,jsonparser.
The project is in alpha stage.
A demo project is included :  get a screenshot of the lazarus forum, get all the recent topics and display then in a memo, get stats.
the demo project has been tested on windows 10 lazarus 2.0.10   with chrome browser and on Ubuntu 20.04 Lazarus 2.0.10 with firefox browser.
You must download the selenium Webdriver for your browser and put it in your project folder. You must use a compatible driver version of your browser :
For Chrome the selenium drivers are here : https://chromedriver.chromium.org/downloads
For Firefox the selenium drivers are here : https://github.com/mozilla/geckodriver/releases
For example if you use Chrome 87.x 64 bits on Windows  download a 87 64 bits version of the driver.

The Software is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders X be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the Software.

Ericwang1104/WebDriver4D has no license so this project will not be published on github. No activity since March 2019

Source code and demo project in attachments

[Jurassic Pork]

hello,
WebDriver4L works also on MacOS with chrome browser.
1 - Put the MacOs chrome webDriver in the lib folder of your user home directory
2 - Modify the demo project source code :

Code: Pascal  [Select][+][-]

1.  {$ELSE}
2.   Robot := TChromeDriver.Create(nil);
3.   // Robot := TFireFoxDriver.Create(nil);
4.   Robot.StartDriver( GetUserDir + '/lib/chromedriver');
5.   // Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/geckodriver');
6.   {$ENDIF}

Code: Pascal  [Select][+][-]

1.  Robot.ScreenShot(GetUserDir + '/Documents/ForumLazarusScreenShot.png');
2.  

Tested with chrome 88 Lazarus 2.0.10 on MacOs Catalina

Friendly, J.P

[Jurassic Pork]

hello,
1 -

For example if you use Chrome 87.x 64 bits on Windows  download a 87 64 bits version of the driver.

   no 64 bits version of selenium webdriver for chrome on windows only 32bits but it works !

2 - More accurate source code in unit1.pas of demo project to find the recent topics and stats :

Code: Pascal  [Select][+][-]

1. //var topics,authors,dateMessage : TWebElements;
2.   topics := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/a');
3.   authors := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/b');
4.   dateMessage := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/span');
5.   For i:=0 to topics.Count - 1 do
6.   begin
7.    Memo1.Append(topics.Items[i].AttributeValue('Title') + ' by '+
8.                 authors.Items[i].Text + ' ' +
9.                 dateMessage.Items[i].Text);
10.   end;
11.   stats := Robot.FindElementByClassName('inline');
12.   Memo1.Append('=================================');
13.   Memo1.Append('Stats : ' + stats.Text);          
14.  

Friendly, J.P

[tobiastromm]

Hi @Jurassic Pork.

First of all, thanks for this.

I am finally tryng that and it seens that some things are missing or I am not finding the correct way to access then. Any chance you help me with that?

1 - sendKeys
- sendKey is working, example: Robot.FindElementByID('inputLogin').SendKey('userLogin');
- sendKeys, in the other hand, appear that does not exist: Error: identifier idents no member "SendKeys".

Links: https://www.selenium.dev/documentation/en/webdriver/keyboard/
They also have a list of common keys, that are constants, that we can pass on SendKeys. Can we have these constants also? Here is the list: https://www.w3.org/TR/webdriver/#keyboard-actions and  https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.keys.html

2 - switchTo
- I can create a New tab, by using this way: Robot.ExecuteScript('window.open()');
- The problem is that switchTo (used to switch between tabs) appear that does not exist. I think this need to be implemented.

This https://www.selenium.dev/documentation/en/webdriver/browser_manipulation/#create-new-window-or-new-tab-and-switch is the right way to open a new tab (instead of ExecuteScript) and manipulate these tabs:

Code: Pascal  [Select][+][-]

1. // Opens a new tab and switches to new tab
2. driver.SwitchTo().NewWindow(WindowType.Tab)
3.  
4. // Opens a new window and switches to new window
5. driver.SwitchTo().NewWindow(WindowType.Window)
6.  
7. //Close the tab or window
8. driver.Close();
9.  
10. //Switch back to the old tab or window
11. driver.SwitchTo().Window(originalWindow);
12.  

Here https://stackoverflow.com/questions/12729265/switch-tabs-using-selenium-webdriver-with-java I found how to get tab IDs and switch beetween these tabs (Java code):

Code: Pascal  [Select][+][-]

1. ArrayList<String> tabs2 = new ArrayList<String> (driver.getWindowHandles());
2. driver.switchTo().window(tabs2.get(1));
3. driver.close();
4. driver.switchTo().window(tabs2.get(0));
5.  

Maybe this link https://blog.testproject.io/2020/06/30/selenium-4-new-window-tab-screenshots/ can also help to better understand  how it work, so it can be implemented.

Well, that's it for now.

[RedOctober]

Hi Jurassic Pork,  thanks for WebDriver4L.  I have downloaded your demo to my platform, which is  Laz 2.0.10  FPC 3.2 Windows Server 2016.  I have downloaded chromedriver for 92.  (which matches my Chrome version).

When I compile and run the demo, the main window is displayed as expected.  However, when I click either of the buttons, I get the following error message:


Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865

[Jurassic Pork]

When I compile and run the demo, the main window is displayed as expected.  However, when I click either of the buttons, I get the following error message:

Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865

i have this problem when i use fpdebug as debugger , with gdb or running without debug it seems OK