Recent

Author Topic: Browser Automation with selenium  (Read 15420 times)

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Browser Automation with selenium
« on: June 05, 2020, 01:58:30 am »
hello,
i have found this library to do that with pascal language : Webdriver For Delphi lazarus.
The problem is that the lazarus part isn't done.
1 - the main problem is the json library used  in the unit Webdriver4D.pas : JsonDataObjects  . How to replace it with fcl-json and jsonparser ?
for example how to replace the code in brackets with the equivalent  of the fcl-json library :
Code: Pascal  [Select][+][-]
  1. function TWebDriver.ExecuteScript(const Script: string;
  2.   const Args: string = '[]'): string;
  3. var
  4.   Command: string;
  5.   Data: string;
  6.   Resp: string;
  7.   Json: TJsonObject;
  8. begin
  9.   Json := TJsonObject.Create;
  10.   try
  11.     Command := Host + '/session/' + FSessionID + '/execute/sync';
  12. {   Json.S['script'] := Script;
  13.     Json.A['args'].FromJSON(Args);
  14.     Data := Json.ToJSON();    }
  15.     Resp := ExecuteCommand(cPost, Command, Data);
  16.     result := ProcResponse(Resp);
  17.   Finally
  18.     FreeAndNil(Json);
  19.   end;
  20. end;            

Friendly, J.P
« Last Edit: June 05, 2020, 02:21:38 am by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Leledumbo

  • Hero Member
  • *****
  • Posts: 8744
  • Programming + Glam Metal + Tae Kwon Do = Me

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #2 on: June 13, 2020, 02:34:04 pm »
hello,
thanks Leledumbo, i have found how to do that with  fpjson :
Code: Pascal  [Select][+][-]
  1. function TWebDriver.ExecuteScript(const Script: string;
  2.   const Args: string = '[]'): string;
  3. var
  4.   Command: string;
  5.   Data: string;
  6.   Resp: string;
  7.   Json: TJsonObject;
  8. begin
  9.   Json := TJsonObject.Create(['script', Script]);
  10.   try
  11.     Command := Host + '/session/' + FSessionID + '/execute/sync';
  12.     Json.Add('args',GETJson(Args));
  13.     Data := Json.AsJSON;
  14.     Resp := ExecuteCommand(cPost, Command, Data);
  15.     result := ProcResponse(Resp);
  16.   Finally
  17.     FreeAndNil(Json);
  18.   end;
  19. end;
I started a lazarus version of WebDriver4D ( name WebDriver4L) .
replace the delphi unit httpclient with the lazarus unit   fphttpclient
replace the delphi unit JsonDataObjects with the lazarus units fpson, jsonparser
version of WebDriver4L is alpha but it begins to work :
Example of code to open chrome and clear image and files cache  in  settings :
Code: Pascal  [Select][+][-]
  1. implementation
  2. uses  Webdriver4L;
  3. {$R *.lfm}
  4.  
  5. { TForm1 }
  6.  
  7. procedure TForm1.Button2Click(Sender: TObject);
  8. const CacheDlg = 'return document.querySelector(''settings-ui'')' +
  9.                  '.shadowRoot.querySelector(''settings-main'')' +
  10.                  '.shadowRoot.querySelector(''settings-basic-page'')' +
  11.                  '.shadowRoot.querySelector(''settings-section > settings-privacy-page'')' +
  12.                  '.shadowRoot.querySelector(''settings-clear-browsing-data-dialog'')';
  13. const btClear  = '.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +
  14.                  '.querySelector(''#clearBrowsingDataConfirm'')';
  15. const chkboxH1  = '.shadowRoot.querySelector(''iron-pages'')' +
  16.                   '.querySelector(''#browsingCheckboxBasic'')';
  17. const chkboxH2  = '.shadowRoot.querySelector(''iron-pages'')' +
  18.                   '.querySelector(''#cookiesCheckboxBasic'')';
  19. const chkboxH3  = '.shadowRoot.querySelector(''iron-pages'')' +
  20.                   '.querySelector(''#cacheCheckboxBasic'')';
  21. var Robot2 : TWebDriver;
  22.     Element : TWebElement;
  23. begin
  24.   Robot2 := TChromeDriver.Create(nil);
  25.   Robot2.StartDriver('chromedriver.exe');
  26.   Robot2.NewSession;
  27.   Robot2.Implicitly_Wait(1000);
  28.   Robot2.Set_Window_Size(640, 640);
  29.   Sleep(5000);
  30.   Robot2.GetURL('chrome://settings/clearBrowserData');
  31.   Sleep(2000);
  32.   Element := Robot2.FindElementByXPath('//settings-ui');
  33.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH1 + ';');
  34.   Memo1.Append('Etiquette case à cocher 1 : ' + Element.AttributeValue('label'));
  35.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
  36.   Sleep(1000);
  37.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';');
  38.   Memo1.Append('Etiquette case à cocher 2 : ' + Element.AttributeValue('label'));
  39.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
  40.   Sleep(1000);
  41.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';');
  42.   Memo1.Append('Etiquette case à cocher 3 : ' + Element.AttributeValue('label'));
  43.   if Element.PropertyValue('checked') = 'False' then  Element.Click();
  44.   Sleep(1000);
  45.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + btClear + ';');
  46.   Element.Click();
  47.   Sleep(30000);
  48.   Robot2.Clear;
  49. end;

Friendly, J.P
« Last Edit: June 13, 2020, 02:36:26 pm by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #3 on: June 15, 2020, 02:39:34 am »
hello,
now my alpha version of webdriver4L is working on linux (centos 8 - lazarus 2.0.8 ) . Screenshots work.
Code: Pascal  [Select][+][-]
  1. implementation
  2. uses  Webdriver4L;
  3. {$R *.lfm}
  4. { TForm1 }
  5. procedure TForm1.Button2Click(Sender: TObject);
  6. const CacheDlg = 'return document.querySelector(''settings-ui'')' +
  7.                  '.shadowRoot.querySelector(''settings-main'')' +
  8.                  '.shadowRoot.querySelector(''settings-basic-page'')' +
  9.                  '.shadowRoot.querySelector(''settings-section > settings-privacy-page'')' +
  10.                  '.shadowRoot.querySelector(''settings-clear-browsing-data-dialog'')';
  11. const btClear  = '.shadowRoot.querySelector(''#clearBrowsingDataDialog'')' +
  12.                  '.querySelector(''#clearBrowsingDataConfirm'')';
  13. const chkboxH1  = '.shadowRoot.querySelector(''iron-pages'')' +
  14.                   '.querySelector(''#browsingCheckboxBasic'')';
  15. const chkboxH2  = '.shadowRoot.querySelector(''iron-pages'')' +
  16.                   '.querySelector(''#cookiesCheckboxBasic'')';
  17. const chkboxH3  = '.shadowRoot.querySelector(''iron-pages'')' +
  18.                   '.querySelector(''#cacheCheckboxBasic'')';
  19. var Robot2 : TWebDriver;
  20.     Element : TWebElement;
  21.     png :  TPortableNetworkGraphic;
  22. begin
  23.   Robot2 := TChromeDriver.Create(nil);
  24.   Robot2.StartDriver('chromedriver');
  25.   Sleep(5000);
  26.   Robot2.NewSession;
  27.   Robot2.Implicitly_Wait(1000);
  28.   Robot2.Set_Window_Size(640, 640);
  29.   Robot2.GetURL('chrome://settings/clearBrowserData');
  30.   Sleep(2000);
  31.   Robot2.ScreenShot('clearBrowserData1.png');
  32.   Element := Robot2.FindElementByXPath('//settings-ui');
  33.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH1 + ';');
  34.   Memo1.Append('Etiquette case à cocher 1 : ' + Element.AttributeValue('label'));
  35.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
  36.   Sleep(1000);
  37.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH2 + ';');
  38.   Memo1.Append('Etiquette case à cocher 2 : ' + Element.AttributeValue('label'));
  39.   if Element.PropertyValue('checked') = 'True' then  Element.Click();
  40.   Sleep(1000);
  41.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + chkboxH3 + ';');
  42.   Memo1.Append('Etiquette case à cocher 3 : ' + Element.AttributeValue('label'));
  43.   if Element.PropertyValue('checked') = 'False' then  Element.Click();
  44.   Sleep(1000);
  45.   Robot2.ScreenShot('clearBrowserData2.png');
  46.   Element.ElementData := Robot2.ExecuteScript(CacheDlg + btClear + ';');
  47.   Element.Click();
  48.   Application.ProcessMessages;
  49.   Sleep(10000);
  50.   Robot2.Clear;
  51.   png := TPortableNetworkGraphic.Create;
  52.   png.Assign(Self.GetFormImage);
  53.   png.SaveToFile('clearBrowserDataLazForm.png');
  54.   png.free;
  55. end;

Results in Attachments (click on the first attachment to see animation)

Friendly, J.P
« Last Edit: June 15, 2020, 02:41:17 am by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Blade

  • Full Member
  • ***
  • Posts: 177
Re: Browser Automation with selenium
« Reply #4 on: August 05, 2020, 06:57:00 pm »
This is very useful.  Thanks for sharing.

devEric69

  • Hero Member
  • *****
  • Posts: 648
Re: Browser Automation with selenium
« Reply #5 on: August 06, 2020, 02:17:08 pm »
Hello @Jurassic Pork,

It's interesting to know.

Just for information, Selenium's successor is called Katalon (HMI web automation). It can use globally (i haven't looked at how deep compatible it is exactly) the same scripts as Selenium, which look like the ones coded as parameters in your Robot.ExecuteScript method.

Nevertheless, what is interesting to know, is that such version of a plugin (Selenium or Katalon) generally imposes a compatibility constraint, with a minimal version of a browser. Your solution is a way to get rid of it, if need be.
use: Linux 64 bits (Ubuntu 20.04 LTS).
Lazarus version: 2.0.4 (svn revision: 62502M) compiled with fpc 3.0.4 - fpDebug \ Dwarf3.

systemgvp

  • New Member
  • *
  • Posts: 29
Re: Browser Automation with selenium
« Reply #6 on: December 09, 2020, 05:07:07 pm »
Hi, I would like to try your library, can you post it?

tobiastromm

  • Newbie
  • Posts: 4
Re: Browser Automation with selenium
« Reply #7 on: January 23, 2021, 06:23:27 am »
Hi, is Webdriver4L available to download?  :D

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #8 on: January 23, 2021, 10:43:40 am »
hello,
the problem is  that WebDriver4L is builded from EricWang  WebDriver4D  Sources  and it seems that there is no licence on the code of this project. I don't know if i can publish my derivated work and how create a new project in github (not a fork) from the webdriver4D sources.
Friendly, J.P
« Last Edit: January 23, 2021, 10:46:22 am by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #9 on: January 25, 2021, 04:29:44 pm »
hello,
 :-X  Finally i put WebDriver4L source code on the forum. Hope to not go to jail  :-\

Quote
WebDriver4L   Selenium Webdriver for Lazarus   // Jurassic Pork 01/2021
Derivative work from Ericwang1104/WebDriver4D  : https://github.com/Ericwang1104/WebDriver4D
Source code modified to use fphttpclient, fpjson,jsonparser.
The project is in alpha stage.
A demo project is included :  get a screenshot of the lazarus forum, get all the recent topics and display then in a memo, get stats.
the demo project has been tested on windows 10 lazarus 2.0.10   with chrome browser and on Ubuntu 20.04 Lazarus 2.0.10 with firefox browser.
You must download the selenium Webdriver for your browser and put it in your project folder. You must use a compatible driver version of your browser :
For Chrome the selenium drivers are here : https://chromedriver.chromium.org/downloads
For Firefox the selenium drivers are here : https://github.com/mozilla/geckodriver/releases
For example if you use Chrome 87.x 64 bits on Windows  download a 87 64 bits version of the driver.

The Software is provided “as is”, without warranty of any kind, express or implied, including but not limited to the warranties of merchantability, fitness for a particular purpose and noninfringement. In no event shall the authors or copyright holders X be liable for any claim, damages or other liability, whether in an action of contract, tort or otherwise, arising from, out of or in connection with the software or the use or other dealings in the Software.

Ericwang1104/WebDriver4D has no license so this project will not be published on github. No activity since March 2019

Source code and demo project in attachments
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #10 on: January 26, 2021, 03:06:56 am »
hello,
WebDriver4L works also on MacOS with chrome browser.
1 - Put the MacOs chrome webDriver in the lib folder of your user home directory
2 - Modify the demo project source code :
Code: Pascal  [Select][+][-]
  1.  {$ELSE}
  2.   Robot := TChromeDriver.Create(nil);
  3.   // Robot := TFireFoxDriver.Create(nil);
  4.   Robot.StartDriver( GetUserDir + '/lib/chromedriver');
  5.   // Robot.StartDriver(ExtractFileDir(Paramstr(0)) + '/geckodriver');
  6.   {$ENDIF}

Code: Pascal  [Select][+][-]
  1.  Robot.ScreenShot(GetUserDir + '/Documents/ForumLazarusScreenShot.png');
  2.  

Tested with chrome 88 Lazarus 2.0.10 on MacOs Catalina

Friendly, J.P

« Last Edit: January 26, 2021, 03:13:09 am by Jurassic Pork »
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #11 on: January 27, 2021, 03:24:08 pm »
hello,
1 -
For example if you use Chrome 87.x 64 bits on Windows  download a 87 64 bits version of the driver.
:-[   no 64 bits version of selenium webdriver for chrome on windows only 32bits but it works !

2 - More accurate source code in unit1.pas of demo project to find the recent topics and stats :
Code: Pascal  [Select][+][-]
  1. //var topics,authors,dateMessage : TWebElements;
  2.   topics := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/a');
  3.   authors := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/b');
  4.   dateMessage := Robot.FindElementsByXPath('//ul[@class="recent_topics"]/li/span');
  5.   For i:=0 to topics.Count - 1 do
  6.   begin
  7.    Memo1.Append(topics.Items[i].AttributeValue('Title') + ' by '+
  8.                 authors.Items[i].Text + ' ' +
  9.                 dateMessage.Items[i].Text);
  10.   end;
  11.   stats := Robot.FindElementByClassName('inline');
  12.   Memo1.Append('=================================');
  13.   Memo1.Append('Stats : ' + stats.Text);          
  14.  

Friendly, J.P
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

tobiastromm

  • Newbie
  • Posts: 4
Re: Browser Automation with selenium
« Reply #12 on: June 23, 2021, 06:33:31 pm »
Hi @Jurassic Pork.

First of all, thanks for this.

I am finally tryng that and it seens that some things are missing or I am not finding the correct way to access then. Any chance you help me with that?

1 - sendKeys
- sendKey is working, example: Robot.FindElementByID('inputLogin').SendKey('userLogin');
- sendKeys, in the other hand, appear that does not exist: Error: identifier idents no member "SendKeys".

Links: https://www.selenium.dev/documentation/en/webdriver/keyboard/
They also have a list of common keys, that are constants, that we can pass on SendKeys. Can we have these constants also? Here is the list: https://www.w3.org/TR/webdriver/#keyboard-actions and  https://www.selenium.dev/selenium/docs/api/py/webdriver/selenium.webdriver.common.keys.html

2 - switchTo
- I can create a New tab, by using this way: Robot.ExecuteScript('window.open()');
- The problem is that switchTo (used to switch between tabs) appear that does not exist. I think this need to be implemented.

This https://www.selenium.dev/documentation/en/webdriver/browser_manipulation/#create-new-window-or-new-tab-and-switch is the right way to open a new tab (instead of ExecuteScript) and manipulate these tabs:

Code: Pascal  [Select][+][-]
  1. // Opens a new tab and switches to new tab
  2. driver.SwitchTo().NewWindow(WindowType.Tab)
  3.  
  4. // Opens a new window and switches to new window
  5. driver.SwitchTo().NewWindow(WindowType.Window)
  6.  
  7. //Close the tab or window
  8. driver.Close();
  9.  
  10. //Switch back to the old tab or window
  11. driver.SwitchTo().Window(originalWindow);
  12.  

Here https://stackoverflow.com/questions/12729265/switch-tabs-using-selenium-webdriver-with-java I found how to get tab IDs and switch beetween these tabs (Java code):

Code: Pascal  [Select][+][-]
  1. ArrayList<String> tabs2 = new ArrayList<String> (driver.getWindowHandles());
  2. driver.switchTo().window(tabs2.get(1));
  3. driver.close();
  4. driver.switchTo().window(tabs2.get(0));
  5.  

Maybe this link https://blog.testproject.io/2020/06/30/selenium-4-new-window-tab-screenshots/ can also help to better understand  how it work, so it can be implemented.

Well, that's it for now.
« Last Edit: June 23, 2021, 06:38:55 pm by tobiastromm »

RedOctober

  • Sr. Member
  • ****
  • Posts: 450
Re: Browser Automation with selenium
« Reply #13 on: August 30, 2021, 06:10:13 pm »
Hi Jurassic Pork,  thanks for WebDriver4L.  I have downloaded your demo to my platform, which is  Laz 2.0.10  FPC 3.2 Windows Server 2016.  I have downloaded chromedriver for 92.  (which matches my Chrome version).

When I compile and run the demo, the main window is displayed as expected.  However, when I click either of the buttons, I get the following error message:


Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865


Jurassic Pork

  • Hero Member
  • *****
  • Posts: 1228
Re: Browser Automation with selenium
« Reply #14 on: August 30, 2021, 06:22:02 pm »
When I compile and run the demo, the main window is displayed as expected.  However, when I click either of the buttons, I get the following error message:

Project demoWebDriver4L raised exception class 'EJSON' with message:
Unknown object member: "sessionId"

 At address 100176865
i have this problem when i use fpdebug as debugger , with gdb or running without debug it seems OK
Jurassic computer : Sinclair ZX81 - Zilog Z80A à 3,25 MHz - RAM 1 Ko - ROM 8 Ko

 

TinyPortal © 2005-2018