Recent

Author Topic: Browser Automation with selenium  (Read 24361 times)

gidesa

  • Full Member
  • ***
  • Posts: 145
Re: Browser Automation with selenium
« Reply #30 on: August 23, 2024, 06:11:51 pm »
BTW: I have written an issue on the repro and ask, if it is allowed to fork ( https://github.com/Ericwang1104/WebDriver4D/issues/6 ) . I have seen, EricWang1104 is from time to time available, so it is possible to get an answer.

I think this is the fastest and easiest way to clear the situation (for Jurrasic Pork and all working on the project).

Hello, I have done a fork of EricWang Delphi project, where I have completed a large part of W3C webdriver standard.
By the way, EricWang's project seems no more updated from some years.


https://github.com/gidesa/WebDriver4D

af0815

  • Hero Member
  • *****
  • Posts: 1380
Re: Browser Automation with selenium
« Reply #31 on: August 24, 2024, 08:34:01 am »
By the way, EricWang's project seems no more updated from some years.


https://github.com/gidesa/WebDriver4D
Your repro is also 3 years not updated :-) And a new fork without clearing the license is IMHO not the best soloution. To use the code, i have to known if it is GPL or LGPL or ...
regards
Andreas

gidesa

  • Full Member
  • ***
  • Posts: 145
Re: Browser Automation with selenium
« Reply #32 on: August 24, 2024, 06:09:35 pm »
Your repro is also 3 years not updated :-)

Aha, ah, yes! Although 3 years is better than 5-6  :)

And a new fork without clearing the license is IMHO not the best soloution. To use the code, i have to known if it is GPL or LGPL or ...

Indeed seems that  Github code in (public) repository WITHOUT explicit license is a totally exclusive one:
https://opensource.stackexchange.com/questions/1720/what-can-i-assume-if-a-publicly-published-project-has-no-license
One can only view the code, and fork the repository ....  :o


Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #33 on: September 01, 2024, 10:50:08 pm »
I recently found this tool and it opened up a world I didn't know about.
Thank you very much for this  :D

Using @paweld's example as a basis, I tried to save a page:
Code: Pascal  [Select][+][-]
  1. ..
  2. Robot.GetURL('https://retrostylegames-com.translate.goog/blog/best-looking-isometric-games/?_x_tr_sl=auto&_x_tr_tl=it&_x_tr_hl=it&_x_tr_pto=wapp');
  3. Robot.SaveCurDocToFile('d:\temp\text.html');
  4. ..
  5.  

But is there a way to save it completely translated? I noticed that when you scroll a page (browser), the displayed portion is updated. Is it possible to force the entire translation or simulate the scroll action?

I did some research but I don't know how to get the result with firefox or chrome

https://stackoverflow.com/questions/53717431/chromedriver-how-to-translate-a-page-using-selenium

https://stackoverflow.com/questions/68761826/running-forced-automated-translation-in-google-chrome-and-edge-translation-not

vfclists

  • Hero Member
  • *****
  • Posts: 1146
    • HowTos Considered Harmful?
Re: Browser Automation with selenium
« Reply #34 on: September 02, 2024, 02:05:53 am »
By the way, EricWang's project seems no more updated from some years.


https://github.com/gidesa/WebDriver4D
Your repro is also 3 years not updated :-) And a new fork without clearing the license is IMHO not the best soloution. To use the code, i have to known if it is GPL or LGPL or ...

It may be covered by the Selenium license and those of other libraries it depends on.
Lazarus 3.0/FPC 3.2.2

kapibara

  • Hero Member
  • *****
  • Posts: 629
Re: Browser Automation with selenium
« Reply #35 on: September 02, 2024, 04:36:49 am »
To scroll on a page you can use ExecuteScript to call Element.ScrollIntoView :

Code: Pascal  [Select][+][-]
  1.   s:= WebDriver.ExecuteScript( 'var xpath = "/element/of/the/xpath/that/you/want/to/use";' +
  2.                         'function getElementByXpath(path) {' +
  3.                         'return document.evaluate(' +
  4.                         '  path,' +
  5.                         '  document,' +
  6.                         '  null,' +
  7.                         '  XPathResult.FIRST_ORDERED_NODE_TYPE,' +
  8.                         '  null' +
  9.                         ').singleNodeValue;' +
  10.                         '}' +
  11.                         'var element = getElementByXpath(xpath);' +
  12.                         'if (!element) {' +
  13.                         'throw new Error("Error: cannot find an element with XPath(" + xpath + ")");' +
  14.                         '}' +
  15.                         'element.scrollIntoView(true);' ) ;
  16.  

Is it possible to force the entire translation or simulate the scroll action?
Lazarus trunk / fpc 3.2.2 / Kubuntu 22.04 - 64 bit

Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #36 on: September 02, 2024, 07:18:36 am »
Thanks @kapibara for the script  :)! Unfortunately it doesn't solve the translation problem as I hoped. It continues to translate one piece at the beginning and one at the end in the same way. I also tried adding a sleep(3000) to give the Browser time before saving but it doesn't solve the problem..

kapibara

  • Hero Member
  • *****
  • Posts: 629
Re: Browser Automation with selenium
« Reply #37 on: September 02, 2024, 09:33:50 am »
I don't know. Sometimes AI can help so I asked: Is it possible to have Googles translation extension for firefox to translate a webpage completely even if i don't scroll to the end of a page?

ChatGPT: (https://talkai.info/chat/)

Quote
As of my last knowledge update in October 2023, Google does not offer a dedicated translation extension for Firefox that translates entire web pages automatically without scrolling. However, there are other ways to achieve webpage translation in Firefox:

    Firefox Translation Feature: Firefox has a built-in translation feature that can translate entire web pages. You can access it by right-clicking on the page and selecting "Translate Page."

    Third-party Extensions: There are various translation extensions available for Firefox, such as "To Google Translate" or "Translate web pages." These extensions can help you translate pages, but the functionality may vary, and the translation might still require you to load the entire content.

    Using Google Translate: You can manually copy the URL of the webpage you want to translate, go to Google Translate (translate.google.com), and paste the URL in the translation box. Google Translate will then open the page translated into your desired language.

Always check the latest extensions in the Firefox Add-ons store, as new features and tools may have been introduced after my last update.

Is it possible to start a webdriver with extensions loaded?

ChatGPT

Yes, you can start a WebDriver session with extensions loaded. This capability can be particularly useful when you want to automate tasks that involve browser extensions, such as translation tools.

Here's how you can do it with popular web drivers like ChromeDriver and Firefox's geckodriver.
Using ChromeDriver

To start Chrome with an extension, you can specify the path to the extension directory when initializing the WebDriver. Here’s an example in Python using Selenium:

    Install the required libraries:

     
    pip install selenium

    Download the ChromeDriver if you haven't done so already, and make sure it matches your version of Chrome.

    Load your desired extension: Assuming you have a folder with the unpacked extension (let's say it's called my_extension), run the following code:

     
    from selenium import webdriver
    from selenium.webdriver.chrome.service import Service
    from selenium.webdriver.chrome.options import Options

    # Path to the ChromeDriver
    chrome_driver_path = '/path/to/chromedriver'

    # Path to the unpacked extension
    extension_path = '/path/to/my_extension'

    chrome_options = Options()
    chrome_options.add_argument(f'--load-extension={extension_path}')

    service = Service(chrome_driver_path)
    driver = webdriver.Chrome(service=service, options=chrome_options)

    # Now you can navigate to a page and the extension should be loaded.
    driver.get('https://example.com')

    # Perform your actions...

    driver.quit()

Using GeckoDriver (Firefox)

Similarly, you can start a Firefox WebDriver session with extensions loaded. Here's how to do it:

    Install the required libraries (if not already done):

     
    pip install selenium

    Download the GeckoDriver if you haven't done so.

    Load your desired extension: Assuming you have a .xpi extension file, run the following code:

     
    from selenium import webdriver
    from selenium.webdriver.firefox.service import Service
    from selenium.webdriver.firefox.options import Options

    # Path to the GeckoDriver
    gecko_driver_path = '/path/to/geckodriver'

    firefox_options = Options()
    firefox_options.add_extension('/path/to/my_extension.xpi')

    service = Service(gecko_driver_path)
    driver = webdriver.Firefox(service=service, options=firefox_options)

    # Now you can navigate to a page and the extension should be loaded.
    driver.get('https://example.com')

    # Perform your actions...

    driver.quit()

Notes

    Make sure that the extension is either unpacked (for Chrome) or in the proper format (e.g., .xpi for Firefox).
    Some extensions may require additional permissions or specific configurations to work correctly when loaded via WebDriver.
    Be aware that certain behaviors of extensions might interfere with automated testing, so always test thoroughly to ensure your scripts run as expected.

Using these methods, you should be able to start a WebDriver session with your desired extensions loaded and use them for automation tasks, including translation functionalities!
Lazarus trunk / fpc 3.2.2 / Kubuntu 22.04 - 64 bit

Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #38 on: September 02, 2024, 07:44:14 pm »
Thanks  :), I confirm that if you use firefox and its translation function (beta) it actually translates the entire page. While passing a link to google translator this does not happen. To test just save without scrolling the page and open the files offline. I noticed that on some opened pages the possibility of using this function does not appear because the relative icon does not appear (but this is another story, maybe calculate how much text there is..)
So I'm happy that the function exists, the only problem remains of how to make it run.

Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #39 on: September 02, 2024, 09:05:19 pm »
Trying to find some useful points I did some research (but unfortunately I don't know much about this topic  :():

Using the extension argument, I think it only works with the old add-on. Now the feature is integrated.

https://support.mozilla.org/en-US/kb/firefox-translations-add-on?redirectslug=firefox-translations&redirectlocale=en-US

https://addons.mozilla.org/it/firefox/addon/firefox-translations

https://github.com/mozilla/firefox-translations

Maybe scrolling should be triggered via "Element SendKeys" by sending the arrow key?

https://developer.mozilla.org/en-US/docs/Web/WebDriver/Commands

The most interesting link I found is about a script but no idea if it can be converted or its actual usefulness..  :-\

https://stackoverflow.com/questions/13030153/translate-website-to-any-specific-language-on-page-load

Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #40 on: September 03, 2024, 10:53:58 pm »
It seems to work with one modification
Code: Pascal  [Select][+][-]
  1. element.scrollIntoView({ behavior: "smooth", block: "start" });
  2.  
  :)

but maybe someone will find a more "automated" method than scrolling..

kapibara

  • Hero Member
  • *****
  • Posts: 629
Re: Browser Automation with selenium
« Reply #41 on: September 04, 2024, 02:20:39 am »
I'm not so good at this, but the problem could be how the scrolling is done. The extension translates page by page when you scroll manually one page at a time. If we simulate this by using "continous scrolling" it might work.

Once again, this is code generated by AI and untested and you'll need to try to make it work. If possible. Please do tell if you succeed.

Code: Pascal  [Select][+][-]
  1. var
  2.   Driver: TWebDriver;
  3.   ScrollScript: String;
  4.   ScrollHeight, NewScrollHeight: Integer;
  5.   I: Integer;
  6.  
  7. begin
  8.   // Initialize the WebDriver
  9.   Driver := TFirefoxDriver.Create;
  10.  
  11.   try
  12.     // Open the desired URL
  13.     Driver.Get('http://example.com');  // Change to your target webpage
  14.  
  15.     // Allow the page to load
  16.     Sleep(3000); // Wait for 3 seconds – adjust as necessary
  17.  
  18.     // Continuous scrolling
  19.     for I := 1 to 10 do // Change the number of iterations as needed
  20.     begin
  21.       // Execute JavaScript to scroll down
  22.       ScrollScript := 'window.scrollTo(0, document.body.scrollHeight);';
  23.       Driver.ExecuteScript(ScrollScript);
  24.      
  25.       // Wait for content to load
  26.       Sleep(1000); // Wait for 1 second – adjust as necessary
  27.  
  28.       // Get the new scroll height
  29.       NewScrollHeight := Driver.ExecuteScript('return document.body.scrollHeight;').AsInteger;
  30.      
  31.       // Break if the scroll height doesn't change, indicating the bottom has been reached
  32.       if NewScrollHeight <= ScrollHeight then
  33.         Break;
  34.  
  35.       ScrollHeight := NewScrollHeight; // Update the last scroll height
  36.     end;
  37.  
  38.   finally
  39.     // Clean up
  40.     Driver.Quit;
  41.   end;
  42. end.
Lazarus trunk / fpc 3.2.2 / Kubuntu 22.04 - 64 bit

Phoenix

  • Full Member
  • ***
  • Posts: 109
Re: Browser Automation with selenium
« Reply #42 on: September 04, 2024, 08:47:35 pm »
Thank you, @kapibara try this  ;):

Code: Pascal  [Select][+][-]
  1. var
  2.  Driver: TWebDriver;
  3.  NewScrollHeight,tmp: Integer;
  4. begin
  5.  Driver:= TFirefoxDriver.Create(nil);
  6.  try
  7.   Driver.StartDriver(ExtractFileDir(ParamStr(0)) + '\geckodriver.exe');
  8.   Driver.NewSession;
  9.   Driver.GetURL(<input google link>);
  10.  
  11.   Sleep(1000);
  12.  
  13.   NewScrollHeight:= 0;
  14.   while True do
  15.   begin
  16.    Driver.ExecuteScript('window.scrollTo(0, window.scrollY+window.outerHeight);');
  17.    Sleep(500);
  18.    tmp:= NewScrollHeight;
  19.    NewScrollHeight:= StrToInt(Driver.ExecuteScript('return window.scrollY;'));
  20.    if tmp = NewScrollHeight then
  21.     Break;
  22.   end;
  23.  
  24.   Sleep(1000);
  25.  
  26.   Driver.SaveCurDocToFile(<save path>);
  27.  finally
  28.   Driver.Quit;
  29.  end;
  30.  

EDIT NOTE:

doing more tests, a modification is needed
Code: Pascal  [Select][+][-]
  1. NewScrollHeight:= Round(StrToFloat(Robot.ExecuteScript('return window.scrollY;')));
  2.  

in my case this was also needed
Code: Pascal  [Select][+][-]
  1. FormatSettings.DecimalSeparator:= '.';
  2.  
« Last Edit: September 04, 2024, 09:40:46 pm by Phoenix »

kapibara

  • Hero Member
  • *****
  • Posts: 629
Re: Browser Automation with selenium
« Reply #43 on: September 05, 2024, 01:12:08 am »
Looks like you made it. Well done.  8)
Lazarus trunk / fpc 3.2.2 / Kubuntu 22.04 - 64 bit

gidesa

  • Full Member
  • ***
  • Posts: 145
Re: Browser Automation with selenium
« Reply #44 on: September 05, 2024, 01:03:29 pm »
The EricWang's library is a pure Pascal/Delphi implementation of the W3C web driver protocol. So no Selenium license apply.
Only method/property names could remind, perhaps, Selenium ones.

It may be covered by the Selenium license and those of other libraries it depends on.

 

TinyPortal © 2005-2018