Run Selenium Python Script on Remote Server - python

I've built a web scraper using Selenium that I need to run when my local machine is sleeping and not connected to a network. I have a remote server that I can use to run this script, but I'm having a hard job conceptualizing how this will work. Can I use Selenium's remote driver to do this? I have it installed on my local, do I need to install anything on my server?
Here's an example of the start of my script which runs great on my local:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
from twilio.rest import TwilioRestClient
import sys
# driver = webdriver.Chrome()
driver = webdriver.Remote(
command_executor='http://127.0.0.1:4444/wd/hub',
desired_capabilities=DesiredCapabilities.CHROME)
driver.get("http://www.southwest.com")

Does your code resides on remote server? Do you want to execute tests on remote server or local machine? Where is your node?
keep your code on remote server -> make it(remote) hub -> register node of your choce where you want to execute test cases -> create a jenkins job for this which will enable to execute on demand.

Related

Can you set a screen size and use pyautogui / selenium chrome driver without a monitor

Does anyone know how to force pyautogui or python in general to recognize a set screen size and take in display data, even if no display is connected?
I have a task that uses pyautogui, and selenium chrome driver, which both require a display, or they fail.
It runs on a server, so the start of the program requires my laptop to remote desktop into the server, allowing it to have a display, which allows launching a page with chromedriver, and pyautogui click components / screen search to work.
The issue arises that should my home network ever be down, it cannot kick off the remote desktop, and therefore my automation would fail.
My solution would be to emulate or force the program to behave as if a display existed, so it can just be run server side.
All of my servers are windows, so XVFB does not seem to be an option based on
Xvfb on Windows
Well I am using something similar on regular basis,
I am using Windows server to run my automated python script which uses selenium webdriver.
so, first your answer is you need to use code for screen size in the script and you can run that script with Windows task scheduler so, you don't have to touch your laptop or desktop to run remote desktop.
you can use chrome options if you are using chrome as headless browser.
I also, suggest if you are using server which is controlled by some third party who provides regular updates then, you can use chromedriver_autoinstaller package to take updated or supported version chrome driver according to your current version of chrome.
code:
import chromedriver_autoinstaller
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
from selenium.webdriver.chrome.options import Options
chromedriver_autoinstaller.install()
#maximize the chrome
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('website address')
For automating task Windows task scheduler is the best option. you can refer this documentation or find something according to your need.
https://towardsdatascience.com/automate-your-python-scripts-with-task-scheduler-661d0a40b279
Note: If you just need to set screen size then your answer start here #maximize the chrome
You should emulate your display driver. Run:
xvfb-run --server-args="-screen 0 1024x768x24" python my_script.py
to launch your script instead of just python my_script.py.
I know the question has been a little while...
I just found out this using PyAutoGUI on remote machines or headless machines
at here https://github.com/asweigart/pyautogui/issues/133
Can look at this guys workaround: http://fredtantini.free.fr/blog/index.php?article58/automatiser-des-actions-avec-selenium-pyautogui-et-xvfb
Thank you

Allow connection selenium Standalone server with python scripts

I've write a script in python for a bot that run using selenium, untill now I used the normal selenium webdriver, but now I want to move to Selenium Standalone server.
I have 2 servers on DigitalOcean, I want to use one server as the selenium standalone server and the other for send the requests.
In the Main server I run java -jar ~/selenium/selenium-server-standalone-3.14.0.jar for run the Selenium server, and it works.
In the second I have my scripts but I can't figure out how to allow the connection.
My options for run the webdriver are:
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('user-data-dir=/var/www/users/'+ Setup.db +'/cookies')
chrome_options.add_argument('--headless')
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument('--lang=en')
chrome_options.add_argument('CHROME')
browser = webdriver.Remote(command_executor='http://MAIN_SERVER_IP:4444/wd/hub', desired_capabilities=DesiredCapabilities.CHROME)
But I can't run the webriver, I see that the 2 servers are "talking" beacause when I try to run the script, in the main server I read Only local connections are allowed. so that mean that there are some problem with the firewall or the settings but I don't know what to do.

Connecting to google using python stays blank

I am learning python and I am running into an issue
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('http://www.google.com')
my firefox launches but it does not go on google.com like it is supposed to be but rather stays as a blank page.
I would recommend using chrome instead because your script does work.
so... http://chromedriver.storage.googleapis.com/2.9/chromedriver_win32.zip Download that and extract it to your python27/scripts folder located in the root directory in the C:\\ Drive and then run this script.
from selenium import webdriver
driver = webdriver.Chrome()
driver.get('http://www.facebook.com')
And It should connect you to facebook.com Hope I helped. :)

Can't open a url when using selenium under firefox [duplicate]

from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
import time
# Create a new instance of the Firefox driver
driver = webdriver.Firefox()
# go to the google home page
driver.get("http://www.google.com")
This opens a Firefox window but does not open a url.
I have a proxy server(but the address bar does not show the passed url)
I have two Firefox profiles.
Can 1 or 2 be an issue? if yes, then how can I resolve it?
It is a defect of Selenium.
I have the same problem in Ubuntu 12.04 behind the proxy.
Problem is in incorrect processing proxy exclusions. Default Ubuntu exclusions are located in no_proxy environment variable:
no_proxy=localhost,127.0.0.0/8
But it seems that /8 mask doesn't work for selenium. To workaround the problem it is enough to change no_proxy to the following:
no_proxy=localhost,127.0.0.1
Removing proxy settings before running python script also helps:
http_proxy= python script.py
I was facing exactly the same issue, after browsing for sometime,I came to know that it is basically version compatibility issue between FireFox and selenium. I have got the latest FireFox but my Selenium imported was older which is causing the issue. Issue got resolved after upgrading selenium
pip install -U selenium
OS: windows Python 2.7
I have resolved this issue.
If your jar files are older than the latest version and the browser has updated to latest version, then download:
the latest jar files from the selenium website http://www.seleniumhq.org/download/, and
the latest geckodriver.exe.
#Neeraj
I've resolved this problem, but i'm not sure if you are the same reason.
In general, my problem was caused by some permission issues.
I tried to move my whole project into ~/:
mv xxx/ ~/
and then i change give it the 777 permission:
chmod -R 777 xxx/
I'm not familiar with linux permission so i just do this to make sure i have permission to execute the program.
Even you don't have permission, the selenium program will not prompt you.
So, good luck.
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
WebDriver driver = new FirefoxDriver();
driver.manage().timeouts().implicitlyWait(30,TimeUnit.SECONDS);
driver.get("http://www.google.com");
OR
import org.openqa.selenium.support.ui.ExpectedConditions;
WebDriverWait wait = new WebDriverWait(driver,30);
driver.get("http://www.google.com");
//hplogo is the id of Google logo on google.com
wait.until(ExpectedConditions.presenceOfElementLocated(By.id("hplogo")));
I had the similar problem. All I had to do was delete the existing geckodriver.exe and download the latest release of the same. You can find the latest release here https://github.com/mozilla/geckodriver/releases.
A spent a lot of time on this issue and finally found that selenium 2.44 not working with node version 0.12.
Use node version 0.10.38.
I got the same error when issuing a URL without the protocol (like localhost:4200) instead of a correct one also specifying the protocol (e.g. http://localhost:4200).
Google Chrome works fine without the protocol (it takes http as the default), but Firefox crashes with this error.
I was getting similar problem and Stating string for URL worked for me. :)
package Chrome_Example;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.chrome.ChromeDriver;
public class Launch_Chrome {
public static void main(String[] args) {
System.setProperty("webdriver.chrome.driver", "C:\\Users\\doyes\\Downloads\\chromedriver_win324\\chromedriver.exe");
String URL = "http://www.google.com";
WebDriver driver = new ChromeDriver();
driver.get(URL);
}
}
I had the same problem but with Chrome.
Solved it using the following steps
Install Firefox/Chrome webdriver from Google
Put the webdriver in Chrome's directory.
Here's the code and it worked fine
from selenium import webdriver
class InstaBot(object):
def __init__(self):
self.driver=webdriver.Chrome("C:\Program
Files(x86)\Google\Chrome\Application\chromedriver.exe")# make sure
#it is chrome driver
self.driver.get("https://www.wikipedia.com")
InstaBot()
Check your browser version and do the following.
1. Download the Firefox/Chrome webdriver from Google
2. Put the webdriver in Chrome's directory.
I was having the save issue when trying with Chrome. I finally placed my chromedrivers.exe in the same location as my project. This fixed it for me.
Update your driver based on your browser.
In my case for chrome,
Download latest driver for your chrome from here : https://chromedriver.chromium.org/downloads
Check chrome version from your browser at : chrome://settings/help.
While initialising your driver,
use driver = webdriver.Chrome(executable_path="path/to/downloaded/driver")
Please have a look at this HowTo: http://www.qaautomation.net/?p=373
Have a close look at section "Instantiating WebDriver"
I think you are missing the following code line:
wait = new WebDriverWait(driver, 30);
Put it between
driver = webdriver.Firefox();
and
driver.getUrl("http://www.google.com");
Haven't tested it, because I'm not using Selenium at the moment. I'm familiar with Selenium 1.x.
I was having the save issue. I assume you made sure your java server was running before you started your python script? The java server can be downloaded from selenium's download list.
When I did a netstat to evaluate the open ports, i noticed that the java server wasn't running on the specific "localhost" host:
When I started the server, I found that the port number was 4444 :
$ java -jar selenium-server-standalone-2.35.0.jar
Sep 24, 2013 10:18:57 PM org.openqa.grid.selenium.GridLauncher main
INFO: Launching a standalone server
22:19:03.393 INFO - Java: Apple Inc. 20.51-b01-456
22:19:03.394 INFO - OS: Mac OS X 10.8.5 x86_64
22:19:03.418 INFO - v2.35.0, with Core v2.35.0. Built from revision c916b9d
22:19:03.681 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
22:19:03.683 INFO - Version Jetty/5.1.x
22:19:03.683 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
22:19:03.685 INFO - Started HttpContext[/selenium-server,/selenium-server]
22:19:03.685 INFO - Started HttpContext[/,/]
22:19:03.755 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler#21b64e6a
22:19:03.755 INFO - Started HttpContext[/wd,/wd]
22:19:03.765 INFO - Started SocketListener on 0.0.0.0:4444
I was able to view my listening ports and their port numbers(the -n option) by running the following command in the terminal:
$netstat -an | egrep 'Proto|LISTEN'
This got me the following output
Proto Recv-Q Send-Q Local Address Foreign Address (state)
tcp46 0 0 *.4444 *.* LISTEN
I realized this may be a problem, because selenium's socket utils, found in: webdriver/common/utils.py are trying to connect via "localhost" or 127.0.0.1:
socket_.connect(("localhost", port))
once I changed the "localhost" to '' (empty single quotes to represent all local addresses), it started working. So now, the previous line from utils.py looks like this:
socket_.connect(('', port))
I am using MacOs and Firefox 22. The latest version of Firefox at the time of this post is 24, but I heard there are some security issues with the version that may block some of selenium's functionality (I have not verified this). Regardless, for this reason, I am using the older version of Firefox.
This worked for me (Tested on Ubuntu Desktop 11.04 with Python-2.7):
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.stackoverflow.com")
Since you mentioned you use a proxy, try setting up the firefox driver with a proxy by following the answer given here proxy selenium python firefox
Try the following code
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.firefox.FirefoxDriver;
WebDriver DRIVER = new FirefoxDriver();
DRIVER.get("http://www.google.com");
You need to first declare url as a sting as below:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait # available since 2.4.0
import time
# Create a new instance of the Firefox driver
String URL = "http://www.google.com";
driver = webdriver.Firefox()
# go to the google home page
driver.get(URL);

Selenium Remote Web Driver: How to start the remote server from Client

I have few questions in Selenium WebDriver (Selenium2) :
My 1st question is, do I need a standalone .jar file for RemoteWebDriver?
My 2nd question is, if the standalone .jar file is needed, then how can I start this .jar file/Selenium Server from the WebDriver's Client side (just like in Selenium RC) using Python ?
I do know how to start the selenium server from command line locally. But I was wondering if WebDriver has any improvements from RC regarding starting the Server. I am also aware that there is no need of a Selenium Server (standalone .jar file) if WebDriver is used locally.
Yes standalone jar file need to be in running state, in order to run tests on remote machine using RemoteWebDriver.
To run the server, just use java -jar that probably you know already.
refer: http://code.google.com/p/selenium/wiki/RemoteWebDriver for more details.

Categories