I have up and running an Apache Server with Python 3.x installed already on it. Right now I am trying to run ON the server a little python program (let's say filename.py). But this python program uses the webdriver for Chrome from Selenium. Also it uses sleep from time (but I think this comes by default, so I figure it won't be a problem)
from selenium import webdriver
When I code this program for the first time on my computer, not only I had to write the line of code above but also to manually download the webdriver for Chrome and paste it on /usr/local/bin. Here is the link to the file in case you wonder: Webdriver for Chorme
Anyway, I do not know what the equivalences are to configure this on my server. Do you have any idea how to do it? Or any concepts I could learn related to installing packages on an Apache Server?
Simple solution:
You don't need to install the driver in usr/local/bin. You can have the .exe anywhere and you can specify that with an executable path, see here for an example.
Solution for running on a server
If you have python installed on the server, ideally >3.4 which comes with pip as default. Then install ChromeDriver on a standalone server, follow the instructions here
Note that, Selenium always need an instance of a browser to control.
Luckily, there are browsers out there that aren't that heavy as the usual browsers you know. You don't have to open IE / Firefox / Chrome / Opera. You can use HtmlUnitDriver which controls HTMLUnit - a headless Java browser that does not have any UI. Or a PhantomJsDriver which drives PhantomJS - another headless browser running on WebKit.
Those headless browsers are much less memory-heavy, usually are faster (since they don't have to render anything), they don't require a graphical interface to be available for the computer they run at and are therefore easily usable server-side.
Sample code of headless setup
op = webdriver.ChromeOptions()
op.add_argument('headless')
driver = webdriver.Chrome(options=op)
It's also worth reading on running Selenium RC, see here on that.
Related
How to use normal chrome completely without chromedriver selenium python not duplicate.
I am using python 3.8.8,os is windows 7 ultimate with pycharm as
IDE and chrome version is around 96. and my problem is that whenever I use my python script to scrape a website it uses chromedriver and when I specify what's given below:
options = Options ()
options.add_argument(r"user-data-dir=my chrome path which is not Executable instead the user data")
#this works but when opening chrome it shows "browser is controlled by automated software" and changing it to normal chrome. Exe won't work
Sure it uses normal chrome with my credentials but it still needs chromedriver to work and when I delete the chromedriver it throw an error and when I go into selenium source code in a file called site.py(or sites.py) which I changed the variable self. executable to chrome.exe path and it worked and it won't show the message browser is controlled by automated software but it won't do anything , it is just stuck there and what I want to do is use chrome as the browser to scrape without chromedriver in my pc is it possible? If yes please tell me how should I go on to do it and you can ask for further Clarification and details and Thanks in advance
By default, selenium is detected as an automated software and is flagged by most websites, and the flag is unable to be removed. There are, however, external libraries that can be installed that can remove the flag.
There are options here to try to get around the default flag and hide the fact the browser is automated.
Edit
I understand the question further, and see that you want a more portable chrome option. Chrome driver is a very specific program controlled by selenium and must be used. There is no substitute. You can use Firefox driver or internet explorer, but a webdriver must be used (hence the name driver for driving the main browser). When you specify the directory for the Chrome binary, you aren’t removing the middleman of the chromedriver, only Specifying where chrome driver needs to look!
Using Selenium you won't be able to initiate/spawn a new Browsing Context i.e. Chrome Browser session without the ChromeDriver.
The Parts and Pieces
As a minimum requirement, the WebDriver i.e the ChromeDriver talks to a browser through a driver and the communication is two way:
WebDriver passes commands to the browser through the driver
Receives information back via the same route.
Hence using ChromeDriver is a mandatory requirement.
Does anyone know how to force pyautogui or python in general to recognize a set screen size and take in display data, even if no display is connected?
I have a task that uses pyautogui, and selenium chrome driver, which both require a display, or they fail.
It runs on a server, so the start of the program requires my laptop to remote desktop into the server, allowing it to have a display, which allows launching a page with chromedriver, and pyautogui click components / screen search to work.
The issue arises that should my home network ever be down, it cannot kick off the remote desktop, and therefore my automation would fail.
My solution would be to emulate or force the program to behave as if a display existed, so it can just be run server side.
All of my servers are windows, so XVFB does not seem to be an option based on
Xvfb on Windows
Well I am using something similar on regular basis,
I am using Windows server to run my automated python script which uses selenium webdriver.
so, first your answer is you need to use code for screen size in the script and you can run that script with Windows task scheduler so, you don't have to touch your laptop or desktop to run remote desktop.
you can use chrome options if you are using chrome as headless browser.
I also, suggest if you are using server which is controlled by some third party who provides regular updates then, you can use chromedriver_autoinstaller package to take updated or supported version chrome driver according to your current version of chrome.
code:
import chromedriver_autoinstaller
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from time import sleep
from selenium.webdriver.chrome.options import Options
chromedriver_autoinstaller.install()
#maximize the chrome
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--window-size=1920x1080")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('website address')
For automating task Windows task scheduler is the best option. you can refer this documentation or find something according to your need.
https://towardsdatascience.com/automate-your-python-scripts-with-task-scheduler-661d0a40b279
Note: If you just need to set screen size then your answer start here #maximize the chrome
You should emulate your display driver. Run:
xvfb-run --server-args="-screen 0 1024x768x24" python my_script.py
to launch your script instead of just python my_script.py.
I know the question has been a little while...
I just found out this using PyAutoGUI on remote machines or headless machines
at here https://github.com/asweigart/pyautogui/issues/133
Can look at this guys workaround: http://fredtantini.free.fr/blog/index.php?article58/automatiser-des-actions-avec-selenium-pyautogui-et-xvfb
Thank you
I am experiencing a very strange behaviour when testing Chrome via selenium webdriver.
Instead of navigating to pages like one would expect the 'get' command leads only to the download of tiny files (no type or.apsx files) from the target site.
Importantly - this behavior only occurs when I pass chrome_options as an argument
to the Chrome driver.
The same testing scripts work flawless with firefox driver.
Code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options # tried with and without
proxy = '127.0.0.1:9951' # connects to a proxy application
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % proxy)
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('whatismyip.com')
Leads to the automatic download of a file called download (no file extension, Size 2 Byte).
While calling other sites results in the download of small aspx files.
This all happens while the browser page remains blank and no interaction with
elements happen = the site is not loaded at all.
No error message, except element not found is thrown.
This is really strange.
Additional info:
I run Debian Wheezy 32 bit and use Python 2.7.
Any suggestions how to solve this issue?
I tried your code and captured the traffic on localhost using an SOCKS v5 proxy through SSH. It is definitely sending data through the proxy but no data is coming back. I have confirmed the proxy was working using Firefox.
I'm running Google Chrome on Ubuntu 14.04 LTS 64-bit. My Chrome browser gives me the following message when I try to configure a proxy in its settings menu:
When running Google Chrome under a supported desktop environment, the
system proxy settings will be used. However, either your system is not
supported or there was a problem launching your system configuration.
But you can still configure via the command line. Please see man
google-chrome-stable for more information on flags and environment
variables.
Unfortunately I don't have a man page for google-chrome-stable.
I also discovered that according to the selenium documentation Chrome is using the system wide proxy settings and according to their documentation it is unknown how to set the proxy in Chrome programmatically: http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp#using-a-proxy
I am using selenium-2.35.0 and Python-2.7.
Testcases are written in python.
my python code to create driver object:
from selenium import webdriver
driver = webdriver.Remote(desired_capabilities={
"browserName": "firefox"
})
And run selenium server by,
java - jar selenium-server-standalone-2.35.0.jar
I had my code working in Firefox - 22 - had the selenium server running, able to run scripts in python, etc. So I'm confident the code works.
Recently, I updated FireFox to 23 and now all I get is
"[Errno 10061] No connection could be made because the target machine actively refused it."
I thought maybe I need to restart the server again, or something. But that seems to do nothing. Is this issue related to selenium webdriver's support for the latest browser version?
But as of this link http://selenium.googlecode.com/git/java/CHANGELOG , selenium supports Firefox - 23. If supported, code that run in Firefox - 22 should also run in Firefox - 23 without any code change.
And how can i make the same code work for chrome?
I have found that the newest version of firefox routinely doens't work immediately well with Selenium. Check out this firefox support matrix on Github that someone made. Unfortunately the only thing you can do is stop Firefox from auto-updating and keep your selenium tests running for firefox newest version minus 1 or 2. Chrome tends to work out of the box for Selenium, sometimes the Beta channel has fixed some selenium issues, so try that if you have a particular issue (on the other hand it may introduce other bugs). So in the end you need to be constantly weary of browser updates and routinely checking how they are working with the current version of selenium.
Check out this guide on how to get Selenium working with rolled back versions of firefox:
http://inkhorn.ca/selenium-python-on-ubuntu-using-firefox/
It will also fix any errors that have to do with “version xul**.0 not defined in file libxul.so”
I'm trying to use selenium from python but I'm having a problem running it on a RHEL5.5 server. I don't seem to be able to really start firefox.
from selenium import webdriver
b = webdriver.Firefox()
On my laptop with ubuntu this works fine and it starts a brings up a firefox window. When I log in to the server with ssh I can run firefox from the command line and get it displayed on my laptop. It is clearly firefox from the server since it has the RHEL5.5 home page.
When I run the python script above on the server it (or run it in ipython) the script hangs at webdriver.Firefox()
I have also tried
from selenium import webdriver
fb = webdriver.FirefoxProfile()
fb.native_events_enabled=True
b=webdriver.Firefox(fb)
Which also hangs on the final line there.
I'm using python2.7 installed in /opt/python2.7. In installed selenium with /opt/python2.7/pip-2.7.
I can see the firefox process on the server with top and it is using a lot of CPU. I can also see from /proc/#/environ that the DISPLAY is set to localhost:10.0 which seems right.
How can I get a browser started with selenium on RHEL5.5? How can I figure out why Firefox is not starting?
It looks like the problem I'm encountering is this selenium bug:
http://code.google.com/p/selenium/issues/detail?id=2852
I used the fix described in comment #9 http://code.google.com/p/selenium/issues/detail?id=2852#c9
That worked for me.