[Solution found, see below]
I am working with the following Python script (using Python 2.7) to open Microsoft Edge and browse to www.freelancer.in (using Selenium 3.8.1):
import os
from selenium import webdriver
# create new Edge session
dir = os.path.dirname(__file__)
edge_path = dir + "\MicrosoftWebDriver.exe"
driver = webdriver.Edge(edge_path)
driver.implicitly_wait(10)
driver.get("https://www.freelancer.in/")
It works properly on my local machine: Windows Pro Version 1709, OS 16299.125. However, it does not work on my virtual machine... I can't figure out why because I have the exact same Windows 10 Pro installed, I am using the same Microsoft Webdriver.exe (16299.15). The Microsoft WebDriver.exe seems to be working since it says:
[15:32:45.548] - Listening on http://localhost:17556/
But after, I receive the following error:
Traceback (most recent call last):
File "C:\Users\program.py", line 9, in <module>
driver = webdriver.Edge(edge_path)
File "C:\Python27\lib\site-packages\selenium\webdriver\edge\webdriver.py", line 43, in __init__
desired_capabilities=capabilities)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 208, in check_response
raise exception_class(value)
WebDriverException: Message: Unknown error
I didn't find any specific config to do in Microsoft Edge.
Any ideas of what could be responsible for this error?
=> Solution found there : selenium.common.exceptions.WebDriverException: Message: Unknown error while trying to use Edge and MicrosoftWebDriver.exe through Selenium
It was not working on the virtual machine because the User Account Control settings were Turned off... Turned on UAC resolved the issue.
=> Solution found there : selenium.common.exceptions.WebDriverException: Message: Unknown error while trying to use Edge and MicrosoftWebDriver.exe through Selenium It was not working on the virtual machine because the User Account Control settings were Turned off... Turned on UAC resolved the issue.
The error says it all :
selenium.common.exceptions.WebDriverException: Message: Unknown error
It's pretty clear that the webdriver instance is not getting invoked. So you have to pass the edge_path along with the argument executable_path as follows :
driver = webdriver.Chrome(executable_path=edge_path)
I got identical issue with Edge. There should not be any specific configuration required for invoking Edge browser. Following code should be enough for opening it:
from selenium.webdriver import Edge
driver = Edge()
This works ok for me on laptop as in your case - but not on virtual machine with Win10... so I guess we got a possible pattern here.
You wrote you tried it with Microsoft Webdriver.exe 16299.15. You can try also newer version 17134 from microsoft. It did not work for me but could for you.
Also there is supposed to be possible to get Microsoft Webdriver.exe directly from your Win10 installation: Settings → Apps → Manage optional features → Add a feature → Microsoft WebDriver. This should install Microsoft Webdriver directly to your machine plus add it to PATH.
Btw... edge_path is not needed to pass if you got MicrosoftWebDriver.exe set in PATH.
Related
I've upgraded Ubuntu to 20.04. It seems that Chrome isn't available as APT package but via snap. After the upgrade I'm getting the error while trying to instantiate Chrome browser:
>>> from selenium.webdriver import Chrome
>>> from selenium.webdriver.chrome.options import Options
>>> o = Options()
>>> o.headless = True
>>> o.add_argument('--disable-dev-shm-usage')
>>> o.add_argument('--no-sandbox')
>>> Chrome(options=o)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/var/www/order/lib/python3.8/site-packages/selenium/webdriver/chrome/webdriver.py", line 76, in __init__
RemoteWebDriver.__init__(
File "/var/www/order/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "/var/www/order/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/var/www/order/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/var/www/order/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: DevToolsActivePort file doesn't exist
chromedriver version is 87.0.4280.20
chromium-browser version is Chromium 87.0.4280.66 snap
I read this discussion. The case is however isn't same. I run python as a regular user. However I've disabled dev-shm-usage and sandbox. But still it doesn't work. It worked before I've upgraded to Ubuntu 20.04. So I assume it has something to do with snap version of Chrome.
I have found out following configuration works:
Start first Chrome chromium-browser --headless --remote-debugging-port=9222 and then:
>>> from selenium.webdriver import Chrome
>>> from selenium.webdriver.chrome.options import Options
>>> o = Options()
>>> o.add_experimental_option('debuggerAddress', 'localhost:9222')
>>> b = Chrome(options=o)
>>> b.get('https://google.com')
>>> b.title
'Google'
>>>
So it seems the problem is with starting the browser.
Opened a bug report
Good explanation was provided by ChromeDriver development team in a response to the ticket I've submitted:
ChromeDriver uses the /tmp directory to communicate with Chromium, but
Snap remaps /tmp directory to a different location (specifically, to
/tmp/snap.chomium/tmp). This causes errors because ChromeDriver can't
find files created by Chromium. ChromeDriver is designed and tested
with Google Chrome, and it may have compatibility issues with
third-party distributions.
There are a couple of workarounds:
Tell ChromeDriver to use a location in your home directory, instead of /tmp. For example, if your home directory is /home/me, you can add
the following line of code to your script:
o.add_argument('--user-data-dir=/home/me/foo')
Explicitly select a port for ChromeDriver to communiate with Chromium. You need to carefully select a port (e.g., 9222) that isn't
already in use, and then add the following line to your script:
o.add_argument('--remote-debugging-port=9222')
and
Another (probably better) workaround is to use the ChromeDriver
installed by Snap, which is at /snap/bin/chromium.chromedriver. E.g.,
driver = Chrome('/snap/bin/chromium.chromedriver', options=o)
Since this version of ChromeDriver runs inside Snap, its /tmp
directory is redirected in the same way as Chromium
The solution to my problem is rather workaround but nevertheless it might be useful.
I have downgraded to Chrome version 86, which I have installed from deb package instead of snap. Installation instructions can be found here
Once I've downgraded chromium and chromedriver to version 86 the problem has gone.
The answer provided by Ralfeus is wholesome, I am using Ubuntu 20.04 and was facing same issues.
I used driver = Chrome('/snap/bin/chromium.chromedriver') , yeah without any additional options and it worked.
I'm trying to run behave tests on Windows Edge in CircleCI, but even simple test fails initializing a session with
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\circleci\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\edge\webdriver.py", line 66, in __init__
desired_capabilities=capabilities)
File "C:\Users\circleci\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py", line 157, in __init__
self.start_session(capabilities, browser_profile)
File "C:\Users\circleci\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Users\circleci\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\circleci\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\errorhandler.py", line 208, in check_response
raise exception_class(value)
selenium.common.exceptions.WebDriverException: Message: Unknown error
Setup
I use standard CircleCI windows image windows-server-2019-vs2019:stable
Install there Edge driver and try to start it as a service:
choco install -y jdk10 selenium-edge-driver selenium
Then I ssh to the machine for debugging and try
python -c "from selenium import webdriver; webdriver.Edge()"
This cause the error above. The path to the web-driver is correct, an incorrect causes another error.
Already tried
Turning off UAC (using cmd.exe):
reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v EnableLUA /t REG_DWORD /d 0 /f
reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v PromptOnSecureDesktop /t REG_DWORD /d 0 /f
reg.exe ADD HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System /v ConsentPromptBehaviorAdmin /t REG_DWORD /d 0 /f
Trying to connect using Chrome settings:
Stating the server by myself:
/c/tools/selenium/MicrosoftWebDriver.exe
[14:48:13.545] - Listening on http://localhost:17556/
python -c "from selenium.webdriver.remote.webdriver import WebDriver as RemoteWebDriver; from selenium.webdriver.common.desired_capabilities import DesiredCapabilities; from selenium.webdriver.chrome.remote_connection import ChromeRemoteConnection; rwd = RemoteWebDriver(command_executor=ChromeRemoteConnection(remote_server_addr='http://localhost:17556', keep_alive=False))"
This results with a 401 error bad request or if I update desired_capabilities something like:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: No matching capabilities found
Tried to change the Edge binary to the latest in https://developer.microsoft.com/en-us/microsoft-edge/tools/webdriver/ but still have the same issue.
Currently, there is no way to install Edge on the CircleCI image. But if services like BrowserStack doesn't suit you, there is a way to run Selenium with Windows/Edge on your own/AWS machine deploying an official Vagrant image.
Download official VM with Edge installed (Vagrant will be fine)
If you plan to use AWS check vagrant-aws to deploy the image on EC2
Install Selenium on the Vagrant image. You can try to use Puppet for it, check this example, but it uses an outdated Edge version, you need to update the manifest.
Spend some time on setting permissions/roles and other IAM stuff to be able to connect to the EC2 from CircleCI.
Now using webdriver.Remote you can connect to the Selenium (hub or stand alone) with Edge and a web-driver for it installed.
My scripts don't work anymore and I can't figure it out.
It is a chrome version problem apparently... But I don't know how to switch to another version (not the latest?) Does exist another way?
My terminal indicates :
Traceback (most recent call last):
File "/Users/.../Documents/SCRIPTS/PYTHON/Scripts/# -- coding: utf-8 --.py", line 21, in
driver = webdriver.Chrome()
File "/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in init
desired_capabilities=desired_capabilities)
File "/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in init
self.start_session(capabilities, browser_profile)
File "/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/opt/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: Chrome version must be between 71 and 75
(Driver info: chromedriver=2.46.628411 (3324f4c8be9ff2f70a05a30ebc72ffb013e1a71e),platform=Mac OS X 10.14.5 x86_64)
Any idea?
If you're running your test with ChromeDriver & Chrome browser, from the log given, i can see the ChromeDriver was not compatible with your current Chrome Browser. Because the ChromeDriver version 2.46 aims to support Chrome browser from v71-v73. (You can refer to the ChromeDriver change log here http://chromedriver.chromium.org/downloads ). Since v75, there's a major and notable change is ChromeDriver now runs in W3C standard compliant mode by default.
The easiest way is to download the newer ChromeDriver which is compatible with your current Chrome Browser rather than sticking with the old , non supported ChromeDriver version.
This possibly happens, as your Chrome Browser or Chromium may be updated to newer versions automatically. But you still run your selenium scripts using the old version of the chromedriver.
Check the current version of your Google chrome or Chromium, then download the chromedriver for that specific version.
Then your scripts may work fine!
I'm using Debian 9 Stretch and Pycharm IDE and trying to learn web-scramping; I installed the Selenium package simply by running:
pip install selenium
and the Firefox webdriver by running:
wget https://github.com/mozilla/geckodriver/releases/download/v0.19.1/geckodriver-v0.19.1-linux64.tar.gz
tar -xvzf geckodriver-v0.19.1-linux64.tar.gz.1
chmod +x geckodriver
respectively, to download the last release, extract it and make the driver executable. After that, I added the driver to the following path:
usr/local/bin
I ran all by using the Pycharm IDE terminal and not the built-in Debian terminal.
In order to open Firefox and web-scrape, I run:
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
webdriver.Firefox(executable_path="/usr/local/bin/geckodriver")
The last line gives an error message as output:
Traceback (most recent call last):
File "<input>", line 1, in <module>
File "/root/PycharmProjects/Example/venv/lib/python3.5/site-packages/selenium/webdriver/firefox/webdriver.py", line 162, in __init__
keep_alive=True)
File "/root/PycharmProjects/Example/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 154, in __init__
self.start_session(desired_capabilities, browser_profile)
File "/root/PycharmProjects/Example/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 243, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "/root/PycharmProjects/Example/venv/lib/python3.5/site-packages/selenium/webdriver/remote/webdriver.py", line 312, in execute
self.error_handler.check_response(response)
File "/root/PycharmProjects/Example/venv/lib/python3.5/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: connection refused
I'm a newbie both in Python and in web-scraping; please, could someone explain what does go wrong with installation and coding and why I got this error?
In the hope to be clear asking the question, I thanks all in advance for the help.
Assuming that the location of the geckodriver is correct, you can check below:
properties of the geckodriver should have the correct permission for the user. You would need to check the box "Allow this file to run as a program" or
if you you have restricted access, save the geckodriver in your home/username/geckodriver then path it to your firefox. Saving it in your home folder will be able to modify the properties of your geckodriver.
[EDIT] Are your running in command line? If so, you need a virtual display, I have used pyvirtualdisplay:
from pyvirtualdisplay import Display
display = Display(visible=0,size(800, 600))
display.start()
driver = webdriver.Firefox(executable_path="/usr/local/bin/geckodriver")
The problem has been solved by removing Mozilla Firefox, that, in Debian 9 Stretch is installed as ESR (Extended Release Support) by default; at the time the Firefox ESR version was 52.0.
After, I installed the by instaling the unstable Firefox version (not Beta) by running on the the terminal as super-user:
su -
gedit /etc/apt/sources.list
and adding deb http://ftp.it.debian.org/debian/ unstable main to the sources list file.
After, I ran:
apt-get update
apt-get install -t unstable firefox
to update the software and install Firefox.
By following the guidelines explained in the question to install and run the selenium Python package everything should work fine (at least, for me!).
Hope this will help other users too!
While woking with Selenium-Python Client v3.10.0 along with GeckoDriver v0.19.1 and Firefox v58.0.2 you have to initialize the WebDriver instance and assign it to a variable, which will in-turn initialize the Web Browser which will in-turn open the desired URL as follows :
from selenium import webdriver
driver = webdriver.Firefox(executable_path='/usr/local/bin/geckodriver')
driver.get('https://www.google.co.in')
print("Page Title is : %s" %driver.title)
driver.quit()
I have the following script
from selenium import webdriver
browser = webdriver.Firefox()
browser.get('http://localhost:8000')
assert 'Django' in browser.title
I get the following error
$ python3 functional_tests.py
Traceback (most recent call last): File "functional_tests.py", line 3, in <module>
browser = webdriver.Firefox() File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/webdriver.py", line 80, in __init__
self.binary, timeout) File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/extension_connection.py", line 52, in __init__
self.binary.launch_browser(self.profile, timeout=timeout) File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 68, in launch_browser
self._wait_until_connectable(timeout=timeout) File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/firefox/firefox_binary.py", line 99, in _wait_until_connectable
"The browser appears to have exited " selenium.common.exceptions.WebDriverException: Message: The browser appears to have exited before we could connect. If you specified a log_file in the FirefoxBinary constructor, check it for details.
pip3 list shows selenium (2.53.6).
firefox -v shows Mozilla Firefox 47.0.
I struggled with this problem as well, and I was unhappy with having to use older versions of Firefox. Here's my solution that uses the latest version of Firefox. It however involves several steps
Step 1. Download v0.9.0 Marionette, the next generation of FirefoxDriver, from this location: https://github.com/mozilla/geckodriver/releases/download/v0.9.0/geckodriver-v0.9.0-linux64.tar.gz
Step 2. Extract the file to a desired folder, and rename it to "wires". In my case I created a folder named "add_to_system_path" under Documents. So the file is in Documents/add_to_system_path/wires (also make sure that the wires file is executable under its properties)
Step 3. Create a file named ".pam_environment" under your home folder, and then adding this line on it and save
PATH DEFAULT=${PATH}:/absolute/path/to/the/folder/where/wires/is/saved
What this does is it tells ubuntu to add the enumerated dir in .pam_environment to your system path
Step 4. Save the file, log out of your user session, and log back in. This is necessary to do so that the files in the newly added system path is recognized by ubuntu
Step 5. Use the code below to instantiate the browser instance:
`
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
capabilities = DesiredCapabilities.FIREFOX
capabilities["marionette"] = True
browser = webdriver.Firefox(capabilities=capabilities)
browser.get('http://your-target-url')`
Firefox should now be able to instantiate as usual.
The last version of Firefox is not working properly with selenium. Try with 46 or 45.
You can download here: ftp.mozilla.org/pub/firefox/releases
or sudo apt-get install firefox=45.0.2+build1-0ubuntu1
You can also do this graphically as shown here http://www.howtogeek.com/117929/how-to-downgrade-packages-on-ubuntu/