We Keep Coding

Selenium ChromeDriver issue using Webdriver Manager for Python

When running this code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from webdrivermanager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().download_and_install())
driver.get("http://www.python.org")
This results in the following exception at the line where the chromedriver is installed:
TypeError: expected str, bytes or os.PathLike object, not tuple
Note that I am aware that there already exist many threads about this topic but since the webdrivermanager seems to have been updated majorly the previous solutions do not work.
Also a quick side note: I installed webdrivermager via conda instead of pip. but that should not be of concern.
EDIT: Entire stack trace:
Traceback (most recent call last): File "C:\Users\stefa\OneDrive -
Johannes Kepler Universität
Linz\Dokumente\GitHub\briefly\src\crawler\crawler.py", line 19, in
driver = webdriver.Chrome(ChromeDriverManager().download_and_install()) File
"C:\Users\stefa\anaconda3\envs\briefly\lib\site-packages\selenium\webdriver\chrome\webdriver.py",
line 73, in init
self.service.start() File "C:\Users\stefa\anaconda3\envs\briefly\lib\site-packages\selenium\webdriver\common\service.py",
line 72, in start
self.process = subprocess.Popen(cmd, env=self.env, File "C:\Users\stefa\anaconda3\envs\briefly\lib\subprocess.py", line 951,
in init
self._execute_child(args, executable, preexec_fn, close_fds, File "C:\Users\stefa\anaconda3\envs\briefly\lib\subprocess.py", line
1360, in _execute_child
args = list2cmdline(args) File "C:\Users\stefa\anaconda3\envs\briefly\lib\subprocess.py", line 565,
in list2cmdline
for arg in map(os.fsdecode, seq): File "C:\Users\stefa\anaconda3\envs\briefly\lib\os.py", line 822, in
fsdecode
filename = fspath(filename) # Does type-checking of filename. TypeError: expected str, bytes or os.PathLike object, not tuple

Install Webdriver Manager for Python:
pip install webdriver-manager
Import ChromeDriverManager:
from webdriver_manager.chrome import ChromeDriverManager
Use webdriver:
service = ChromeService(executable_path=ChromeDriverManager().install())
driver = webdriver.Chrome(service=service)

There are two issues in your code block as follows:
You need to import ChromeDriverManager from webdriver_manager.chrome
As per Webdriver Manager for Python download_and_install() isn't supported and you have to use install()
So your effective code block will be:
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("http://www.python.org")
On windows-10 system the console output will be:
C:\Users\Admin\Desktop\Python Programs>python webdriver-manager_ChromeDriverManager.py
[WDM] -
[WDM] - ====== WebDriver manager ======
[WDM] - Current google-chrome version is 95.0.4638
[WDM] - Get LATEST driver version for 95.0.4638
[WDM] - There is no [win32] chromedriver for browser 95.0.4638 in cache
[WDM] - Get LATEST driver version for 95.0.4638
[WDM] - Trying to download new driver from https://chromedriver.storage.googleapis.com/95.0.4638.54/chromedriver_win32.zip
[WDM] - Driver has been saved in cache [C:\Users\Admin\.wdm\drivers\chromedriver\win32\95.0.4638.54]
DevTools listening on ws://127.0.0.1:50921/devtools/browser/c26df2aa-67aa-4264-b1dc-34d6148b9174
You can find a relevant detailed discussion in ModuleNotFoundError: No module named 'webdriver_manager' error even after installing webdrivermanager

Here my solution:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
options = webdriver.ChromeOptions()
s = Service('chromedriver/chromedriver96.exe')
driver = webdriver.Chrome(service=s, options=options)

Selenium Timout on RaspberryPi at Expected Condition

Task
I want to perform a web task using Selenium, geckodriver (ARM build) and Iceweasel on my RaspberryPi 3. The code runs seamlessly on my development machine (Ubuntu, Intel, 64bit), of course using the suitable geckodriver build. The code also prooved successful inside a dockerized container. However, when executing on the RaspberryPi, I get the stated timeout error.
Code
The website is opened in a new window, hence I am switching to the second in window.handles.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
from helpers.automation_helper import *
to_website.click()
el = WebDriverWait(driver, 20).until(
EC.number_of_windows_to_be(2)
)
driver.switch_to.window(driver.window_handles[1])
while get_events_left(driver):
...
In the opened page, I expect edit buttons (= my events) which I intend to return as soon as they are visible. An empty array shall be returned if no events are available, so causing the while loop not to be executed. The expected condition is where the error occurs on the RaspberryPi.
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
def get_events_left(driver):
WebDriverWait(driver, 20).until(
EC.visibility_of_all_elements_located((By.XPATH, "//button[contains(#id, 'protocolEvent')]"))
)
return driver.find_elements_by_xpath("//button[./span/span/svg-icon[#name='edit']]")
Error
Traceback (most recent call last):
File "app.py", line 61, in <module>
while get_events_left(driver):
File "/path/to/app/helpers/automation_helper.py", line 39, in get_events_left
EC.visibility_of_all_elements_located((By.XPATH, "//*[contains(#id,'protocolEvent')]"))
File "/path/to/app/venv/lib/python3.7/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:

Script can't find Element but Console can [Selenium] [Python]

I'm trying to write script to automate the process of downloading the instagram stories but I'm failing already when trying to log in.
I'm writing the code inside Pycharm. I just tried my usual approach to any problem. First, solve it with typing the commands out in the console and if it works writing the commands which worked inside the console down in a script. But here is the issue. The function which worked perfectly fine inside the python console fails inside the script.
I've noticed that my selenium was outdated but upgrading it didn't help ether. I also made a new project to test weather that made difference, which it didn't.
I've also tried skipping the first step inside the script and just opening the url to which I'm redirected. But the second commands failed as well.
When I create a new variable to store the output of the driver.find_element_by_link_text() in, it returns an empty list. This leads me to belive that somehow selenium is unable to search the contetns of the page.
I've also tried the same on Chrome and Safari. This also didn't work.
Here is the code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://instagram.com/")
#next command fails
driver.find_element_by_link_text("Melde dich an.").click()
#if the first command is skipped by entering in the url
#in driver.get(https://www.instagram.com/accounts/login/?source=auth_switcher)
#the following command fails as well.
driver.find_element_by_name("username").send_keys("HereIsTheUsername")
driver.find_element_by_name("password").send_keys("HereIsThePassword")
driver.find_element_by_name("password").send_keys(Keys.RETURN)
driver.close()
In the console these commands worked as mentioned,
Here is what I've entered into the console:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://instagram.com/")
driver.find_element_by_link_text("Melde dich an.").click()
#if it failed here would be an error message
element = driver.find_element_by_name("username")
With the script the error message is this:
Traceback (most recent call last): File
"/Users/alisot2000/PycharmProjects/Instagram downloader/venv/Main.py",
line 6, in
driver.find_element_by_link_text("Melde dich an.").click() File "/Users/alisot2000/PycharmProjects/Instagram
downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py",
line 428, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text) File "/Users/alisot2000/PycharmProjects/Instagram
downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py",
line 978, in find_element
'value': value})['value'] File "/Users/alisot2000/PycharmProjects/Instagram
downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py",
line 321, in execute
self.error_handler.check_response(response) File "/Users/alisot2000/PycharmProjects/Instagram
downloader/venv/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py",
line 242, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.NoSuchElementException: Message: Unable to
locate element: Melde dich an.

Issues you might be experiencing:
1. Synchronization Issue
For most automation tasks, there will be different loading times of web pages based on the processing power of the machine and how strong your internet connection is.
To solve this there are library import Waits from selenium that we can use.
Here is a sample below:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://somedomain/url_that_delays_loading")
try:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
finally:
driver.quit()
2. Wrong language set in selenium profile
Selenium will use your locale in most cases when running automation scripts but in the case that you might want another language here is a sample code for FireFox.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
profile = webdriver.FirefoxProfile()
# switch out 'de' with another two character language code
profile.set_preference("intl.accept_languages",'de')
driver = webdriver.Firefox(firefox_profile=profile, executable_path='<insert_your_gecko_driver_path_here>')
driver.get("https://instagram.com/")
driver.close()
3. Working Code(Tested on Mojave 10.14.5)
Here is a diff of your code and the altered code: https://www.diffchecker.com/G0WWB4Ry
setup a virtualenv
pip install selenium
download geckodriver
set path to gecko driver in code
run script with success result
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
# these two imports are for setting up firefox driver and options
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
# import these three lines below if you are having synchronization issues
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
profile = webdriver.FirefoxProfile()
# here is where you need to set your language explicitly if its defaulting to an undesired language
# just replace the second parameter with your 2 character language code
# this line is not needed if your desired language is locale
profile.set_preference("intl.accept_languages",'de')
# throw in your path here <insert_your_gecko_driver_path_here>
driver = webdriver.Firefox(firefox_profile=profile, executable_path='<insert_your_gecko_driver_path_here>')
driver.get("https://instagram.com/")
# added these two lines below to solve synchronization issue
# element wasnt clickable until page finished loading
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.LINK_TEXT, "Melde dich an.")))
#next command fails
driver.find_element_by_link_text("Melde dich an.").click()
#if the first command is skipped by entering in the url
#in driver.get(https://www.instagram.com/accounts/login/?source=auth_switcher)
#the following command fails as well.
driver.find_element_by_name("username").send_keys("HereIsTheUsername")
driver.find_element_by_name("password").send_keys("HereIsThePassword")
driver.find_element_by_name("password").send_keys(Keys.RETURN)
driver.close()

def ClickElementByName(name,driver):
while True:
try:
driver.find_element_by_name(name).click()
break
except:
sleep(1)
pass
Too long to wait the website run.
Replace ClickElementByName("username", driver)

driver.find_element_by_xpath('//input[#name="username"]').send_keys("HereIsTheUsername")
driver.find_element_by_xpath('//input[#name="password"]').send_keys("HereIsTheUsername")
driver.find_element_by_xpath('//div[text()="Log In"]').click()

Python Selenium Chromedriver not working with --headless option

I am running chromedriver to try and scrape some data off of a website. Everything works fine without the headless option. However, when I add the option the webdriver takes a very long time to load the url, and when I try to find an element (that is found when run without --headless), I receive an error.
Using print statements and getting the html after the url "loaded", I find that there is no html, it's empty (See in output below).
class Fidelity:
def __init__(self):
self.url = 'https://eresearch.fidelity.com/eresearch/gotoBL/fidelityTopOrders.jhtml'
self.options = Options()
self.options.add_argument("--headless")
self.options.add_argument("--window-size=1500,1000")
self.driver = webdriver.Chrome(executable_path='.\\dependencies\\chromedriver.exe', options = self.options)
print("init")
def initiate_browser(self):
self.driver.get(self.url)
time.sleep(5)
script = self.driver.execute_script("return document.documentElement.outerHTML")
print(script)
print("got url")
def find_orders(self):
wait = WebDriverWait(self.driver, 15)
data= wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]'))) #ERROR ON THIS LINE
This is the entire output:
init
<html><head></head><body></body></html>
url
Traceback (most recent call last):
File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 102, in <module>
orders = scrape.find_tesla_orders()
File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 75, in find_tesla_orders
tesla = self.driver.find_element_by_xpath("//a[#href='https://qr.fidelity.com/embeddedquotes/redirect/research?symbol=TSLA']")
File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//a[#href='https://qr.fidelity.com/embeddedquotes/redirect/research?symbol=TSLA']"}
(Session info: headless chrome=74.0.3729.169)
(Driver info: chromedriver=74.0.3729.6 (255758eccf3d244491b8a1317aa76e1ce10d57e9-refs/branch-heads/3729#{#29}),platform=Windows NT 10.0.17763 x86_64)
New error with updated code:
init
<html><head></head><body></body></html>
url
Traceback (most recent call last):
File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 104, in <module>
orders = scrape.find_tesla_orders()
File "C:\Users\Zachary\Documents\Python\Tesla Stock Info\Scraper.py", line 76, in find_tesla_orders
tesla = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]')))
File "C:\Program Files (x86)\Python37-32\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
I have tried finding the answer to this through google but none of the suggestions work. Is anyone else having this issue with certain websites? Any help appreciated.
Update
This script still does not work unfortunately, the webdriver is not loading the page correctly for some reason while headless, even though everything works perfectly without running this using the headless option.

For anyone in the future who is wondering the fix to this, some websites just don't load correctly with the headless option of chrome. I don't think there is a way to fix this. Just use a different browser (like firefox). Thanks to user8426627 for this.

Have you tried using a User-Agent?
I was experiencing the same error. First what I did was to download the HTML source page for both headless and normal with:
html = driver.page_source
file = open("foo.html","w")
file.write(html)
file.close()
The HTML source code for the headless mode was a short file with this line nearly at the end: The page cannot be displayed. Please contact the administrator for additional information. But the normal mode was the expected HTML.
I solve the issue by adding an User-Agent:
from fake_useragent import UserAgent
user_agent = 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.2 (KHTML, like Gecko) Chrome/22.0.1216.0 Safari/537.2'
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument(f'user-agent={user_agent}')
driver = webdriver.Chrome(executable_path = f"your_path",chrome_options=chrome_options)

Try setting the window size as well as being headless. Add this:
chromeOptions.add_argument("--window-size=1920,1080")
The default size of the headless browser is tiny. If the code works when headless is not enabled it might be because your object is outside the window.

Add explicit wait. You should also use another locator, the current one match 3 elements. The element has unique id attribute
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
wait = WebDriverWait(self.driver, timeout)
data = wait.until(ec.visibility_of_element_located((By.CSS_SELECTOR, '[id*="t_trigger_TSLA"]')))

some websites just don't load correctly with the headless option of chrome.
The previous statement is actually wrong. I just got into this problem where Chrome wasn't detecting the elements. When I saw the #LuckyZakary answer I was shocked because someone created a scrapping for the same website with nodeJs and didn't got this error.
#AtulGumar answer helped on Windows but on Ubuntu server it failed. So it wasn't enough. After reading this, all to the bottom, what #AtulGumar missed was to add the –disable-gpu flag.
So it work for me on Windows and Ubuntu server with no GUI with those options:
webOptions = webdriver.ChromeOptions()
webOptions.headless = True
webOptions.add_argument("--window-size=1920,1080")
webOptions.add_argument("–disable-gpu")
driver = webdriver.Chrome(options=webOptions)
I also installed xvfb and other packages as suggested here:
sudo apt-get -y install xorg xvfb gtk2-engines-pixbuf
sudo apt-get -y install dbus-x11 xfonts-base xfonts-100dpi xfonts-75dpi xfonts-cyrillic xfonts-scalable
and executed:
Xvfb -ac :99 -screen 0 1280x1024x16 &
export DISPLAY=:99

strong texttry to add executable path into Service object
options = Options()
options.add_argument('---incognito')
options.add_argument('---disable-extension')
options.add_argument("--no-sandbox")
options.add_argument('-–disable-gpu')
options.add_argument('--headless')
service = Service (executable_path=ChromeDriverManager().install() )
return webdriver.Chrome(service=service , options=options)
its work for me :)

How to start Firefox with with specific profile Selenium Python geckodriver

Here is my code:
profile = webdriver.FirefoxProfile('C:\\Users\\Administrator\\AppData\\Roaming\\Mozilla\\Firefox\\Profiles\\kvycjolb.Prdel')
driver = webdriver.Firefox(profile)
Im not getting any error and firefox starts, but it just does not load with this profile: I have tried changing / to // etc.. but no luck.
This also does not work:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary("C:\\Program Files\\Mozilla Firefox\\firefox.exe")
profile = FirefoxProfile("C:\\Users\\Administrator\\AppData\\Roaming\\Mozilla\\Firefox\\Profiles\\kvycjolb.Prdel")
driver = webdriver.Firefox(firefox_profile=profile, firefox_binary=binary, executable_path="C:\\aprog\\geckodriver.exe")
driver.get('https://google.com')
Im getting error:
C:\aprog>testff
Traceback (most recent call last):
File "C:\aprog\testff.py", line 7, in <module>
driver = webdriver.Firefox(firefox_profile=profile, firefox_binary=binary, e
xecutable_path="C:\\aprog\\geckodriver.exe")
File "C:\Python27\lib\site-packages\selenium\webdriver\firefox\webdriver.py",
line 152, in __init__
keep_alive=True)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 98, in __init__
self.start_session(desired_capabilities, browser_profile)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 188, in start_session
response = self.execute(Command.NEW_SESSION, parameters)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 256, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Unable to find a matchin
g set of capabilities

I think the official answer is found in documentation.
Presently that is:
# Custom profile folder to keep the minidump files
profile = tempfile.mkdtemp(".selenium")
print("*** Using profile: {}".format(profile))
# Use the above folder as custom profile
opts = Options()
opts.add_argument("-profile")
opts.add_argument(profile)
opts.binary = "/Applications/Firefox.app/Contents/MacOS/firefox"
driver = webdriver.Firefox(options=opts,
# hard-code the Marionette port so geckodriver can connect
service_args=["--marionette-port", "2828"])

To start Mozilla Firefox with a specific Firefox Profile through Selenium 3.4.3, geckodriver v0.18.0, Mozila Firefox 53.0 and Python 3.6, you need to create a separate Firefox Profile with the Firefox Profile Manager as per the documentation here.
I have created a Firefox Profile by the name debanjan. This profile got stored in this subdirectory:
"C:\Users\AtechM_03\AppData\Roaming\Mozilla\Firefox\Profiles"
The name of the profile (folder) is w8iy627a.debanjan. So while initiating the WebDriver instance we have to pass the absolute path of the Firefox Profile named as w8iy627a.debanjan as follows:
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
binary = FirefoxBinary("C:\\Program Files\\Mozilla Firefox\\firefox.exe")
profile = FirefoxProfile("C:\\Users\\AtechM_03\\AppData\\Roaming\\Mozilla\\Firefox\\Profiles\\w8iy627a.debanjan")
driver = webdriver.Firefox(firefox_profile=profile, firefox_binary=binary, executable_path="C:\\Utility\\BrowserDrivers\\geckodriver.exe")
driver.get('https://google.com')
Let me know if this answers your question.

def setFirefoxDriver():
profilePath = r"PathHere"
driverPath = r"pathHere\driver.exe"
options = Options()
options.add_argument("-profile")
options.add_argument(profilePath)
dService = Service(driverPath)
d = webdriver.Firefox(service=dService, options=options)
return d
d = setFirefoxProfile()
d.get('https://www.amazon.com/)
to know profile paths search in your firefox about:support or about:profiles
You can test it loading your own profile and see if cookies are loading, ie: when I go to amazon.com amazon recognizes me.
Notice that you can't be using the same profile in 2 different instances, so if you wanna load your profile to test in selenium you shouldn't be using that firefox profile, but another one.

Always use double backslashes in the path (for Windows paths at least):
profile = webdriver.FirefoxProfile('C:\\Users\\Administrator\\AppData\\Roaming\\Mozilla\\Firefox\\Profiles\\kvycjolb.Prree')
In your code, you use both backslashes and forward slashes.