I have following code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument('--headless')
options.add_argument('--disable-gpu')
driver = webdriver.Chrome("chromedriver.exe", chrome_options=options)
driver.get("*some site*")
When i start debugging, it stucks at the driver.get("*some site*") line. It simply does not start anything.
Any suggestions please?
Everything is up to date. Using windows 7.
EDIT: using Python 3.6
EDIT 2: hope its helpful
try running with this options, for me (on my win machine) --no-sandbox helped.
--log-path and verbose obviously helps with debugging.
ch_options.add_argument('--headless')
ch_options.add_argument('--disable-gpu')
ch_options.add_argument('--no-sandbox')
ch_options.add_argument('--log-path=chromedriver.log')
ch_options.add_argument('--verbose')
and for user agent (in case some websites protest):
ch_options.add_argument(
'--user-agent="valid user agent :)"')
and starting it:
driver = webdriver.Chrome(chrome_options=ch_options)
my chromedriver is in PATH environmental variable so no need to pass executable in here.
btw. phantomjs driver is not updated any more so try to resolve this and switch to chromedriver. I.e. for me the phantomjs driver wont even let me change user-agent, just on python binding though.
Related
I have been trying to work on my first web-scraping project for which I am using Selenium. However, I seem to be running into some issues with importing the ChromeDriver. I am using Selenium 3.0.0 and am working on Chrome.
webdriver_service = Service(ChromeDriverManager().install())
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
I keep getting the following message: 'chromedriver.exe' executable needs to be in PATH.
Let me know if there's some issue with the file path I am using as I think that is where the issue is coming from.
As answer tells, you should specify the chromedriver-path as following:
driver = webdriver.Chrome('/Users/MyUsername/Downloads/chromedriver.exe')
The executable_path= argument stands for chrome.exe, meaning google-chrome browser.
This should be done through the OS when possible and not through code as it needs to be done for every project this way but it is possible to set environment variables through code
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
# Setting said environment variable
os.environ["CHROME_DRIVER_PATH"] = "Z:\\port\\driver\\chromedriver"
# Reading variable we created
DRIVER_PATH = os.environ.get('CHROME_DRIVER_PATH')
s=Service(DRIVER_PATH)
Got my answer from here
I'm going to be honest and say the path looks good to me. I even tried to run it in Visual Studio and it worked perfectly, so I have no idea why it isn't working for you. That being said I do have a way you can fix it. You can just not use executable_path. You already imported webdriver whose sole purpose is to handle the webdriver for selenium for you.
See here.
You don't even have to change much.
#pip install webdriver-manager, pip install selenium==3.0.0
import os
import logging
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(ChromeDriverManager().install())
#driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
driver.get('https://www.google.com/')
Hope this helps.
You have to take care of a couple of things:
As you are using Selenium 3.0.0 you don't have to create any Service() object. So you can remove the line:
webdriver_service = Service(ChromeDriverManager().install())
Incase you don't intend to use ChromeDriverManager() you can remove the following lines:
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
Unless you are executing your test in headless mode you won't require the argument --headless. So you can remove the line:
chrome_options.add_argument("--headless") # Ensure GUI is off
Unless you are executing your test as a root/administrator you won't nee the argument --no-sandbox. So you can remove the line:
chrome_options.add_argument("--no-sandbox")
Now you can download the matching ChromeDriver version from ChromeDriver page, unzip/untar it and use the following lines of code:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/path/to/chromedriver') # for windows OS use chromedriver.exe
driver.get('https://www.google.com/')
https://www.codegrepper.com/code-examples/python/python+selenium+brave+browser
I see this example to use brave browser on windows. Is it supposed to work on Catalina as well by just replacing driver_path and brave_path?
Also, Chromedriver is only for Chrome. How to determine which version of chromedriver should be used for brave browser?
https://chromedriver.chromium.org
from selenium import webdriver
driver_path = "C:/Users/username/PycharmProjects/chromedriver.exe"
brave_path = "C:/Program Files (x86)/BraveSoftware/Brave-Browser/Application/brave.exe"
option = webdriver.ChromeOptions()
option.binary_location = brave_path
# option.add_argument("--incognito") OPTIONAL
# option.add_argument("--headless") OPTIONAL
# Create new Instance of Chrome
browser = webdriver.Chrome(executable_path=driver_path, chrome_options=option)
browser.get("https://www.google.es")
Prerequisites:
Your chromedriver version should match your Brave Browser web driver version.
To make sure it does:
Check ChromeDriver version with brew info chromedriver. Along the lines of the output it should read chromedriver: 89.0.4389.23 (latest version as of writing this post)
Open Brave Browser, in menu bar click Brave -> About Brave. Along the lines it should read Version 1.22.71 Chromium: 89.0.4389.114 (Official Build) (x86_64) (again, latest as of writing this post)
These two should match, however, i am not entirely sure to which degree, since, as you can see here, last entries (.23 and .114) don't match, yet this works perfectly fine on my machine (macOS Big Sur 11.2.3) I don't think macOS version should really matter, but i still mentioned it for the sake of completeness.
Finally run the following code (replace paths with ones on your machine if they are different):
from selenium import webdriver
driverPath = '/usr/local/Caskroom/chromedriver/89.0.4389.23/chromedriver'
binaryPath = '/Applications/Brave Browser.app/Contents/MacOS/Brave Browser'
options = webdriver.ChromeOptions()
options.binary_location = binaryPath
browser = webdriver.Chrome(executable_path=driverPath, chrome_options=options)
browser.get("https://www.google.es")
If you have never used chromedriver before, after runnning the code you should see a macOS prompt saying that chromedriver is from unknown developer or was downloaded from the internet, smth like that. Close that prompt (It's important that you do this before moving on). Then go to System Preferences -> Security & Privacy -> press the lock icon and unlock it and then approve chromedriver to run on your machine. Run the above code again, a new macOS prompt will appear saying smth about unknown developer again, this time you can just click Open. Brave Browser window should pop up at this point. At least it did on my machine.
P.S. I apologise for possibly going into too much detail, but sometimes i get really frustrated with answers which skip parts which are considered to be obvious
For the next person looking, this is the most current way of using Brave with Selenium on Mac:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
driverPath = "/Applications/chromedriver" # Path to ChromeDriver
service = Service(driverPath)
options = webdriver.ChromeOptions()
options.binary_location = "/Applications/Brave Browser.app/Contents/MacOS/Brave Browser" # Path to Brave Browser (this is the default)
driver = webdriver.Chrome(service=service, options=options)
# From here its Selenium as usual, example:
driver.get("https://google.com")
print(driver.title)
driver.close()
I am using python selenium for web scraping, but after running the below codes, chrome is launched but did not get the website as I want, instead, it shows 'data;' in url bar.
Could anyone help with the problem? Many thanks!!
PS: My chrome is 88.and chromedriver is also 88. the path of chrome and chromedriver are different, one is in desktop and the other is C://
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
import random
option = Options()
option.add_experimental_option('excludeSwitches', ['enable-logging'])
option.add_argument('--remote-debugging-port=9222')
driver =webdriver.Chrome(executable_path='C:/Users/Desktop/chromedriver.exe')
driver.get("https://www.youtube.com")
Instead of using this
from selenium.webdriver.chrome.options import Options
options = options()
use this
options = webdriver.ChromeOptions()
and I would suggest you to put everything in the same project directory. which is a best practice.
Once you've put the chromedriver in the same directory.
do this:
from os import getcwd
driver = webdriver.Chrome(getcwd() + "\chromedriver.exe", options=options)
Also, try to close the old chrome windows.
It will go to data:// and after that it will redirect to your website.
And also follow Jiya's answer as well.
Hi guys I'm running some crawling script with Selenium and Python, I want to run the Chrome in headless mode so I set the headless options to true as below
from selenium.webdriver.chrome.options import Options
from selenium import webdriver
options = Options()
options.headless = True
options.add_argument("--start-maximized")
options.add_argument("--window-size=1920,1080")
options.add_argument("--disable-gpu")
options.add_argument("--no-sandbox")
driver = webdriver.Chrome('chromedriver.exe', options=options)
But when running the script, Chrome return the web in mobile version (I've captured screenshot to check the error). Because of this my script cannot run properly
I've try many ways to change it back to desktop website, added arguments like "--window-size=1920,1080", "--start-maximized", etc.. then set browser.maximize_window() and browser.set_window_size(). I also try different chromedriver version but it doesn't work at all
Can anyone help me please? Many thanks.
Yep, I had a very similar issue. The first thing you need to do is manually identify your user agent, check out this site. For example, your user agent could be a long string describing a Safari browser running on macOS that renders web pages using the WebKit engine.
Now go ahead and add in an option to manually set your user agent
options.add_argument("user-agent=User-Agent: your user agent string here")
An example might look like this:
options.add_argument("user-agent=User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/507.06 Safari/507.06")
I've successfully managed to load a Chrome Profile on MAC and I was trying to replicate the same on Linux but without success (Debian). I'm using Python, and the following works just fine on a MAC
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("user-data-dir=/Users/username/Library/Application Support/Google/Chrome")
driver = webdriver.Chrome('./chromedriver', options=chrome_options)
The same code on Debian, just doesn't work...
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--user-data-dir=/home/username/.config/google-chrome")
# I've tried also without the `--` but same outcome
# chrome_options.add_argument("user-data-dir=/home/username/.config/google-chrome")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
I honestly now idea what's wrong. I'm using chromedriver 2.45 https://chromedriver.storage.googleapis.com/index.html?path=2.45/ and the issue is related to "Debian GNU/Linux 9 (stretch)" ...
In terms of launching Chrome, they both works. The difference is that on MAC it loads the profile, on Debian it doesn't.
Anyone has an idea why this is happening?
Right, so after many headaches, apparently this is something to do with the fact than I'm using CRD (Chrome Remote Desktop) to connect to the Linux instances!
In fact, you can check the profile location loading chrome://version. When connecting with CRD, this changes from the usual /home/user/.config/google-chrome to /home/user/.config/chrome-remote-desktop/chrome-profile/
All I needed to do is basically replace with the CRD directory to get all the profile information I wanted!
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument("--user-data-dir=/home/user/.config/google-chrome")
chrome_options.add_argument("--user-data-dir=/home/user/.config/chrome-remote-desktop/chrome-profile/")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
Hopefully this will be helpful for others! :)