So I was happily using selenium together with Firefox and it seems that my firefox profile wouldn't load anymore some morning. It is driving me nuts to be honest. Whatever I try, the profile that is being used keeps being in the temp folder.
The following snippet is what I am doing
def getFireFoxBrowserWithUserFolder(folder) :
options = webdriver.FirefoxOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-data-dir=" + folder)
firefox = webdriver.FirefoxProfile(folder)
return webdriver.Firefox(options=options, executable_path=r'/home/bunsen/seleniumDriver/geckodriver', firefox_profile=firefox)
This worked up until very recent.
I got the newest version of the geckodriver(0.31.0), running it all on Debian stable, so my FF version is 91.10.0ESR (that is also what I see in the selenium browser).
I am at a loss at this point, anyone with the same problem?
first install webdriver_manager:
pip install webdriver-manager
then:
from selenium import webdriver
from webdriver_manager.firefox import GeckoDriverManager
from selenium.webdriver import Firefox
def getFireFoxBrowserWithUserFolder(folder) :
options = webdriver.FirefoxOptions()
options.add_argument("--disable-blink-features=AutomationControlled")
options.add_argument("user-data-dir=" + folder)
# here you can choose driver version
driver = Firefox(options = options, executable_path=GeckoDriverManager(version="2.26").install())
return driver
More informations about webdriver manager here.
Related
I have been trying to work on my first web-scraping project for which I am using Selenium. However, I seem to be running into some issues with importing the ChromeDriver. I am using Selenium 3.0.0 and am working on Chrome.
webdriver_service = Service(ChromeDriverManager().install())
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
I keep getting the following message: 'chromedriver.exe' executable needs to be in PATH.
Let me know if there's some issue with the file path I am using as I think that is where the issue is coming from.
As answer tells, you should specify the chromedriver-path as following:
driver = webdriver.Chrome('/Users/MyUsername/Downloads/chromedriver.exe')
The executable_path= argument stands for chrome.exe, meaning google-chrome browser.
This should be done through the OS when possible and not through code as it needs to be done for every project this way but it is possible to set environment variables through code
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
# Setting said environment variable
os.environ["CHROME_DRIVER_PATH"] = "Z:\\port\\driver\\chromedriver"
# Reading variable we created
DRIVER_PATH = os.environ.get('CHROME_DRIVER_PATH')
s=Service(DRIVER_PATH)
Got my answer from here
I'm going to be honest and say the path looks good to me. I even tried to run it in Visual Studio and it worked perfectly, so I have no idea why it isn't working for you. That being said I do have a way you can fix it. You can just not use executable_path. You already imported webdriver whose sole purpose is to handle the webdriver for selenium for you.
See here.
You don't even have to change much.
#pip install webdriver-manager, pip install selenium==3.0.0
import os
import logging
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(ChromeDriverManager().install())
#driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
driver.get('https://www.google.com/')
Hope this helps.
You have to take care of a couple of things:
As you are using Selenium 3.0.0 you don't have to create any Service() object. So you can remove the line:
webdriver_service = Service(ChromeDriverManager().install())
Incase you don't intend to use ChromeDriverManager() you can remove the following lines:
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
Unless you are executing your test in headless mode you won't require the argument --headless. So you can remove the line:
chrome_options.add_argument("--headless") # Ensure GUI is off
Unless you are executing your test as a root/administrator you won't nee the argument --no-sandbox. So you can remove the line:
chrome_options.add_argument("--no-sandbox")
Now you can download the matching ChromeDriver version from ChromeDriver page, unzip/untar it and use the following lines of code:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/path/to/chromedriver') # for windows OS use chromedriver.exe
driver.get('https://www.google.com/')
Good morning, I use selenium to do some webscraping, until yesterday everything worked fine, now I get this error, I know it is due to updating the binary, but as I want to share the programm, I would like the binary to be in the folder I created, so that it works with whoever opens the programme.
This is the code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.edge.options import Options
from selenium.webdriver.edge.service import Service
from selenium.webdriver.support.ui import WebDriverWait
# options
options = Options()
options.use_chromium = True
options.add_argument("--headless")
options.add_argument("disable-gpu")
options.add_argument('--allow-running-insecure-content')
options.add_argument('--ignore-certificate-errors')
options.add_experimental_option('excludeSwitches', ['enable-logging'])
# Selenium driver path
s=Service("./Monatseinteilung/driver/msedgedriver.exe")
driver = webdriver.Edge(service=s, options=options)
this is the error:
selenium.common.exceptions.SessionNotCreatedException: Message: session not created: This version of MSEdgeDriver only supports MSEdge version 100
Current browser version is 102.0.1245.30 with binary path C:\Program Files (x86)\Microsoft\Edge\Application\msedge.exe
You have to change your msedgedriver.exe driver to match the MS Edge version you have installed on your computer (102). You can download that from here. Replace your msedgedriver.exe and this script should start working. You will have to do that every time MS Edge updates.
Since you are using Python and since you want to share the program, this may not be very convenient. So instead, you can try libraries such as selenium_driver_updater, or webdriver_auto_update although webdriver_auto_update only seems to support Chrome. After that, you can check for the latest driver available every time you run your script
For selenium_driver_updater
filename = DriverUpdater.install(path=base_dir, driver_name=DriverUpdater.chromedriver, upgrade=True, check_driver_is_up_to_date=True, old_return=False)
driver = webdriver.Chrome(filename)
For webdriver_auto_update:
check_driver('folder/path/of/your/chromedriver')
I am using python selenium for web scraping, but after running the below codes, chrome is launched but did not get the website as I want, instead, it shows 'data;' in url bar.
Could anyone help with the problem? Many thanks!!
PS: My chrome is 88.and chromedriver is also 88. the path of chrome and chromedriver are different, one is in desktop and the other is C://
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
import random
option = Options()
option.add_experimental_option('excludeSwitches', ['enable-logging'])
option.add_argument('--remote-debugging-port=9222')
driver =webdriver.Chrome(executable_path='C:/Users/Desktop/chromedriver.exe')
driver.get("https://www.youtube.com")
Instead of using this
from selenium.webdriver.chrome.options import Options
options = options()
use this
options = webdriver.ChromeOptions()
and I would suggest you to put everything in the same project directory. which is a best practice.
Once you've put the chromedriver in the same directory.
do this:
from os import getcwd
driver = webdriver.Chrome(getcwd() + "\chromedriver.exe", options=options)
Also, try to close the old chrome windows.
It will go to data:// and after that it will redirect to your website.
And also follow Jiya's answer as well.
I am currently working on instagram login script, however i cannot even reach instagram login page with below code, is that the "chromedriver" is being blocked or any idea of my chromedriver configuration or whats wrong am I ?!!
This is my code:
# chromium-chromedrive (Not Python Library)
#!apt-get update # Update OS Files
#!apt install chromium-chromedriver
#!cp /usr/lib/chromium-browser/chromedriver /usr/bin
#!pip install selenium
from selenium import webdriver
import sys
##################################################################
# Add The System Path
##################################################################
sys.path.insert(0,'/usr/bin/chromedriver')
##################################################################
# Config The Chrome Driver Setting In Python
##################################################################
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--no-sandbox") #bypass OS security model
driver = webdriver.Chrome('chromedriver',options=chrome_options)
print("Loading Instagram")
driver.get("https://www.instagram.com/accounts/login/?hl=en")
print(driver.page_source)
It return "...Error Please wait a few minutes before you try again...."
I have tried to (1)remove cookies OR (2)change server IP (using Singapore / US Region IP) OR (3)even using google colab . Same result return. Anymore idea/method that I should try?
P.S. No such problem if i open instagram with my Ubuntu Chrome (With GUI).
I am able to load Instagram with the code below. Make sure you are using the latest version of chrome driver. https://chromedriver.chromium.org/downloads
#Importing selenium
import selenium
from selenium import webdriver
#chromedriver
from webdriver_manager.chrome import ChromeDriverManager
#defining driver
driver = webdriver.Chrome(ChromeDriverManager().install())
#opeining instagram
driver.get('https://www.instagram.com/accounts/login/?hl=en')
I've successfully managed to load a Chrome Profile on MAC and I was trying to replicate the same on Linux but without success (Debian). I'm using Python, and the following works just fine on a MAC
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("user-data-dir=/Users/username/Library/Application Support/Google/Chrome")
driver = webdriver.Chrome('./chromedriver', options=chrome_options)
The same code on Debian, just doesn't work...
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("--user-data-dir=/home/username/.config/google-chrome")
# I've tried also without the `--` but same outcome
# chrome_options.add_argument("user-data-dir=/home/username/.config/google-chrome")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
I honestly now idea what's wrong. I'm using chromedriver 2.45 https://chromedriver.storage.googleapis.com/index.html?path=2.45/ and the issue is related to "Debian GNU/Linux 9 (stretch)" ...
In terms of launching Chrome, they both works. The difference is that on MAC it loads the profile, on Debian it doesn't.
Anyone has an idea why this is happening?
Right, so after many headaches, apparently this is something to do with the fact than I'm using CRD (Chrome Remote Desktop) to connect to the Linux instances!
In fact, you can check the profile location loading chrome://version. When connecting with CRD, this changes from the usual /home/user/.config/google-chrome to /home/user/.config/chrome-remote-desktop/chrome-profile/
All I needed to do is basically replace with the CRD directory to get all the profile information I wanted!
from selenium import webdriver
chrome_options = webdriver.ChromeOptions()
#chrome_options.add_argument("--user-data-dir=/home/user/.config/google-chrome")
chrome_options.add_argument("--user-data-dir=/home/user/.config/chrome-remote-desktop/chrome-profile/")
driver = webdriver.Chrome('./chromedriver_linux', options=chrome_options)
Hopefully this will be helpful for others! :)