I am using python selenium for web scraping, but after running the below codes, chrome is launched but did not get the website as I want, instead, it shows 'data;' in url bar.
Could anyone help with the problem? Many thanks!!
PS: My chrome is 88.and chromedriver is also 88. the path of chrome and chromedriver are different, one is in desktop and the other is C://
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import time
import random
option = Options()
option.add_experimental_option('excludeSwitches', ['enable-logging'])
option.add_argument('--remote-debugging-port=9222')
driver =webdriver.Chrome(executable_path='C:/Users/Desktop/chromedriver.exe')
driver.get("https://www.youtube.com")
Instead of using this
from selenium.webdriver.chrome.options import Options
options = options()
use this
options = webdriver.ChromeOptions()
and I would suggest you to put everything in the same project directory. which is a best practice.
Once you've put the chromedriver in the same directory.
do this:
from os import getcwd
driver = webdriver.Chrome(getcwd() + "\chromedriver.exe", options=options)
Also, try to close the old chrome windows.
It will go to data:// and after that it will redirect to your website.
And also follow Jiya's answer as well.
Related
I would like to use an existing installation of chrome (or firefox or brave browser) with selenium. Like that I could set prespecified settings / extensions (e.g. start nord-vpn when opening a new instance) that are active when the browser is opened with selenium.
I know there is selenium.webdriver.service with the "executeable-path" option, but it doesn't seem to work when you specify a specific chrome.exe, the usage seems to be for the chrome-driver only and then it still opens a "fresh" installation of chrome.
Starting selenium with extension-file I think is also not an option to use with the nord-vpn extension, as I have two-factor authentication active and login every single time would take too much time and effort, if possible at all.
Firefox profile
To use the existing installation of firefox you have to pass the profile path through set_preference() method using an instance of Option from selenium.webdriver.common.options as follows:
from selenium.webdriver import Firefox
from selenium import webdriver
from selenium.webdriver.firefox.service import Service
from selenium.webdriver.firefox.options import Options
profile_path = r'C:\Users\Admin\AppData\Roaming\Mozilla\Firefox\Profiles\s8543x41.default-release'
options=Options()
options.set_preference('profile', profile_path)
service = Service('C:\\BrowserDrivers\\geckodriver.exe')
driver = Firefox(service=service, options=options)
driver.get("https://www.google.com")
You can find a relevant detailed discussion in Error update preferences in Firefox profile: 'Options' object has no attribute 'update_preferences'
Chrome profile
Where as to use an existing installation of google-chrome you have to pass the user profile path through add_argument() using the user-data-dir key through an instance of Option from selenium.webdriver.common.options as follows:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("user-data-dir=C:\\Users\\username\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.google.com/")
You can find a relevant detailed discussion in How to open a Chrome Profile through Python
I have been trying to work on my first web-scraping project for which I am using Selenium. However, I seem to be running into some issues with importing the ChromeDriver. I am using Selenium 3.0.0 and am working on Chrome.
webdriver_service = Service(ChromeDriverManager().install())
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
I keep getting the following message: 'chromedriver.exe' executable needs to be in PATH.
Let me know if there's some issue with the file path I am using as I think that is where the issue is coming from.
As answer tells, you should specify the chromedriver-path as following:
driver = webdriver.Chrome('/Users/MyUsername/Downloads/chromedriver.exe')
The executable_path= argument stands for chrome.exe, meaning google-chrome browser.
This should be done through the OS when possible and not through code as it needs to be done for every project this way but it is possible to set environment variables through code
import os
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
# Setting said environment variable
os.environ["CHROME_DRIVER_PATH"] = "Z:\\port\\driver\\chromedriver"
# Reading variable we created
DRIVER_PATH = os.environ.get('CHROME_DRIVER_PATH')
s=Service(DRIVER_PATH)
Got my answer from here
I'm going to be honest and say the path looks good to me. I even tried to run it in Visual Studio and it worked perfectly, so I have no idea why it isn't working for you. That being said I do have a way you can fix it. You can just not use executable_path. You already imported webdriver whose sole purpose is to handle the webdriver for selenium for you.
See here.
You don't even have to change much.
#pip install webdriver-manager, pip install selenium==3.0.0
import os
import logging
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
chrome_options = Options()
chrome_options.add_argument("--headless") # Ensure GUI is off
chrome_options.add_argument("--no-sandbox")
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
driver = webdriver.Chrome(ChromeDriverManager().install())
#driver = webdriver.Chrome(executable_path='/Users/MyUsername/Downloads/chromedriver.exe')
driver.get('https://www.google.com/')
Hope this helps.
You have to take care of a couple of things:
As you are using Selenium 3.0.0 you don't have to create any Service() object. So you can remove the line:
webdriver_service = Service(ChromeDriverManager().install())
Incase you don't intend to use ChromeDriverManager() you can remove the following lines:
# Silent download of drivers
logging.getLogger('WDM').setLevel(logging.NOTSET)
os.environ['WDM_LOG'] = 'False'
Unless you are executing your test in headless mode you won't require the argument --headless. So you can remove the line:
chrome_options.add_argument("--headless") # Ensure GUI is off
Unless you are executing your test as a root/administrator you won't nee the argument --no-sandbox. So you can remove the line:
chrome_options.add_argument("--no-sandbox")
Now you can download the matching ChromeDriver version from ChromeDriver page, unzip/untar it and use the following lines of code:
from selenium import webdriver
driver = webdriver.Chrome(executable_path='/path/to/chromedriver') # for windows OS use chromedriver.exe
driver.get('https://www.google.com/')
from selenium import webdriver
from selenium.webdriver import FirefoxProfile
driver = webdriver.Firefox(
executable_path=r'C:\Users\subha\Downloads\geckodriver-v0.30.0-win64\geckodriver.exe')
profile = FirefoxProfile(r'C:\Users\subha\AppData\Roaming\\Mozilla\Firefox\Profiles\e7j8p01g.F1')
profile.add_extension(r'C:\Users\subha\AppData\Roaming\Mozilla\Firefox\Profiles\e7j8p01g.F1\extensions\CanvasBlocker#kkapsner.de.xpi')
driver.get('https://github.com')
This is a possible duplicate from previous question. A more elegant solution than the workaround proposed there is the following option from the official documentation which works as expected:
driver = webdriver.Firefox('path/to/executable')
driver.install_addon('~/path/to/addon.xpi')
In your case do:
from selenium import webdriver
driver = webdriver.Firefox(executable_path=r'C:\Users\subha\Downloads\geckodriver-v0.30.0-win64\geckodriver.exe')
driver.install_addon(r'C:\Users\subha\AppData\Roaming\Mozilla\Firefox\Profiles\e7j8p01g.F1\extensions\CanvasBlocker#kkapsner.de.xpi')
#Maybe time.sleep() a few seconds to give time for the installation, then
driver.get('https://github.com')
I'm developing a scraping script to collect some data which is behind an authwall, I've got a custom filter in ublock which gets me past the authwall however when i load chromium with ublock using Selenium it doesn't have the filters. I'm using Linux if that helps.
I've tried getting it to pause before getting the information to allow me to check the filters in place and it is blank.
Here is a portion of the code
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_option_settings = Options()
chrome_option_settings.add_argument('--window-size=1920x1080')
extension_path = r'/home/user/.config/chromium/Default/Extensions/cjpalhdlnbpafiamejdnhcphjbkeiagm/1.20.0_0'
chrome_option_settings.add_argument('load-extension='+extension_path)
chrome_driver = "/usr/bin/chromedriver"
driver = webdriver.Chrome(chrome_options=chrome_option_settings, executable_path=chrome_driver)
driver.get(url)
I've also tried to load the Chrome profile using either however neither help.
chrome_options.add_argument("user-data-dir=/home/user/.config/chromium/Default")
or
chrome_options.add_argument("--profile-directory=/home/user/.config/chromium/Default")
Any help would be greatly appreciated
You could try using Options() and invoking add_extension with the path to ublock, hope this helps
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
executable_path = "path_to_webdriver"
extension_path = r'/home/user/.config/chromium/Default/Extensions/cjpalhdlnbpafiamejdnhcphjbkeiagm/1.20.0_0'
chrome_options.add_extension(extension_path)
driver = webdriver.Chrome(executable_path=executable_path, chrome_options=chrome_options)
Trying to open "Google" or any other page (website) from Chrome via selenium chrome driver in python.
The code is :
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from time import sleep
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
import time
driver = webdriver.Chrome()
driver.get('https://google.com')
However, this opens my chrome window with the specified link and "data;" tab.
Why that data; tab opens? How to fix it?
Using latest versions of Chrome and Chromedriver
You don't need as much module for this just remove all of those apart from:
from selenium import webdriver
And try again you will not get another tab with congaing data.
import time
time.sleep(1)
driver.switch_to.window(driver.window_handles[1])
driver.close()
driver.switch_to.window(driver.window_handles[0])
time.sleep(1)
I'm not sure if its the same problem, but some time ago I made an exe script to run in another PCs and in one of the PCs selenium just didn't worked with Chrome.
This is the question I posted, but the answers didn't help me, hope it works with you: Chromedriver do not open a new session, it opens a new tab in a existing session
If it doesn't work, I made a workaround to run with Firefox instead of Chrome to ensure it would work properly.
With selenium 4 (or newer), you can use the following code to launch a new browser, (without the Chrome is being controlled message), and then the browser navigates to Google in the same tab:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
options = webdriver.ChromeOptions()
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option("useAutomationExtension", False)
service = ChromeService()
driver = webdriver.Chrome(service=service, options=options)
driver.get('https://google.com')