I am trying to learn Selenium to scrape some Javascript heavy websites. I can locate and extract information just fine. However, I find that for some sites I need to switch my user agent. I did it the following way to test it:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(chrome_options=options, executable_path=PATH)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
The code works and my user agent is switched, however there is one bug that occurs now which did not occur before. The webdriver/browser (Chrome driver) automatically closes after displaying the website for a second without me specifying the driver.quit() argument. When I do not switch my user agent it does not close unless I do and I want to study the page a bit before closing it. I have tried to wait using time.sleep() but this doesn't work.
How can I make the webdriver not close until specified?
Answers are greatly appreciated, preferably with a code example of how to implement the solution.
This should do you nicely:
options.add_experimental_option("detach", True)
in your code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(chrome_options=options, executable_path=PATH)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
I am not sure if it is possible that the problem is related to the webdriver version you are using or not but when I tried to add time.sleep(n) to your code while using webdriver_manager library to download most recent version of ChromeWebDriver I had the chance to look at the website and the browser didn't close until the timer finished.
My code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from webdriver_manager.chrome import ChromeDriverManager
import time
# PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
time.sleep(100)
Related
I have been looking around stackoverflow and I cannot find a solution to this. The solutions I did find were apparently old.
from selenium import webdriver
import time
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("https://www.google.com/")
time.sleep(20)
driver.quit()
I need code to Block all cookies
I try to open the following site using selenium:
https://www.honestdoor.com/
Normally this works fine with every site with the following code:
(I am currently using google-chrome version 98.0.4758 - using ChromeDriverManager for downloading the version - see below in the code)
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import WebDriverWait
from webdriver_manager.chrome import ChromeDriverManager
from fake_useragent import UserAgent
import time
if __name__ == '__main__':
ua = UserAgent()
userAgent = ua.random
options = Options()
# options.add_argument('--headless')
options.add_experimental_option ('excludeSwitches', ['enable-logging'])
options.add_argument("start-maximized")
options.add_argument('window-size=1920x1080')
options.add_argument('--no-sandbox')
options.add_argument('--disable-gpu')
options.add_argument(f'user-agent={userAgent}')
srv=Service(ChromeDriverManager().install())
driver = webdriver.Chrome (service=srv, options=options)
waitWebDriver = WebDriverWait (driver, 10)
link = "https://www.honestdoor.com/"
# link = "https://www.bcassessment.ca/"
# driver.minimize_window() # optional
driver.get (link)
time.sleep(1000)
The site opens with selenium as allways but then the site goes immediately complete white and is still loading forever with the cicle going around in the top left corner (I can only kill the chrome-task in the task manager).
When I open the site in normal chrome or incognito chrome everything works fine - it seem to only crash when I open it with selenium. With other sites (like https://www.bcassessment.ca/ I have no problems at all and the open with selenium as allways)
Why is this not working for this particular website?
Not that super clear about the exact issue you are facing while loading the website. However I was able to load the website using the following code block:
Code Block:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
options = Options()
options.add_argument("start-maximized")
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('excludeSwitches', ['enable-logging'])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
s = Service('C:\\BrowserDrivers\\chromedriver.exe')
driver = webdriver.Chrome(service=s, options=options)
driver.get("https://www.honestdoor.com/")
Browser Snapshot:
from bs4 import BeautifulSoup
import requests
from urllib.request import urlopen
import json
import time
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import ElementNotVisibleException
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_experimental_option("excludeSwitches", ["enable-logging"])
search_key = '성남 스터디카페'
url = "https://map.naver.com/v5/search/" + search_key
driverPath = "chromedriver.exe"
driver = webdriver.Chrome(driverPath, options=chrome_options)
driver.get(url)
I am making web crawler and I can see the page that I entered, but it keeps closing after 2~3 seconds.
I also used detach option.
Chrome version is 91.0.4472.124, so I downloaded 91 version webdriver, but the browser is still closing.
Is there any problem with my code?
It's closing because the main python process will stop after running the code.
If you want to keep the browser is opened, simply add in the end:
time.sleep(1000)
I'm just trying to make a script that plays with the website kijiji (login, post ads). I did a search but can't seem to find a solution to this. One other thread
said to use chrome options and a module called fake useragent but I am still getting a blank screen after clicking the sign in button. Kijiji.ca itself loads up fine but not when signing in. I don't understand why this is happening.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from time import sleep
PATH = executable_path=r"...path to chromedriver.exe..."
options = Options()
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options.add_argument(f'user-agent={userAgent}')
browser = webdriver.Chrome(PATH,options=options)
def login():
browser.get("https://kijiji.ca")
sleep(2)
browser.find_element_by_link_text('Sign In').click()
sleep(2)
login()
I want to get my Whatsapp web (web.whatsapp.com) logged in, at the second time opening the Whatsapp web on chrome driver. Following is my code based on Python need your help.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_path = r"chromedriver.exe"
options = Options();
options.add_argument("user-data-
dir=C:/Users/Username/AppData/Local/Google/Chrome/User Data");
#options.add_argument("--start-maximized");
driver = webdriver.Chrome(chrome_path,chrome_options=options);
#driver = webdriver.Chrome();
driver.get('https://web.whatsapp.com/')
I tried on my Mac, below code and it worked perfectly fine, I don't need to login again
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=/tmp/tarun")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://web.whatsapp.com/')
driver.quit()
For window you can try changing the path as below
options.add_argument("user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("user-data-dir=C:\\Users\\Username\\AppData\\Local\\Google\\Chrome\\User Data")
driver = webdriver.Chrome(chrome_options=options)
driver.get('https://web.whatsapp.com/')
driver.quit()
Here it is for Windows. Works perfect on Python 3.6