This question already has answers here:
Can a website detect when you are using Selenium with chromedriver?
(25 answers)
How to find the $cdc_ (chromedriver params) in website?
(1 answer)
Selenium Webdriver can't access a website (The requested URL was rejected)
(1 answer)
Closed 2 years ago.
i am facing an issue that i hope anyone can help me with.
i'm trying to scrape a webpage using selenium package on python but it keeps detecting that i'm using selenium and redirects me to a log in page that i can't use.
i tried using a fake agent but it still detected me, i tried changing the "$cdc" in the chrome driver but it still didn't work. apperciate if anyone can help me.
here is the code i'm using :
options = webdriver.ChromeOptions()
ua = UserAgent()
userAgent = ua.random
options.add_argument('--ignore-certificate-errors')
options.add_argument('--incognito')
options.add_argument("--window-size=1920,1080")
options.add_argument('enable-automation')
options.add_argument(f'user-agent={userAgent}')
#options.add_argument('--headless')
driver = webdriver.Chrome()
driver.get("https://www.mcmaster.com/nuts/hex-nuts/medium-strength-steel-hex-nuts-grade-5/")
Related
This question already has answers here:
How to save login data to be recognized for Python Selenium webdriver
(2 answers)
Error when loading cookies into a Python request session
(6 answers)
Closed 2 years ago.
import pickle
import selenium
from selenium import webdriver
import selenium.webdriver
import options
path = "C:\Program Files (x86)\chromedriver.exe"
driver = webdriver.Chrome(path)
chrome_options = Options()
chrome_options.add_argument("user-data-dir=selenium")
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get("http://www.google.com")
the error that i am getting is: NameError: name 'Options' is not defined
am i doing soomething tottaly wrong. im trying to save the cookies so that next time i can login in to my whatsapp without having to manualy do the qr code
This question already has answers here:
ElementNotVisibleException: Message: element not interactable error while trying to click a button through Selenium and Python
(2 answers)
Closed 2 years ago.
I just started web automation and I can't figure out this error..
Code-
from selenium import webdriver
site = webdriver.Chrome()
site.get('https://www.youtube.com')
searchbar = site.find_element_by_xpath('//*[#id="search"]')
searchbar.click()
searchbar.send_keys('Fireproof')
searchButton = site.find_element_by_xpath('//*[#id="search-icon-legacy"]')
searchButton.click()
Error in cmd-
Its opening the browser but not doing anything after that
Issue is with XPath, user below code having updated XPath.
searchbar = driver.find_element_by_xpath('//input[#id="search"]')
searchButton = driver.find_element_by_xpath('//button[#id="search-icon-legacy"]')
This question already has answers here:
How to make Selenium not wait till full page load, which has a slow script?
(2 answers)
Closed 3 years ago.
I have built a scraper in python 3.6 using selenium and scrapinghub crawlera. I am trying to fetch this car and download its photos. https://www.cars.com/vehicledetail/detail/800885995/overview/ but the page just keep loading for long periods of time. What I am trying to figure out is how can I stop the browser from continuously loading after 4 mins.
I have tried both explicit and implicit wait and none has worked.
driver = webdriver.Chrome('/usr/bin/chromedriver',
desired_capabilities=capabilities,
options=chrome_options)
driver.implicitly_wait(180)
driver.get(url)
You need to set the max waiting time for loading with driver.set_page_load_timeout().
In case the page passes its loading time, the browser will throw a TimeoutException. All you need to do is to take care of it
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome('/usr/bin/chromedriver',
desired_capabilities=capabilities,
options=chrome_options)
driver.set_page_load_timeout(time_to_wait)
try:
driver.get(url)
except TimeoutException:
# Do what you need here
This question already has answers here:
Scrolling to top of the page in Python using Selenium
(8 answers)
What is the difference between the different scroll options?
(2 answers)
Closed 3 years ago.
I am trying to scroll Up in a whats app conversation so I record the entire chat (into csv).
chrome_options = Options()
chrome_options.add_argument("--user-data-dir=chrome-data") #saves login
driver = webdriver.Chrome(chrome_options=chrome_options)
driver.get('https://web.whatsapp.com/')
#choose converstaion
#scroll to top not working
Here is what I tried but it didn't work, Ideally I will run headless so I prefer not to use pyautogui
def scroll_to_top():
driver.execute_script("window.scrollTo(0, -document.body.scrollHeight);")
time.sleep(4)
#driver.execute_script("window.scrollTo(0, 0);")
#bd = driver.find_element_by_xpath('//body')
#bd.click()
#bd.send_keys(Keys.CONTROL+Keys.HOME) #throws element not interact-able
Any help is appreciated
Please use below code to scroll to top of your page
solution 1 :
driver.execute_script("window.scrollTo(0, 220)")
solution 2 :
from selenium.webdriver.common.keys import Keys
html = driver.find_element_by_tag_name('html')
html.send_keys(Keys.PAGE_UP)
This question already has answers here:
Selenium: Point towards default Chrome session
(2 answers)
How to open a Chrome Profile through --user-data-dir argument of Selenium
(3 answers)
How to use Chrome Profile in Selenium Webdriver Python 3
(12 answers)
Closed 3 years ago.
So thats my code, I want to open chrome with my default profile information in selenium. but its doesn't load and crashes when I click on profile icon in chrome
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = webdriver.ChromeOptions()
options.add_argument("--user-data-dir=/Users/Library/Application Support/Google/Chrome/Default/Default")
driver = webdriver.Chrome(executable_path='/Users/Desktop/supbot/chromedriver', options=options, service_log_path="/tmp/log")
driver.get('google.com')
You should use full path to the driver file like:
C:\pathtodriver\chromedriver.exe
Also, avoid using Capabilities, try this approach:
System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
WebDriver driver = new ChromeDriver();
driver.get('google.com')