I'm just trying to make a script that plays with the website kijiji (login, post ads). I did a search but can't seem to find a solution to this. One other thread
said to use chrome options and a module called fake useragent but I am still getting a blank screen after clicking the sign in button. Kijiji.ca itself loads up fine but not when signing in. I don't understand why this is happening.
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from time import sleep
PATH = executable_path=r"...path to chromedriver.exe..."
options = Options()
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options.add_argument(f'user-agent={userAgent}')
browser = webdriver.Chrome(PATH,options=options)
def login():
browser.get("https://kijiji.ca")
sleep(2)
browser.find_element_by_link_text('Sign In').click()
sleep(2)
login()
Related
This question already has answers here:
Python selenium keep browser open
(4 answers)
Closed 5 months ago.
I'm using Selenium to auto-fill forms using forminfo.py for the inputted information. (Right now I'm testing it out on Instagram just to figure everything out.) What I have done is, I have it check to see if the google chrome driver is already installed in a particular location and if not it installs it and if so it just continues and runs the code to fill out the form.
The problem I'm having is, everything runs fine IF the driver isn't installed and it installs it manually. BUT If the driver is already installed, after the form is filled out and it hits the submit button it closes the browser immediately. Ive only been learning Python for about a few weeks now and I cant figure out whats wrong. Also every where I've look I can find a solution to the problem. Also, I'm not getting any errors in my terminal.
################## IMPORTS ##################
import forminfo
import os
import urllib.request as urllib
import webbrowser
import zipfile
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.common.proxy import Proxy, ProxyType
from time import sleep
################## CHROME DRIVER DOWNLOAD ##################
path = "C:/Users/"+os.getlogin()+"/Documents/chromedriver.exe"
isFile = os.path.isfile(path)
if not os.path.isfile(path):
webbrowser.get('windows-default').open("http://www.google.com", new=0)
zip_path, _ = urllib.urlretrieve(forminfo.downloadurl)
urllib.urlopen = zip_path
extract_dir = "C:/Users/"+os.getlogin()+"/Documents"
zipfile.ZipFile(zip_path, "r").extractall(extract_dir)
else:
print("Starting now....")
################## PROXY INFO ##################
prox = Proxy()
prox.proxy_type = ProxyType.MANUAL
prox.socks_version = 5
prox.socks_proxy = forminfo.socks5ip
#prox.http_proxy = "ip_addr:port"
#prox.ssl_proxy = "ip_addr:port"
################## BROWSER ##################
capabilities = webdriver.DesiredCapabilities.CHROME
prox.add_to_capabilities(capabilities)
#
capabilities['acceptInsecureCerts'] = True
capabilities['acceptSslCerts'] = True
#
driver_service = Service(executable_path="C:/Users/"+os.getlogin()+"/Documents/chromedriver")
chrome_options = Options()
chrome_options.add_argument("--incognito")
chrome_options.add_argument("start-maximized")
chrome_options.add_experimental_option("useAutomationExtension", False)
chrome_options.add_experimental_option("excludeSwitches",["enable-automation"])
#
chrome_options.add_argument("--ignore-certificate-error")
chrome_options.add_argument("--ignore-ssl-errors")
#
driver = webdriver.Chrome(service=driver_service, options=chrome_options, desired_capabilities=capabilities)
driver.get('https://www.instagram.com')
################## BOT ##################
sleep(3)
elem = driver.find_element(By.NAME, "username")
elem.send_keys(forminfo.username)
elem = driver.find_element(By.NAME, "password")
elem.send_keys(forminfo.password)
sleep(3)
elem = driver.find_element(By.XPATH, "//*[#id=\"loginForm\"]/div/div[3]/button/div")
elem.click()
The only solution Ive found to the problem is here by adding a breakpoint() at the very end of my code. But I feel like this would mess some things up down the road if I decided to expand on this code and eventually turn it into a .exe file with pyIntsaller or Nuitak. So I wanted to see If there was a solution before just moving forward with that resolution.
I know it's a problem with the ################## CHROME DRIVER DOWNLOAD ################## code because I wasn't having this problem until I decided to add that code. This Is the very most recent part I added and everything was fine until then.
Converting comment to an answer.
You can use the detach option to let the browser Window stay open after using it with Selenium, example:
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
I am using Selenium webdriver to fill out a form, the fields are good, but when I do a .submit using the login button, or do a .submit after the password field, the chrome browser will say "Unable to connect to internet. Please check your internet connection."
the internet is definitely up since I am able to open other websites on another browser (including a new session of chrome).
This is my code:
import time
import selenium
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
chrome_options = webdriver.ChromeOptions();
chrome_options.add_experimental_option("excludeSwitches", ['enable-automation']);
options = Options()
# options.add_argument('start-maximumized')
# options.add_argument('disable-infobars')
chrome_options.add_argument('no-sandbox')
PATH = "c:\scripts\chromedriver.exe"
# driver = webdriver.Chrome(executable_path=PATH)
driver = webdriver.Chrome(chrome_options=chrome_options,
executable_path='c:\scripts\chromedriver.exe')
driver.get('https://stupidwebsite.org')
time.sleep(1)
input_username = driver.find_element_by_id('username')
time.sleep(1)
input_username.send_keys("myself#email.com")
input_password = driver.find_element_by_id('password')
input_password.send_keys("mypassword#123")
input_password.submit()
# esubmit=driver.find_element_by_xpath
('/html/body/div/div[1]/div[2]/div/div/form/div[2]/div/button')
# esubmit = driver.find_element_by_name('MuiButton-label')
# esubmit.submit()
time.sleep(2)
# driver.quit()
I changed it a bit, and tried out google, and it works:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
#set chromodriver.exe path
driver = webdriver.Chrome(executable_path="C:\scripts\chromedriver.exe")
driver.implicitly_wait(0.5)
#launch URL
driver.get("https://www.google.com/")
#identify search box
m = driver.find_element_by_name("q")
#enter search text
m.send_keys("Tutorialspoint")
time.sleep(0.2)
#perform Google search with Keys.ENTER
m.send_keys(Keys.ENTER)
Please help me out. Thank you.
I am trying to learn Selenium to scrape some Javascript heavy websites. I can locate and extract information just fine. However, I find that for some sites I need to switch my user agent. I did it the following way to test it:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(chrome_options=options, executable_path=PATH)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
The code works and my user agent is switched, however there is one bug that occurs now which did not occur before. The webdriver/browser (Chrome driver) automatically closes after displaying the website for a second without me specifying the driver.quit() argument. When I do not switch my user agent it does not close unless I do and I want to study the page a bit before closing it. I have tried to wait using time.sleep() but this doesn't work.
How can I make the webdriver not close until specified?
Answers are greatly appreciated, preferably with a code example of how to implement the solution.
This should do you nicely:
options.add_experimental_option("detach", True)
in your code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
options.add_experimental_option("detach", True)
driver = webdriver.Chrome(chrome_options=options, executable_path=PATH)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
I am not sure if it is possible that the problem is related to the webdriver version you are using or not but when I tried to add time.sleep(n) to your code while using webdriver_manager library to download most recent version of ChromeWebDriver I had the chance to look at the website and the browser didn't close until the timer finished.
My code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from fake_useragent import UserAgent
from webdriver_manager.chrome import ChromeDriverManager
import time
# PATH ="C:/my/path/to/chromedriver.exe"
ua = UserAgent()
userAgent = ua.random
print(userAgent)
options = Options()
options.add_argument(f'user-agent={userAgent}')
driver = webdriver.Chrome(ChromeDriverManager().install(), chrome_options=options)
driver.get("https://www.whatismybrowser.com/detect/what-is-my-user-agent")
time.sleep(100)
from bs4 import BeautifulSoup
import requests
from urllib.request import urlopen
import json
import time
from seleniumwire import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import ElementNotVisibleException
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
chrome_options.add_experimental_option("excludeSwitches", ["enable-logging"])
search_key = '성남 스터디카페'
url = "https://map.naver.com/v5/search/" + search_key
driverPath = "chromedriver.exe"
driver = webdriver.Chrome(driverPath, options=chrome_options)
driver.get(url)
I am making web crawler and I can see the page that I entered, but it keeps closing after 2~3 seconds.
I also used detach option.
Chrome version is 91.0.4472.124, so I downloaded 91 version webdriver, but the browser is still closing.
Is there any problem with my code?
It's closing because the main python process will stop after running the code.
If you want to keep the browser is opened, simply add in the end:
time.sleep(1000)
I've a browser(not generated by selenium) which is already logged in to the website.
I need selenium to use the existing cookies so as to not prompt each generated browser for a new login.
here's what i've already tried from what i searched but still can't make it work.
code summary: if the selenium generated browser is already logged in, the browser will close and if it redirected to the login page, it would wait 60 seconds before closing.
import selenium.webdriver.support.ui as ui
import contextlib
import getpass
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
websiteURL = "https://website.alreadylogged.in"
currentUser = getpass.getuser()
chromeOptions = Options()
chromeOptions.add_argument("--user-data-dir=C:\\Users\\" + currentUser + "\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
driver = webdriver.Chrome(executable_path="C:\\Users\\"+currentUser+"\\Documents\\chromedriver.exe",options=chromeOptions)
with contextlib.closing(driver) as chromeDriver:
chromeOptions.add_argument("user-data-dir=C:\\Users\\" + currentUser + "\\AppData\\Local\\Google\\Chrome\\User Data\\Default")
chromeDriver.get(websiteURL)
wait = ui.WebDriverWait(chromeDriver, 60) # timeout after 60 seconds
wait.until(lambda driver: chromeDriver.find_elements_by_class_name(' windows chrome '))