This is a script, for my personal study and my educational/didactic purpose, with Tor and Selenium connection.
Both the scraping (team name list) and the Tor connection worked fine.
Then I added a code with Web Driver Wait to press the Cookie button, but now nothing works correctly anymore. The code entered is in contrast to that of Tor.
How can I solve by keeping both the Tor code and the Web Driver Wait code active?
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import sqlite3
### CONNESSIONE TOR ###
from selenium import webdriver
from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
import os
torexe = os.popen('/home/mypc/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US')
profile = FirefoxProfile('/home/mypc/.local/share/torbrowser/tbb/x86_64/tor-browser_en-US/Browser/TorBrowser/Data/Browser/profile.default')
profile.set_preference('network.proxy.type', 1)
profile.set_preference('network.proxy.socks', '127.0.0.1')
profile.set_preference('network.proxy.socks_port', 9050)
profile.set_preference("network.proxy.socks_remote_dns", False)
profile.update_preferences()
firefox_options = webdriver.FirefoxOptions()
firefox_options.binary_location = '/usr/bin/firefox'
driver = webdriver.Firefox(
firefox_profile=profile, options=firefox_options,
executable_path='/usr/bin/geckodriver')
#Scraping SerieA
driver.maximize_window()
wait = WebDriverWait(driver, 20)
driver.get("link")
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[id='onetrust-accept-btn-handler']"))).click()
for SerieA in driver.find_elements(By.CSS_SELECTOR, "a[href^='/squadra'][class^='rowCellParticipantName']"):
print(SerieA.text)
### SALVARE IN DATABASE: Nomi Campionati
con = sqlite3.connect('/home/mypc/Scrivania/folder/Database.db')
cursor = con.cursor()
records_added_Risultati = 0
Values = SerieA
sqlite_insert_query = 'INSERT INTO ARCHIVIO_Squadre_Campionato (Nome_Squadra) VALUES (?);'
count = cursor.executemany(sqlite_insert_query, Values) #executemany, no execute
con.commit()
print("Record inserted successfully ", cursor.rowcount)
records_added_Risultati = records_added_Risultati + 1
cursor.close()
The error is:
Traceback (most recent call last):
File "/usr/lib/python3.8/idlelib/run.py", line 559, in runcode
exec(code, self.locals)
File "/home/mypc/Scrivania/folder/example.py", line 31, in <module>
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[id='onetrust-accept-btn-handler']"))).click()
File "/home/mypc/.local/lib/python3.8/site-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
You are mixing implicitly wait with explicit wait. This is not recommended and causes problems.
I'm quite sure your problems are caused by this.
You can read more about this here
Also see this
UPD
You have a typo in both element locators.
Just a spaces, but totally braking the locators.
Also, in case accept cookies is not stable i.e. sometimes not appearing put it inside try-accept block as following:
driver.maximize_window()
wait = WebDriverWait(driver, 10)
driver.get("https://www.thesiteurl/bla/bla/discrete")
try:
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[id='onetrust-accept-btn-handler']"))).click()
except:
pass
for SerieA in driver.find_elements(By.CSS_SELECTOR, "a[href^='/squadra'][class^='rowCellParticipantName']"):
print(SerieA.text)
Related
I have tried the below code and it is always timing out. But when I look it up with the browser inspector, I can see the Username input element.
I have also tried to find it by ID but not able to.
Tried almost all of the existing questions/solutions on this forum but was unable to figure it out.
driver.get("https://www.dat.com/login")
time.sleep(5)
driver.find_element_by_css_selector("a[href*='https://power.dat.com/']").click()
time.sleep(5)
# explicitly waiting until input element load
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, "username"))
)
except TimeoutException:
print(
"Waited 10 seconds for the username input to load but did not happen..."
)
except Exception as e:
print(f"Exception while waiting for username input to appear: \n {e}")
sys.exit(1)
2 issues here:
After clicking on "DAT Power" on the first page a new tab is opened. To continue working there you need to switch the driver to the second tab.
You will probably want to enter a text into the username there. If so you need to wait for element clickability, not presence only.
Also, the current Selenium version (4.x) no more supports find_element_by_* methods, the new style should be used, as following presented.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.dat.com/login"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://power.dat.com/']"))).click()
new_tab = driver.window_handles[1]
driver.switch_to.window(new_tab)
wait.until(EC.element_to_be_clickable((By.NAME, "username"))).send_keys("name#gmail.com")
The result is
I was trying to use python selenium to log in to a website. But the login is in a bootstrap modal dialog box, I cannot log in successfully using my code. Could someone please let me know what is wrong with my code?
I have tried the following code:
import requests
import time
import os, sys
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver import ActionChains
gecko = os.path.normpath(os.path.join(os.path.dirname(__file__), 'geckodriver'))
binary = FirefoxBinary(r'C:/Program Files/Mozilla Firefox/firefox.exe')
driver = webdriver.Firefox(firefox_binary=binary, executable_path=gecko +'.exe')
driver.maximize_window()
wait = WebDriverWait(driver, 10)
time.sleep(1)
pageurl = "https://www.producemarketguide.com"
driver.get(pageurl)
popup = wait.until(EC.element_to_be_clickable((By.XPATH, '//*[#id="close-dialog"]')))
popup.click()
time.sleep(2)
login = wait.until(EC.element_to_be_clickable((By.XPATH, '/html/body/div[1]/header/div/div[2]/div[2]')))
login.click()
user = wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/app-main/app-widget/screen-layout/main/current-screen/div/screen-login/p[3]')))
user.send_keys('xxx#gmail.com')
password = wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/app-main/app-widget/screen-layout/main/current-screen/div/screen-login/p[4]/input')))
password.send_keys('XXXX')
submit = wait.until(EC.element_to_be_clickable((By.XPATH, '/html/body/app-main/app-widget/screen-layout/main/current-screen/div/screen-login/p[6]/button')))
submit.click()
The error message I got is
Traceback (most recent call last):
File "pmg.py", line 34, in <module>
user = wait.until(EC.presence_of_element_located((By.XPATH, '/html/body/app-main/app-widget/screen-layout/main/current-screen/div/screen-login/p[3]')))
File "C:\Program Files\Python38\lib\site-packages\selenium\webdriver\support\wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
That modal dialog is an iframe, so you need to switch to iframe to enter the credentials, try the below code:
accept_cookies = driver.find_element(By.CSS_SELECTOR, "button#onetrust-accept-btn-handler")
if accept_cookies.is_displayed():
accept_cookies.click()
else:
print("Accept Cookies button not displayed")
popup_dialog = driver.find_element(By.ID,"popup-dialog-window")
close_icon = driver.find_element(By.ID, "close-dialog")
if popup_dialog.is_displayed():
close_icon.click()
else:
print("Welcome dialog not displayed")
time.sleep(1)
# Clicking Login/Register link
driver.find_element(By.CSS_SELECTOR, ".user-login.user-login-front.pull-right #user-piano-login-register").click()
time.sleep(1)
# Switch to the login dialog iframe
driver.switch_to.frame(driver.find_element(By.XPATH,".//iframe[starts-with(#id,'piano-id-')]"))
# enter the credentials and click Login button
driver.find_element(By.XPATH,".//input[#fieldloginemail]").send_keys("emailid")
time.sleep(0.5)
driver.find_element(By.XPATH,".//input[#fieldloginpassword]").send_keys("password")
time.sleep(0.5)
driver.find_element(By.XPATH,".//button[#actionlogin]").click()
I've made a bot that automatically logs me to my Instagram account. Everything works fine, except the fact that I can't automatically press the "Log In" button. This is the error I get:
Traceback (most recent call last):
File "C:/Users/Carole/PycharmProjects/InstagramBot1/main.py", line 40, in <module>
LoginPassword = ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "password"))).send_keys(InstagramPassword).send_keys(Keys.ENTER)
AttributeError: 'NoneType' object has no attribute 'send_keys'
Process finished with exit code 1
And here's my code:
# Importing
from selenium.webdriver.chrome.service import Service
from selenium import webdriver
from selenium.webdriver.chrome.webdriver import WebDriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
import time
from selenium.webdriver.support import ui
# Setting everything up
service = Service("\Program Files (x86)\chromedriver.exe")
# Maximizing Window
# Program Starts
RandomnessTime = input("Select Randomness time:")
InstagramUsername = input("Enter Username:")
InstagramPassword = input("Enter Password:")
#WaitRandomnessTime = "wait"+str(RandomnessTime)
driver = webdriver.Chrome(service=service)
driver.get('https://www.instagram.com/')
wait = ui.WebDriverWait(driver, 10)
driver.maximize_window()
wait.until(ec.element_to_be_clickable((By.CSS_SELECTOR, ".aOOlW.bIiDR"))).click()
RandomnessTime = WebDriverWait(driver, (1 - int(RandomnessTime)))
# Logging in
# def Login(self, )
LoginUsername = ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "username"))).send_keys(InstagramUsername)
LoginPassword = ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "password"))).send_keys(InstagramPassword).send_keys(Keys.ENTER)
# Clicking "Not Now".
ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.CSS_SELECTOR, ".aOOlW.HoLwm"))).click()
Can someone spot the problem with what I've written?
I would rather have this xpath
//div[text()='Log In']//parent::button
to click on Login button.
In you code something like this :
ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.XPATH, "//div[text()='Log In']//parent::button"))).click()
The issue that you've with your code is :
ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "password"))).send_keys(InstagramPassword).send_keys(Keys.ENTER)
does not return anything, not sure why you have a returned variable for that.
If you wanna stick with your way, I would suggest to pass enter in same send_keys like this.
ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "password"))).send_keys(InstagramPassword, Keys.ENTER)
send_keys() is a method and returns nothing, to approach what you want you need to:
LoginPassword = ui.WebDriverWait(driver, 10).until(ec.element_to_be_clickable((By.NAME, "password")))
LoginPassword.send_keys(InstagramPassword)
LoginPassword.send_keys(Keys.ENTER)
I have a simple task that I want to automate. I want to open a URL, click a button which takes me to the next page, fills in a search term, clicks the "search" button and prints out the url and source code of the results page. I have written the following.
from selenium import webdriver
import time
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
#open URL
driver.get("https://www.searchiqs.com/nybro/")
time.sleep(5)
#click Log In as Guest button
driver.find_element_by_id('btnGuestLogin').click()
time.sleep(5)
#insert search term into Party 1 form field and then search
driver.find_element_by_id('ContentPlaceholder1_txtName').send_keys("Moses")
driver.find_element_by_id('ContentPlaceholder1_cmdSearch').click()
time.sleep(10)
#print and source code
print driver.current_url
print driver.page_source
driver.quit()
I am not sure where I am going wrong but I have followed a number of tutorials on how to click buttons and fill forms. I get this error instead.
Traceback (most recent call last):
File "phant.py", line 12, in <module>
driver.find_element_by_id('btnGuestLogin').click()
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 269, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 752, in find_element
'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: {"errorMessage":"Unable to find element with id 'btnGuestLogin'","request":{"headers":{"Accept":"application/json","
Accept-Encoding":"identity","Connection":"close","Content-Length":"94","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:35670","User-Agent":"Python-urllib/2.7"
},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"d38e5fa0-5349-11e6-b0c2-758ad3d2c65e\", \"value\": \"btnGuestLogin\"}","url":"/element","urlP
arsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":""
,"protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/d38e5fa0-5349-11e6-b0c2-758ad3d2c65e/element"}}
Screenshot: available via screen
The error seems to suggest that the element with that id does not exist yet it does.
--- EDIT: Changed code to use WebDriverWait ---
I have changed some things around to implement WebDriverWait
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
#open URL
driver.get("https://www.searchiqs.com/nybro/")
#click Log In as Guest button
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "btnGuestLogin"))
)
element.click()
#wait for new page to load, fill in form and hit search
element2 = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "ContentPlaceholder1_cmdSearch"))
)
#insert search term into Party 1 form field and then search
driver.find_element_by_id('ContentPlaceholder1_txtName').send_keys("Moses")
element2.click()
driver.implicitly_wait(10)
#print and source code
print driver.current_url
print driver.page_source
driver.quit()
It still raises this error
Traceback (most recent call last):
File "phant.py", line 14, in <module>
EC.presence_of_element_located((By.ID, "btnGuestLogin"))
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/support/wait.py", line 80, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Screenshot: available via screen
The WebDriverWait approach actually works for me as is:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
driver.get("https://www.searchiqs.com/nybro/")
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "btnGuestLogin"))
)
element.click()
No errors. PhantomJS version 2.1.1, Selenium 2.53.6, Python 2.7.
The issue might be related to SSL and PhantomJS, either work through http:
driver.get("http://www.searchiqs.com/nybro/")
Or, try ignoring SSL errors:
driver = webdriver.PhantomJS(service_args=['--ignore-ssl-errors=true', '--ssl-protocol=any'])
I am writing a bot that will follow the users on www.quora.com . following is the part of code I am using where I get timeout exception:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
import urllib
driver = webdriver.Firefox()
driver.get('http://www.quora.com/')
time.sleep(10)
wait = WebDriverWait(driver, 10)
form = driver.find_element_by_class_name('regular_login')
time.sleep(10)
#add explicit wait
username = form.find_element_by_name('email')
time.sleep(10)
#add explicit wait
username.send_keys('abc#gmail.com')
time.sleep(30)
#add explicit wait
password = form.find_element_by_name('password')
time.sleep(30)
#add explicit wait
password.send_keys('def')
time.sleep(30)
#add explicit wait
password.send_keys(Keys.RETURN)
time.sleep(30)
#search = driver.find_element_by_name('search_input')
search = wait.until(EC.presence_of_element_located((By.XPATH, "//form[#name='search_form']//input[#name='search_input']")))
search.clear()
search.send_keys('Kevin Rose')
search.send_keys(Keys.RETURN)
link = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Kevin Rose")))
link.click()
#Wait till the element is loaded (Asynchronusly loaded webpage)
handle = driver.window_handles
driver.switch_to.window(handle[1])
#switch to new window
element = WebDriverWait(driver, 2).until(EC.presence_of_element_located((By.PARTIAL_LINK_TEXT, "Followers")))
time.sleep(30)
element.click()
#goes to Kevin Rose followers page
time.sleep(30)
button = driver.find_elements_by_xpath("//a[contains(text(), 'Follow')]")
#Locate follow button on the page
no_of_followers = len(button)
#total number of unfollowed users
print no_of_followers
while(no_of_followers > 0):
# execute only if there are unfollowed users on page
count = 1
while(count < no_of_followers):
time.sleep(30)
link = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Follow")))
time.sleep(30)
link.click()
time.sleep(30)
print count
count = count + 1
time.sleep(30)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
time.sleep(30)
button = driver.find_elements_by_xpath("//a[contains(text(), 'Follow')]")
time.sleep(30)
no_of_followers = len(button)
After executing the code i am getting "TimeoutException" error in the inner loop after successful execution once.
How can I solve this?
Traceback:
Traceback (most recent call last): File "C:\Python27\quorabot7",
line 72, in
link = wait.until(EC.presence_of_element_located((By.LINK_TEXT, "Follow"))) File
"C:\Python27\lib\site-packages\selenium\webdriver\support\wait.py",
line 71, in until
raise TimeoutException(message) TimeoutException: Message: ''
You're getting a TimeoutException because Selenium can't find that element within the time that you've set as your wait. This means that your locator strategy is incorrect.
I have not tested your other locators, but if it is truly the inner loop that is failing... my solution is below.
After looking through the DOM on Kevin Hart's page, I can see that the button you're interested in is:
<a class="follow_button with_count" href="#" action_click="UserFollow" id="__w2_Mab4s9V_follow_user">Follow<span class="count">43.8k</span></a>
You should try this:
link = wait.until(EC.presence_of_element_located(\
(By.className, "follow_button with_count")))
or this:
link = wait.until(EC.presence_of_element_located(\
(By.XPATH, '//a[#action_click="UserFollow"]')))
The correct is By.CLASS_NAME, not By.ClassName