We Keep Coding

Python Selenium - Using Try and except in a loop for automated reading on a website

I'm trying to create an automatic soundcloun player. I started learning selenium on python a few days ago, I'm still a beginner.
So I would like to create a loop that works when he enters the page he clicks on "I accept" (For cookies) then presses the space button and refreshes after 60s
Here is the code:
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from typing import KeysView
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.implicitly_wait(17)
driver.get("https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjkk_fNxM_6AhXrgc4BHWcSCjoQFnoECAYQAQ&url=https%3A%2F%2Fsoundcloud.com%2Fuser-997915825%2Fpmh-ah-aah&usg=AOvVaw2vFzZHA4d8DH1r5GjZ6e4y")
try:
element = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
driver.implicitly_wait(33)
except NoSuchElementException:
ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
while True:
sleep(60)
driver.refresh()
But when I put try: then Except: NoSuchElementException I don't even have time to decompile that I get an error in the lines like :
Try statement must have at least one except or finally clause. Pylance [Ln 17]
Expected expression. Pylance [Ln 28]
I used the right libraries I think:
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import NoSuchElementException
And finally when I try to run it I have this nice error : SyntaxError: expected 'except' or 'finally' block.
When I remove try and except as follows:
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from typing import KeysView
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.implicitly_wait(17)
driver.get("https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjkk_fNxM_6AhXrgc4BHWcSCjoQFnoECAYQAQ&url=https%3A%2F%2Fsoundcloud.com%2Fuser-997915825%2Fpmh-ah-aah&usg=AOvVaw2vFzZHA4d8DH1r5GjZ6e4y")
element = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
driver.implicitly_wait(33)
ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
while True:
sleep(60)
driver.refresh()
He enters the page, he clicks on "I accept" and then he presses the SPACE button to start reading but when he refreshes, since he doesn't have a "cookie acceptance" window anymore, he doesn't even start reading.
that's why I'm looking to use conditions. I've searched several threads about this, I tried to understand their code before copying it but so far all the methods I've tried have failed and it's already been 2 days constantly that it's blocking me, my eyes are starting to hurt.
If anyone can find a solution to my problem, I would be delighted I'll be glad and can find the problem or I'm the problem.
Thanks for your correction, Prophet, here is the new code :
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from typing import KeysView
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.get("https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjkk_fNxM_6AhXrgc4BHWcSCjoQFnoECAYQAQ&url=https%3A%2F%2Fsoundcloud.com%2Fuser-997915825%2Fpmh-ah-aah&usg=AOvVaw2vFzZHA4d8DH1r5GjZ6e4y")
try:
element = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
except NoSuchElementException:
ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
while True:
sleep(7)
driver.refresh()

You are missing indentation.
This is a very basic python syntax rule.
The code inside try or except or any other block should have indentation relatively to previous code.
With the indentation your code could look like this:
driver = webdriver.Chrome(executable_path="chromedriver.exe")
driver.implicitly_wait(17)
driver.get("https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja&uact=8&ved=2ahUKEwjkk_fNxM_6AhXrgc4BHWcSCjoQFnoECAYQAQ&url=https%3A%2F%2Fsoundcloud.com%2Fuser-997915825%2Fpmh-ah-aah&usg=AOvVaw2vFzZHA4d8DH1r5GjZ6e4y")
try:
element = driver.find_element(By.ID, 'onetrust-accept-btn-handler')
action = ActionChains(driver)
action.click(on_element = element)
action.perform()
driver.implicitly_wait(33)
except NoSuchElementException:
ActionChains(driver).key_down(Keys.SPACE).key_up(Keys.SPACE).perform()
Also, driver.implicitly_wait(33) is not a pause command.
driver.implicitly_wait is set once per the session. Is sets the timeout, how long time to try finding (pooling) an element on the page.
If you want to make a pause time.sleep() should be used, but not recommended since we don't generally recommend to use hardcoded sleeps in Selenium code.

Selenium Timeout Exception python

So, i was creating a signup machine using selenium. When this page loads to https://mail.protonmail.com/create/new?language=en it is unable to find element by id/xpath of username. On the other side it was able to find password,passwordc elements. I tried to use WebDriverWait function but it is giving timeout error. Tried many things but this thing is still giving me error. If possible then suggest a way to find element of username on the final page or a perfect WebDriverWait code. Below is my code
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
import time
url = 'https://protonmail.com/'
driver = webdriver.Chrome('C://Users/AAA/Desktop/chromedriver.exe')
driver.get(url)
driver.find_element_by_xpath('//*[#id="bs-example-navbar-collapse-1"]/ul/li[8]/a').click()
time.sleep(2)
driver.find_element_by_xpath('//*[#id="signup-plans"]/div[5]/div[1]/div[1]/div/div[1]/h4').click()
time.sleep(1)
driver.find_element_by_id('freePlan').click()
time.sleep(1)
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, "username")))
driver.find_element_by_xpath('//*[#id="username"]').send_keys('santaking44455')
time.sleep(1)
driver.find_element_by_id('password').send_keys('25J8e5b8')
time.sleep(1)
driver.find_element_by_id('passwordc').send_keys('25J8e5b8')```

You can't find it because it's in an iframe tag higher up in the html source. Switch first to the iframe, then you should be able to interact with the element.
iframe=driver.find_element_by_xpath('//*[#title="Registration form"]')
driver.switch_to.frame(iframe)
driver.find_element_by_xpath('//*[#id="username"]').send_keys('santaking44455')

WebDriverWait on finding element by CSS Selector

I want to retrieve the price of the flight of this webpage using Python 3: https://www.google.es/flights?lite=0#flt=/m/0h3tv./m/04jpl.2018-12-17;c:EUR;e:1;a:FR;sd:1;t:f;tt:o
At first I got an error which after many hours I realized was due to the fact that I wasn't giving the webdriver enough time to load all elements. So to ensure that it had enough time I added a time.sleep like so:
time.sleep(1)
This made it work! However, I've read and was advised to not use this solution and to use WebDriverWait instead. So after many hours and several tutorials im stuck trying to pinpoint the exact CSS class the WebDriverWait should wait for.
The closest I think I've got is:
WebDriverWait(d, 1).until(EC.presence_of_element_located((By.CSS_SELECTOR, ".flt-subhead1.gws-flights-results__price.gws-flights-results__cheapest-price")))
Any ideas on what I'm missing on?

You could use a css attribute = value selector to target, or if that value is dynamic you can use a css selector combination to positional match.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
driver.get("https://www.google.es/flights?lite=0#flt=/m/0h3tv./m/04jpl.2018-12-17;c:EUR;e:1;a:FR;sd:1;t:f;tt:o")
#element = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR , '[jstcache="9322"]')))
element = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR , '.flt-subhead1.gws-flights-results__price.gws-flights-results__cheapest-price span + jsl')))
print(element.text)
#driver.quit()
No results case:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
driver = webdriver.Chrome()
url ="https://www.google.es/flights?lite=0#flt=/m/0h3tv./m/04jpl.2018-12-17;c:EUR;e:1;a:FR;sd:1;t:f;tt:o" #"https://www.google.es/flights?lite=0#flt=/m/0h3tv./m/04jpl.2018-11-28;c:EUR;e:1;a:FR;sd:1;t:f;tt:o"
driver.get(url)
try:
status = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR , 'p[role=status')))
print(status.text)
except TimeoutException as e:
element = WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR , '.flt-subhead1.gws-flights-results__price.gws-flights-results__cheapest-price span + jsl')))
print(element.text)
#driver.quit()

I may be wrong but I think you are trying to get the price of the flight trip.
If my assumption is correct, take a look at my approach. I find the Search Results list, then all the Itinerary inside the Search Results list, loop over and get all the price information. This is the best approach I can come up with and avoiding all the dynamic attributes
from selenium.webdriver import Chrome
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
wait = 20
driver = Chrome()
driver.get("https://www.google.es/flights?lite=0#flt=/m/0h3tv./m/04jpl.2018-12-17;c:EUR;e:1;a:FR;sd:1;t:f;tt:o")
# Get the Search Result List
search_results= WebDriverWait(driver, wait).until(EC.presence_of_element_located((By.CSS_SELECTOR , 'ol[class="gws-flights-results__result-list"]')))
# loop through all the Itinerary
for result in search_results.find_elements_by_css_selector('div[class*="gws-flights-results__collapsed-itinerary"]'):
price = result.find_element_by_css_selector('div[class="gws-flights-results__itinerary-price"]')
print(price.text)
Output
€18

Issues when scraping a web page using selenium in python

I have been given a model to run a successful web scraper on a selected website, however, when i alter this to collect data from a second website, it keeps returning as an error. I'm not sure if it is an error in the code or the website is refusing my requests. Could you please look through this and see where my issue lies. Any help hugely appreciated!
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
try:
driver.get("http://www.caiso.com/TodaysOutlook/Pages/supply.aspx") # load the page
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.highcharts-legend-item highcharts-pie-series highcharts-color-0'))) # wait till relevant elements are on the page
except:
driver.quit() # quit if there was an error getting the page or we've waited 15 seconds and the stats haven't appeared.
stat_elements = driver.find_elements_by_css_selector('.highcharts-legend-item highcharts-pie-series highcharts-color-0')
for el in stat_elements:
print(el.find_element_by_css_selector('b').text)
print(el.find_element_by_css_selector('br').text)
driver.quit()

First of all you are passing wrong CSS as it should be like this
.highcharts-legend-item.highcharts-pie-series.highcharts-color-0
not as you have mentioned.
Then you are closing the browser and then trying to again close it getting the error
try:
driver.get("http://www.caiso.com/TodaysOutlook/Pages/supply.aspx") # load the page
WebDriverWait(driver, 5).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.highcharts-legend-item.highcharts-pie-series.highcharts-color-0'))) # wait till relevant elements are on the page
except:
driver.quit()
Next on list item you are fetching text
print(el.find_element_by_css_selector('b').text)
Debugged Code here:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.common.exceptions import NoSuchElementException
driver = webdriver.Chrome()
try:
driver.get("http://www.caiso.com/TodaysOutlook/Pages/supply.aspx") # load the page
WebDriverWait(driver, 30).until(EC.presence_of_element_located((By.CSS_SELECTOR, '.highcharts-legend-item.highcharts-pie-series.highcharts-color-0'))) # wait till relevant elements are on the page
#driver.quit() # quit if there was an error getting the page or we've waited 15 seconds and the stats haven't appeared.
except TimeoutException:
pass
finally:
try:
stat_elements = driver.find_elements_by_css_selector('.highcharts-legend-item.highcharts-pie-series.highcharts-color-0')
for el in stat_elements:
for i in el.find_elements_by_tag_name('b'):
print(i.text)
for i in el.find_elements_by_tag_name('br'):
print(i.text)
except NoSuchElementException:
print("No Such Element Found")
driver.quit()
I hope this has solved your problem if not then let me know.

Wait for page redirect Selenium WebDriver (Python)

I have a page which loads dynamic content with ajax and then redirects after a certain amount of time (not fixed). How can I force Selenium Webdriver to wait for the page to redirect then go to a different link immediately after?
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome();
driver.get("http://www.website.com/wait.php")

You can create a custom Expected Condition to wait for the URL to change:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
wait = WebDriverWait(driver, 10)
wait.until(lambda driver: driver.current_url != "http://www.website.com/wait.php")
The Expected Condition is basically a callable - you can wrap it into a class overwriting the __call__() magic method as the built-in conditions are implemented.

There are several ExpectedConditions that can be used along with ExplicitWait to handle page redirection:
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
Wait for new page title with title_is(title_name):
wait.until(EC.title_is('New page Title'))
Wait for current URL to be changed to any other URL with url_changes(exact_url):
wait.until(EC.url_changes('https://current_page.com'))
Wait for navigating to exact URL with url_to_be(exact_url):
wait.until(EC.url_to_be('https://new_page.com'))
Also
url_contains(partial_url) could be used to wait for URL that contains specified substring or url_matches(pattern) - to wait for URL matched by passed regex pattern or title_contains(substring) - to wait for new title that contains specified substring

It's a good practice to "wait" for unique (for new page) element to appear.
You may use module expected_conditions.
For example, you could wait for user logo on signing in:
from selenium import webdriver
from selenium.common.exceptions import TimeoutException
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Firefox()
driver.get('http://yourapp.url')
timeout = 5
try:
logo_present = EC.presence_of_element_located((By.ID, 'logo_id'))
WebDriverWait(driver, timeout).until(logo_present)
except TimeoutException:
print "Timed out waiting for page to load"