We Keep Coding

Scraping a website with selenium in python: Problem in clicking "download images" Buttons

Good morning. I am new in using python and webscraping. I have to download a series of images from this link (just the last part of the urll change for the following pages): https://historisch.cbs.nl/detail.php?nav_id=0-1&index=2&id=30568043
What I want to do is click on the two download buttons in order that at the end I download the images on my laptop (https://i.stack.imgur.com/zWAJg.png).
Here is my code until now:
browser = webdriver.Chrome(service=s, options=chrome_options)
for i in range(8047,8051):
no = str(i)
browser.get ("https://historisch.cbs.nl/detail.php?nav_id=1-1&index=2&id=3056"+str(i)+".jpeg")
download = WebDriverWait(browser, 5).until(EC.element_to_be_clickable((By.XPATH,'//*[#id="downloadDirect"]')))
download.click()
t.sleep(1)
download = WebDriverWait(browser, 3).until(EC.element_to_be_clickable((By.XPATH,'//*[#id="downloadResLink"]')))
download.click()
I recive this error:
----> 8 download = WebDriverWait(browser, 5).until(EC.element_to_be_clickable((By.XPATH,'//*[#id="downloadDirect"]')))
9 download.click()
10 t.sleep(1)
~\anaconda3\lib\site-packages\selenium\webdriver\support\wait.py in until(self, method, message)
88 if time.monotonic() > end_time:
89 break
---> 90 raise TimeoutException(message, screen, stacktrace)
91
92 def until_not(self, method, message: str = ""):
TimeoutException: Message:
Stacktrace:
Backtrace:
Ordinal0 [0x00371ED3+2236115]
Ordinal0 [0x003092F1+1807089]
Ordinal0 [0x002166FD+812797]
Ordinal0 [0x002455DF+1005023]
Ordinal0 [0x002457CB+1005515]
Ordinal0 [0x00277632+1209906]
Ordinal0 [0x00261AD4+1120980]
Ordinal0 [0x002759E2+1202658]
Ordinal0 [0x002618A6+1120422]
Ordinal0 [0x0023A73D+960317]
Ordinal0 [0x0023B71F+964383]
GetHandleVerifier [0x0061E7E2+2743074]
GetHandleVerifier [0x006108D4+2685972]
GetHandleVerifier [0x00402BAA+532202]
GetHandleVerifier [0x00401990+527568]
Ordinal0 [0x0031080C+1837068]
Ordinal0 [0x00314CD8+1854680]
Ordinal0 [0x00314DC5+1854917]
Ordinal0 [0x0031ED64+1895780]
BaseThreadInitThunk [0x75B4FA29+25]
RtlGetAppContainerNamedObjectPath [0x779A7BBE+286]
RtlGetAppContainerNamedObjectPath [0x779A7B8E+238]
If someone could help me, I would be very grateful.

You're trying to click on a button located in an iframe. You need to switch to that iframe first, then locate the button, then click on it:
WebDriverWait(browser, 5).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, '//iframe[#class="detail-media-viewer"]')))
t.sleep(5) ## let elements within iframe load properly
[.. do your stuff locating/clicking whatever buttons are there]
If you plan on interacting /locating again some elements from the main page(outside iframe) you need to switch back to main page:
browser.switch_to.default_content()
Selenium documentation can be found here: https://www.selenium.dev/documentation/

how to print a PDF document using selenium python?

I need to print a PDF file after having "downloaded" it using a selenium script. The issue is that it opens in a pop up window, however selenium should be able to handle that.
This is the source code of the toolbar i want to interact with:
<div id="toolbar"><div id="start"><cr-icon-button id="sidenavToggle" iron-icon="cr20:menu" title="Menu" aria-label="Menu" aria-expanded="true" aria-disabled="false" role="button" tabindex="0"></cr-icon-button><span id="title">rdlComprobanteJubilados</span></div><div id="center"><viewer-page-selector style="--page-length-digits:1;"></viewer-page-selector><span class="vertical-separator"></span><span id="zoom-controls"><cr-icon-button iron-icon="pdf:remove" title="Zoom out" aria-label="Zoom out" aria-disabled="false" role="button" tabindex="0"></cr-icon-button><input type="text" value="100%" aria-label="Zoom level"><cr-icon-button iron-icon="pdf:add" title="Zoom in" aria-label="Zoom in" aria-disabled="false" role="button" tabindex="0"></cr-icon-button></span><span class="vertical-separator"></span><cr-icon-button id="fit" aria-label="Fit to page" title="Fit to page" aria-disabled="false" role="button" tabindex="0" iron-icon="pdf:fit-to-height"></cr-icon-button><cr-icon-button iron-icon="pdf:rotate-left" dir="ltr" aria-label="Rotate counterclockwise" title="Rotate counterclockwise" aria-disabled="false" role="button" tabindex="0"></cr-icon-button></div><div id="end"><viewer-download-controls id="downloads"></viewer-download-controls><cr-icon-button id="print" iron-icon="cr:print" title="Print" aria-label="Print" aria-disabled="false" role="button" tabindex="0"></cr-icon-button><cr-icon-button id="more" iron-icon="cr:more-vert" title="More actions" aria-label="More actions" aria-disabled="false" role="button" tabindex="0"></cr-icon-button></div></div>
although the full source code is here.
I want to interact with this element:
<cr-icon-button id="print" iron-icon="cr:print" title="Print" aria-label="Print" aria-disabled="false" role="button" tabindex="0"></cr-icon-button>
my current approach is this:
for handle in driver.window_handles:
if driver.current_window_handle!=handle:
driver.switch_to.window(handle)
print("changed")
pbutton=filebutton=driver.find_element(By.ID,"print")
pbutton.click()
Select(WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="print"]')))).select_by_index(2)
print("pass")
which yields this error:
NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="print"]"}
(Session info: chrome=104.0.5112.81)
Stacktrace:
Backtrace:
Ordinal0 [0x00E35FD3+2187219]
Ordinal0 [0x00DCE6D1+1763025]
Ordinal0 [0x00CE3E78+802424]
Ordinal0 [0x00D11C10+990224]
Ordinal0 [0x00D11EAB+990891]
Ordinal0 [0x00D3EC92+1174674]
Ordinal0 [0x00D2CBD4+1100756]
Ordinal0 [0x00D3CFC2+1167298]
Ordinal0 [0x00D2C9A6+1100198]
Ordinal0 [0x00D06F80+946048]
Ordinal0 [0x00D07E76+949878]
GetHandleVerifier [0x010D90C2+2721218]
GetHandleVerifier [0x010CAAF0+2662384]
GetHandleVerifier [0x00EC137A+526458]
GetHandleVerifier [0x00EC0416+522518]
Ordinal0 [0x00DD4EAB+1789611]
Ordinal0 [0x00DD97A8+1808296]
Ordinal0 [0x00DD9895+1808533]
Ordinal0 [0x00DE26C1+1844929]
BaseThreadInitThunk [0x75906739+25]
RtlGetFullPathName_UEx [0x770390AF+1215]
RtlGetFullPathName_UEx [0x7703907D+1165]
i´m running Python 3.9.7 and Selenium 4.3.0 and i work with Jupyter Notebooks installed with Anaconda and the browser used is Google Chrome Versión 104.0.5112.81 (Build oficial) (64 bits)
I need the solution to work as a standalone script and preferably OS agnostic although if that´s not possible it would be enough to have it run in Linux Mint

To click on the desired element you can use either of the following locator strategies:
Using CSS_SELECTOR:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "cr-icon-button#print[title='Print'][aria-label='Print']"))).click()
Using XPATH:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//cr-icon-button[#id='print' and #title='Print'][#aria-label='Print']"))).click()
Note: You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC

I finally found a way to solve it WITHOUT interacting with the PDF viewer. i used the option module:
from selenium.webdriver.chrome.options import Options
DRIVER_PATH = r"C:\Users\HP\Downloads\chromedriver_win32\chromedriver.exe"
chrome_options = Options()
chrome_options.add_experimental_option('prefs', {
"download.default_directory": r"C:\Users\HP\Desktop\test downloads", #Change default directory for downloads
"download.prompt_for_download": False, #To auto download the file
"download.directory_upgrade": True,
"plugins.always_open_pdf_externally": True #It will not show PDF directly in chrome
})
driver = webdriver.Chrome(executable_path=DRIVER_PATH,options = chrome_options)
driver.implicitly_wait(60)

Python Selenium - How to select this button and click it

I've been trying to get this button selected and click it in order to add a trading view strategy to a chart. I've automated everything up to this point including the locating and clicking of similar buttons. This one seems a bit finicky. Here is the code that I've already implemented that works (except for the 'Add To Chart' button)
WHAT WORKS ON OTHER BUTTONS
# Select the strategy
select_strat = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CLASS_NAME, 'title-phaedJZ1')))
select_strat.click()
# Close the strategy window
close_strat_window = WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CLASS_NAME, 'close-tuOy5zvD')))
close_strat_window.click()
WHAT I'M TRYING TO LOCATE AND CLICK
WHAT I'VE ALREADY TRIED IMPLEMENTING
# Add strat to chart
# button-TuYnJRjv rightControlsBlock__button-TuYnJRjv button-9pA37sIi isInteractive-9pA37sIi newStyles-9pA37sIi ace_layer ace_cursor-layer ace_hidden-cursors
add_to_chart = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, '//*[#id="tv-script-pine-editor-header-root"]/div[1]/div/div[2]/div[3]')))
add_to_chart.click()
Note - I've tried every combination of the comment of the class below for CLASS_NAME
# Add strat to chart
# button-TuYnJRjv rightControlsBlock__button-TuYnJRjv button-9pA37sIi isInteractive-9pA37sIi newStyles-9pA37sIi ace_layer ace_cursor-layer ace_hidden-cursors
add_to_chart = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'button-TuYnJRjv')))
add_to_chart.click()
REQUEST FROM COMMENTS FOR STACKTRACE
pipenv run python .\dev\tv_data_collector.py
Loading .env environment variables...
C:\Users\REDACTED\Documents\Trading\Automation\dev\tv_data_collector.py:20: DeprecationWarning: executable_path has been deprecated, please pass in a Service object
driver = webdriver.Chrome(executable_path=path, chrome_options=options)
C:\Users\tgall\Documents\Trading\Automation\dev\tv_data_collector.py:20: DeprecationWarning: use options instead of chrome_options
driver = webdriver.Chrome(executable_path=path, chrome_options=options)
DevTools listening on ws://127.0.0.1:65270/devtools/browser/93062f9b-ac9d-465e-9471-df4c5098e655
[26756:18156:0817/175454.408:ERROR:device_event_log_impl.cc(214)] [17:54:54.409] USB: usb_device_handle_win.cc:1048 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
Traceback (most recent call last):
File "C:\Users\REDACTED\Documents\Trading\Automation\dev\tv_data_collector.py", line 126, in <module>
tv_get_security(driver)
File "C:\Users\REDACTED\Documents\Trading\Automation\dev\tv_data_collector.py", line 73, in tv_get_security
script_search = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME, 'input-CcsqUMct')))
File "C:\Users\REDACTED\.virtualenvs\Automation--luj7l49\lib\site-packages\selenium\webdriver\support\wait.py", line 90, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
Ordinal0 [0x006E78B3+2193587]
Ordinal0 [0x00680681+1771137]
Ordinal0 [0x005941A8+803240]
Ordinal0 [0x005C24A0+992416]
Ordinal0 [0x005C273B+993083]
Ordinal0 [0x005EF7C2+1177538]
Ordinal0 [0x005DD7F4+1103860]
Ordinal0 [0x005EDAE2+1170146]
Ordinal0 [0x005DD5C6+1103302]
Ordinal0 [0x005B77E0+948192]
Ordinal0 [0x005B86E6+952038]
GetHandleVerifier [0x00990CB2+2738370]
GetHandleVerifier [0x009821B8+2678216]
GetHandleVerifier [0x007717AA+512954]
GetHandleVerifier [0x00770856+509030]
Ordinal0 [0x0068743B+1799227]
Ordinal0 [0x0068BB68+1817448]
Ordinal0 [0x0068BC55+1817685]
Ordinal0 [0x00695230+1856048]
BaseThreadInitThunk [0x76466739+25]
RtlGetFullPathName_UEx [0x778890AF+1215]
RtlGetFullPathName_UEx [0x7788907D+1165]
This error has not prevented anything from working going forward (until now perhaps.. not sure)

If you are clicking on the button then you should use element to be clickable instead of presece of element like below:
add_to_chart = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CLASS_NAME, 'button-TuYnJRjv')))
add_to_chart.click()
OR
add_to_chart = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[normalize-space()='Add to chart']")))
add_to_chart.click()
OR
add_to_chart = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//div[contains(text(),'Add to chart')]")))
add_to_chart.click()
For DeprecationWarning the Selenium v4 compatible Code Block
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.get("https://www.google.com")
For more info visit this

in your picture, div.js-button-text.text-9pA37sli is CSS SELECTOR
so
EC.presence_of_element_located((By.CLASS_NAME, 'button-TuYnJRjv'))
into
EC.presence_of_element_located((By.CSS_SELECTOR, 'div.js-button-text.text-9pA37sli'))

Error finding element with selenium python of transfer market website?(tried with xpath,class name etc) but nothing works

Hi Dear Programmers and webscrapers.I have tried so much times to locate the element and then press it but it always gives me error in selenium.I have treid using XPATH,class name and id of the element is not present so I tried with multiple things but nothing works and returns me an error.
This is the website link that I want to scrap.
Website to scrap
Here is the element that I wanted to scrap.
This element had class selector-title so I tried with class name and also with xpath of this element and with its sub elements also but unfortunately was not able to locate it.
Here is the code that i used.This code is working for other functions like I have located search box and have searched pressed button it works.Right.Where it dosent work is the element I have been trying it to work.Please let me know what mistake is from mine side so this can be workable or I can locate it and then click on it.
Actually this I have to do for web scraping the data of all the players in a club or in a league so for that I have to select the league of country first then I will be able to scrap the data so its very much important for me to actually be able to locate and click the marked element.Please help me if any one of you can.Thanks.
XPATH of the marked element
//*[#id="main"]/header/div[3]/tm-quick-select-bar//div/tm-quick-select[1]/div/div
Here is my code
import os
from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
os.environ['PATH'] += r"C:/ChromeDriver"
browser = webdriver.Chrome()
browser.get("https://www.transfermarkt.us/")
element = browser.find_element(By.XPATH, '//*[#id="schnellsuche"]/input[1]')
element.send_keys("J1 League")
time.sleep(5)
# element2 = browser.find_element(By.XPATH, '//*[#id="main"]/header/div[3]/tm-quick-select-bar//div/tm-quick-select[1]/div/div[1]/strong/text()')
# element2.click()
button = wait(browser, 10).until(EC.presence_of_element_located(
(By.CLASS_NAME, 'selector-title')))
button.click()
time.sleep(10)
The error I get is the browser instantly closes as soon as it reaches the last element target element to click
DevTools listening on ws://127.0.0.1:61330/devtools/browser/d6f23f8f-6bea-41dd-8500-08f85c9cb502
[18424:9224:0523/200949.930:ERROR:device_event_log_impl.cc(214)] [20:09:49.929] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[18424:9224:0523/200949.936:ERROR:device_event_log_impl.cc(214)] [20:09:49.935] USB: usb_device_handle_win.cc:1049 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
[18424:9224:0523/200949.951:ERROR:device_event_log_impl.cc(214)] [20:09:49.951] Bluetooth: bluetooth_adapter_winrt.cc:1075 Getting Default Adapter failed.
Traceback (most recent call last):
File "d:\WeBScraping\FreecodecampWebScrapingTutorial\selenium_tutorial_freecodecamp\scrapingtarget.py", line 18, in <module>
button = wait(browser, 10).until(EC.presence_of_element_located(
File "C:\Users\bilal\AppData\Roaming\Python\Python310\site-packages\selenium\webdriver\support\wait.py", line 89, in until
raise TimeoutException(message, screen, stacktrace)
selenium.common.exceptions.TimeoutException: Message:
Stacktrace:
Backtrace:
Ordinal0 [0x00E1B8F3+2406643]
Ordinal0 [0x00DAAF31+1945393]
Ordinal0 [0x00C9C748+837448]
Ordinal0 [0x00CC92E0+1020640]
Ordinal0 [0x00CC957B+1021307]
Ordinal0 [0x00CF6372+1205106]
Ordinal0 [0x00CE42C4+1131204]
Ordinal0 [0x00CF4682+1197698]
Ordinal0 [0x00CE4096+1130646]
Ordinal0 [0x00CBE636+976438]
Ordinal0 [0x00CBF546+980294]
GetHandleVerifier [0x01089612+2498066]
GetHandleVerifier [0x0107C920+2445600]
GetHandleVerifier [0x00EB4F2A+579370]
GetHandleVerifier [0x00EB3D36+574774]
Ordinal0 [0x00DB1C0B+1973259]
Ordinal0 [0x00DB6688+1992328]
Ordinal0 [0x00DB6775+1992565]
Ordinal0 [0x00DBF8D1+2029777]
BaseThreadInitThunk [0x76CDFA29+25]
RtlGetAppContainerNamedObjectPath [0x76F57A7E+286]
RtlGetAppContainerNamedObjectPath [0x76F57A4E+238]

Not sure if I understood you correctly, but the code below return the following text "United States".
import time
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
driver = webdriver.Chrome(options=options, desired_capabilities=capabilities)
# open url:
driver.get('https://www.transfermarkt.us/')
# switch to cookies frame
wait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "sp_message_iframe_575847")))
time.sleep(3)
# click "Accept" cookies button
driver.find_element(By.XPATH, '//button[#title="ACCEPT ALL"]').click()
time.sleep(3)
# go back to main frame
driver.switch_to.default_content()
# here's the trick, what you are looking for is inside a "shadow-root" DOM so to access it you need to execute the script and then use CSS selector, I don't think XPATH works here:
country = driver.execute_script("return document.querySelector('tm-quick-select-bar').shadowRoot.querySelector('div > tm-quick-select:nth-child(2) > div > div > strong')")
# print
print(country.text)
# close the driver
driver.close()

i got an error while running a python script using the code generated by selenium ide browser extension and i have no clue how to resolve this issue

this is the error message.
it appeared in a red coloured background box.
i ran it in visual studio code.
i have no clue what to do and i have no understanding what this error message means.
the goal was to run the automation software to automate some actions in chrome.
i have the correct chromedriver version.
Exception has occurred: NoSuchElementException
Message: no such element: Unable to locate element: {"method":"css selector","selector":"#mat-option-165 .pr-2:nth-child(1)"}
(Session info: chrome=101.0.4951.67)
Stacktrace:
Backtrace:
Ordinal0 [0x00E8B8F3+2406643]
Ordinal0 [0x00E1AF31+1945393]
Ordinal0 [0x00D0C748+837448]
Ordinal0 [0x00D392E0+1020640]
Ordinal0 [0x00D3957B+1021307]
Ordinal0 [0x00D66372+1205106]
Ordinal0 [0x00D542C4+1131204]
Ordinal0 [0x00D64682+1197698]
Ordinal0 [0x00D54096+1130646]
Ordinal0 [0x00D2E636+976438]
Ordinal0 [0x00D2F546+980294]
GetHandleVerifier [0x010F9612+2498066]
GetHandleVerifier [0x010EC920+2445600]
GetHandleVerifier [0x00F24F2A+579370]
GetHandleVerifier [0x00F23D36+574774]
Ordinal0 [0x00E21C0B+1973259]
Ordinal0 [0x00E26688+1992328]
Ordinal0 [0x00E26775+1992565]
Ordinal0 [0x00E2F8D1+2029777]
BaseThreadInitThunk [0x761BFA29+25]
RtlGetAppContainerNamedObjectPath [0x77A47A7E+286]
RtlGetAppContainerNamedObjectPath [0x77A47A4E+238]
File "C:\Users\Public\Untitled-2.py", line 38, in <module>
driver.find_element(By.CSS_SELECTOR, "#mat-option-165 .pr-2:nth-child(1)").click()
here is the code that led up to the error message.
the code between the "#####" and "###" are copied from the file generated by the selenium ide.
the last line of the code is what causes the error message.
#####
import pytest
import time
import json
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support import expected_conditions
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities
###
from selenium import webdriver
driver = webdriver.Chrome(executable_path = r"C:\chromedriver\chromedriver.exe")
driver.get("https://www.travelstart.co.za")
#####
driver.get("https://www.travelstart.co.za/")
driver.set_window_size(1552, 840)
driver.find_element(By.ID, "dept_city0").click()
driver.find_element(By.ID, "dept_city0").send_keys("cpt")
time.sleep(1)
driver.find_element(By.CSS_SELECTOR, ".sub_txt:nth-child(6)").click()
driver.find_element(By.ID, "arr_city0").click()
driver.find_element(By.ID, "arr_city0").send_keys("jnb")
time.sleep(1)
driver.find_element(By.CSS_SELECTOR, "#mat-option-165 .pr-2:nth-child(1)").click()
###
i am using python 3.10.4