I have been trying to access the widget data from https://www.dukascopy.com/trading-tools/widgets/quotes/historical_data_feed so that my program can scrape the website and select the search bar to enter the data and return the result in the iframe. However, even when I convert my driver to the iframe it still can't find the iframe data elements. the error:
I tried to add the time delay for everything to load but still no luck.
elenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"d-e-Xg"}
There are actually two nested iframes in play here. That, combined with some WebDriverWaits allowed me to arrive at the following solution. Here I locate the "Instrument" field and input text into it:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
options = Options()
options.add_argument("--headless")
driver = webdriver.Chrome(
options=options, executable_path=r"C:\chromedriver\chromedriver.exe"
)
driver.implicitly_wait(10)
try:
driver.get(
"https://www.dukascopy.com/trading-tools/widgets/quotes/historical_data_feed"
)
driver.switch_to.frame(driver.find_element(By.ID, "widget-container"))
driver.switch_to.frame(driver.find_element(By.TAG_NAME, "iframe"))
# Wait for "Loading" overlay to disappear
WebDriverWait(driver, 10).until(
ec.invisibility_of_element_located((By.CLASS_NAME, "d-e-r-k-n"))
)
# Click "Instrument"
driver.find_element(By.CLASS_NAME, "d-oh-i-ph-qh-rh-m").click()
# Enter text into now-visible instrument field
driver.find_element(By.CLASS_NAME, "d-e-Xg").send_keys("metlife")
finally:
driver.quit()
Related
I'm using selenium to try and scrape a listing of products in this website:
https://www.zonacriativa.com.br/harry-potter
However, I'm having trouble getting the full listing of products. the page list 116 products, yet only a few are shown at a time. If I want to see the other ones, I need to click on the "Carregar mais Produtos" (load more products) button at the bottom a few times to get the full listing.
I'm having trouble locating this button, as it doesn't have an id and its class is a huge string. I've tried several things, like the examples below, but they don't seem to work. Any suggestions?
driver.find_element("xpath", "//button[text()='Carregar mais Produtos']").click()
driver.find_element("css selector", ".vtex-button__label.flex.items-center.justify-center.h-100.ph5").click()
driver.find_element(By.CLASS_NAME, "vtex-button.bw1.ba.fw5.v-mid.relative.pa0.lh-solid.br2.min-h-small.t-action--small.bg-action-primary.b--action-primary.c-on-action-primary.hover-bg-action-primary.hover-b--action-primary.hover-c-on-action-primary.pointer").click()
The element you trying to click is initially out of the visible screen so you can't click it. Also this XPath at least for me doesn't locate that element.
What you need to do is to scroll the page down untill that button becomes visible and clickable and then click it.
The following code clicks that button 1 time:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://www.zonacriativa.com.br/harry-potter"
driver.get(url)
while True:
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"))).click()
break
except:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
The above code can be simply modified to scroll and click that button until we reach the latest page where this button is not presented:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 5)
url = "https://www.zonacriativa.com.br/harry-potter"
driver.get(url)
while driver.find_elements(By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"):
try:
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[contains(#class,'buttonShowMore')]//button"))).click()
except:
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
I have tried the below code and it is always timing out. But when I look it up with the browser inspector, I can see the Username input element.
I have also tried to find it by ID but not able to.
Tried almost all of the existing questions/solutions on this forum but was unable to figure it out.
driver.get("https://www.dat.com/login")
time.sleep(5)
driver.find_element_by_css_selector("a[href*='https://power.dat.com/']").click()
time.sleep(5)
# explicitly waiting until input element load
try:
WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.NAME, "username"))
)
except TimeoutException:
print(
"Waited 10 seconds for the username input to load but did not happen..."
)
except Exception as e:
print(f"Exception while waiting for username input to appear: \n {e}")
sys.exit(1)
2 issues here:
After clicking on "DAT Power" on the first page a new tab is opened. To continue working there you need to switch the driver to the second tab.
You will probably want to enter a text into the username there. If so you need to wait for element clickability, not presence only.
Also, the current Selenium version (4.x) no more supports find_element_by_* methods, the new style should be used, as following presented.
The following code works:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.dat.com/login"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='https://power.dat.com/']"))).click()
new_tab = driver.window_handles[1]
driver.switch_to.window(new_tab)
wait.until(EC.element_to_be_clickable((By.NAME, "username"))).send_keys("name#gmail.com")
The result is
It's my first time using Selenium and webscraping. I have been stuck with the annoying GDPR iframe. I am simply trying to go to a website, search something in the search bar and then click in one of the results. But it does not seem to do anything after I get rid of the GDPR consent.
Important, it does not give any errors.
This is my very simple code:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import time
#Web driver
driver = webdriver.Chrome(executable_path="C:\Program Files (x86)\chromedriver.exe")
driver.get("https://transfermarkt.co.uk/")
search = driver.find_element_by_name("query")
search.send_keys("Sevilla FC")
search.send_keys(Keys.RETURN)
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it((By.ID, "sp_message_iframe_382445")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//button[text()='ACCEPT ALL']"))).click()
try:
sevfclink = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.ID, "368")))
sevfclink.click()
except:
driver.quit()
time.sleep(5)
driver.quit()
Not sure where you get the iframe from but the id might be dynamic so try this.
driver.get("https://transfermarkt.co.uk/")
wait = WebDriverWait(driver,10)
search = wait.until(EC.element_to_be_clickable((By.NAME, "query")))
search.send_keys("Sevilla FC", Keys.RETURN)
wait.until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[id^='sp_message_iframe']")))
wait.until(EC.element_to_be_clickable((By.XPATH, "//button[text()='ACCEPT ALL']"))).click()
driver.switch_to.default_content()
try:
sevfclink = wait.until(EC.element_to_be_clickable((By.ID, "368")))
sevfclink.click()
except:
pass
It looks like the two lines starting WebDriverWait throw an error. If I skip those to the try statement you get the results of the search. A page that gives an overview of Sevilla FC shows up. I presume the WebDriverWait lines are there to make sure you wait for something, but from what I can tell they are unnecessary.
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
url = 'https://www.msha.gov/mine-data-retrieval-system'
driver = webdriver.Chrome(executable_path='chromedriver')
driver.get(url)
#driver.find_element_by_xpath('//*[#id="mstr90"]/div[1]/div/div') error
#driver.find_elements_by_xpath('//input') gives 3 while in driver gives 10
I am unable to find element where the input "Search by Mine ID by typing here.." is, the document is fully loaded but it can't locate it. What I want to do is simply pass in an input "0100003" then submit
Iframe is present on your page. Before you interact with inputbox you need to switch on to iframee. Refer below code to resolve your issue.
wait = WebDriverWait(driver, 10)
driver.get("https://www.msha.gov/mine-data-retrieval-system")
driver.switch_to.frame("iframe1")
wait = WebDriverWait(driver, 10)
inputBox = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-empty']"))).click()
inputBox1 = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='mstrmojo-SimpleObjectInputBox-container mstrmojo-scrollNode']//input")))
inputBox1.send_keys("0100003")
Updated Code to handle dropdown
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div#mstr100,mstrmojo-Popup.mstrmojo.SearchBoxSelector-suggest"))).click()
Note: please add below imports to your solution
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
Output:
The element you are trying to find is inside an iframe, so you will need to switch to that iframe first and then do your find element. Also, it's a best practice to use waits to give pages/elements time to load before a find element timeouts and throws an error.
iframe = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#iframe1')))
driver.switch_to.frame(iframe)
mine_id = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.XPATH, '//*[#id="mstr90"]/div[1]/div/div')))
Then you need to click this element to make it interactable.
mine_id.click()
Once you click then you need to re-find the input box before sending keys.
mine_id_input = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr90 input')))
mine_id_input.send_keys('0100003')
To select the suggestion displayed:
suggestion = WebDriverWait(driver, 15).until(EC.presence_of_element_located((By.CSS_SELECTOR, '#mstr100')))
suggestion.click()
if you wanted to continue on interacting outside the iframe after this is done, you will want to switch back out of the iframe like this:
driver.switch_to.default_content()
I am trying to scrape some data from an iframe located within a webpage. The URL of the webpage is https://www.nissanoflithiasprings.com/schedule-service. I am trying to access the button shown in the image below:
When I right-click on the button (located inside the iframe) to view the source code, I am able to see the HTML id and name (see screenshot below):
The "id" for the button is "new_customer_button". However, when I use selenium webdriver's driver.find_element_by_id("new_customer_button") to access the button, the code is not able to locate the button inside the iframe and throws the following error:
NoSuchElementException: no such element: Unable to locate element: {"method":"id","selector":"new_customer_button"}
Below is the code that I have tried so far:
from selenium import webdriver
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
dest_iframe = driver.find_elements_by_tag_name('iframe')[0]
driver.switch_to.frame(dest_iframe)
driver.find_element_by_id("new_customer_button")
Not sure why this is happening. Any help will be appreciated. Thanks!
The element is inside multiple <iframe> tags, you need to switch to them one by one. You should also maximize the window and use explicit wait as it take some time to load
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
chrome_path = r"C:\Users\gh455\Downloads\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.maximize_window()
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
wait = WebDriverWait(driver, 10)
# first frame - by css selector
wait.until(ec.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR, '[src^="https://consumer.xtime.com"]')))
# second frame - by ID
wait.until(ec.frame_to_be_available_and_switch_to_it('xt01'))
driver.find_element_by_id("new_customer_button")
To click() on the element with text as Make · Year · Model as the the desired element is within an nested <iframe>s so you have to:
Induce WebDriverWait for the desired parent frame to be available and switch to it.
Induce WebDriverWait for the desired child frame to be available and switch to it.
Induce WebDriverWait for the desired element_to_be_clickable().
You can use the following solution:
Code Block:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument("start-maximized")
chrome_options.add_argument('disable-infobars')
driver = webdriver.Chrome(options=chrome_options, executable_path=r'C:\WebDrivers\chromedriver.exe')
driver.get("https://www.nissanoflithiasprings.com/schedule-service")
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='com/scheduling']")))
WebDriverWait(driver, 10).until(EC.frame_to_be_available_and_switch_to_it((By.CSS_SELECTOR,"iframe[src*='consumerportal']")))
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button.button.button--action.btn.btn-secondary#new_customer_button"))).click()
Browser Snapshot:
Here you can find a relevant discussion on Ways to deal with #document under iframe