I'm new to selenium and trying to automate the download of some government data. When using the code below. I manage to navigate to the right page and enter the right parmeter in the form, but then can't find a way to click the 'submit' button. I've tried find_element_by_partial_link_text("Subm").click() and I've tried find_element_by_class_name on a number of class names. Nothing works. Any ideas?
import time
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
main_url="http://data.stats.gov.cn/english/easyquery.htm?cn=E0101"
driver = webdriver.Firefox()
driver.get(main_url)
time.sleep(8)
driver.find_element_by_partial_link_text("Industry").click()
time.sleep(8)
driver.find_element_by_partial_link_text("Main Economic Indicat").click()
time.sleep(8)
driver.find_element_by_id("mySelect_sj").click()
time.sleep(3)
driver.find_element_by_class_name("dtText").send_keys("last72")
time.sleep(4)
try:
driver.find_element_by_class_name("dtFoot").click()
except:
driver.find_element_by_class_name("dtFoot").submit()
Solved my own problem, the key was using
driver.find_element_by_class_name(`dtTextBtn`)
instead of
driver.find_element_by_class_name(`dtTextBtn f10`)
The latter was what I saw in the source code, but the f10 blocked selenium.
Related
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.support.ui import Select
from selenium.webdriver.chrome import options
import unittest
import time
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
#link to website
website = 'https://www.ncei.noaa.gov/access/monitoring/climate-at-a-glance'
path = ('../chromedriver') #Folder location where is the chromedriver
driver = webdriver.Chrome(path)
driver.get(website)
driver.maximize_window()
#Selection of the sections where the information I am looking for is located
state_click = driver.find_element(by=By.XPATH, value='//*[#id="show-statewide"]/a').click()
time_series_click = driver.find_element(by=By.XPATH, value='.//*[#id="time-series"]/div[3]/button').click()
#selection of the years (for all files the same range of 1950 - 2021)
star_year_dropdown =Select(driver.find_element(by=By.ID, value='begyear'))
star_year_dropdown.select_by_visible_text('1950')
end_year_dropdown = Select(driver.find_element(by=By.ID, value='endyear'))
end_year_dropdown.select_by_visible_text('2021')
#selection of the parameter to download: Average temperature
parameter_dropdown = Select(driver.find_element(by=By.ID, value='parameter'))
parameter_dropdown.select_by_visible_text('Average Temperature')
#Creating a loop to loop through all the states and all the months:
#state selection
select_state = driver.find_element(by=By.XPATH, value='.//*[#id="state"]')
opcion_state = select_state.find_elements(by=By.TAG_NAME, value='option')
#month selection
select_month = driver.find_element(by=By.XPATH, value = '//*[#id="month"]')
opcion_month = select_month.find_elements(by = By.TAG_NAME, value='option')
for option in opcion_month:
option.click()
for option in opcion_state:
option.click()
time.sleep(3)
plot = driver.find_element(by=By.XPATH, value='.//input[#id="submit"]').click()
dowload = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, '//*[#id="csv-download"]'))).click()
time.sleep(3)
The code works fine, but in the plot and download functions created (at the end of the whole) when trying to click it gives an error and cannot be solved. I think it is because the web page at the time of executing that command does not display the button in the screen to plot the graph and download the csv. I have tried modifying the waiting time, but not work. Let's see if someone can help me.
Thanks in advance!!
So the problem that you are having with downloading the CSV file is not that the command is wrong (it does work), however, the issue is that the download CSV button is not visible on the page, which prevents you from clicking it.
A way to get around having to visibly see the element on the page and still click it you can do the following:
driver.execute_script("arguments[0].click();", driver.find_element(By.XPATH, '//*[#id="csv-download"]'))
This would be the preferred method, otherwise you would have to scroll to where the button is visible on the page and then click the button. The page has a scrolling effect if trying to click on a button which you can do the following (but the previous method is preferred as it is cleaner and does not take any additional time to do):
from selenium.common.exceptions import ElementClickInterceptedException
download_btn = driver.find_element(By.XPATH, '//*[#id="csv-download"]')
try:
download_btn.click() # the first time you try to click it will throw an error, while navigating the button into the page view
except ElementClickInterceptedException: # catch the exception and proceed as if clicking again
download_btn.click()
Other comments on your code
you have option.click() twice in your code, and it is unclear if you want to click the month or state option - there may be a confusion about which options are clicked. I would suggest renaming your iterator variables appropriately so that you know which buttons are being clicked
I have been working on this problem for quite a bit now, and can't figure out why this is happening. I am trying to click a button. The button and the corresponding text always changes, but the XPATH stays the same.
Button I am trying to click
It doesn't work with CSS Selector either. I am using Chrome web version 106. Does anyone know why?
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.common.exceptions import NoSuchElementException
PATH = r"C:\Users\###\Downloads\chromedriver.exe"
driver = webdriver.Chrome(PATH)
# time to login
driver.get("https://clever.com")
time.sleep(60)
try:
driver.find_element(By.XPATH, '//*[#id="root"]/div/div/div[2]/div[3]/div/div/div/div[2]/button[2]').click()
except NoSuchElementException:
print("no such element")
pass
time.sleep(5)
Capturing the XPATH for the element
Try this:
driver.find_element("xpath", '//*[#id="root"]/div/div/div[2]/div[3]/div/div/div/div[2]/button[2]').click()
May be you using not correct syntax
I am trying to log on to Target's website using Selenium in Python with the Chrome WebDriver..
When I am prompted to log in, I use the following code:
self.browser.find_element_by_name("password").send_keys(pw)
self.browser.find_element_by_id("login").submit()
After the field is submitted, I am presented with this error in the DOM:
DOM Error Alert
..and this in the console:
401 Error
Note:
I have tried logging in with Selenium on Instagram, and it works.. So I know it has something to do with the structure of Target's website. Has anyone run into this issue before?
Thanks!
So I originally tried solving this issue using Chrome, but could not figure out why the page would not precede after entering the login data. I thought maybe the page was protected by some bot software, but could not find any proof.
I decided to try Safari on my MAC, and actually had success. See the below code:
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
driver = webdriver.Safari(executable_path='/usr/bin/safaridriver')
driver.get('https://www.target.com/')
action = ActionChains(driver)
driver.find_element(By.XPATH, '//*[#id="account"]').click()
WebDriverWait(driver, 30).until(ec.presence_of_element_located((By.XPATH, '//*[#id="accountNav-signIn"]')))
action.send_keys(Keys.ENTER)
action.perform()
WebDriverWait(driver, 10).until(ec.presence_of_element_located((By.XPATH, '//h2[#class="sc-hMqMXs sc-esjQYD eXTUDl"]')))
driver.find_element(By.ID, 'username').click()
driver.find_element(By.ID, 'username').send_keys('foo')
time.sleep(5)
driver.find_element(By.ID, 'password').click()
driver.find_element(By.ID, 'password').send_keys('bar')
time.sleep(5)
driver.find_element(By.XPATH, "//button[#id=\'login\']").send_keys(Keys.ENTER)
time.sleep(10)
driver.quit()
You will notice some time.sleeps which I am using to slow the program down (you can take these out).
I also tried on FireFox and Edge, but had the same problems as Chrome.
Conclusion, it seems there could be some sort of bot protection which is blocking you from using Chrome (also Edge and FireFox). Given these webdrivers are being detected as automated. Safari (I believe) does not get detected as such.
I would also suggest reading through the below post, it may offer more insight.
Website navigates to no-access page using ChromeDriver and Chrome through Selenium probably Bot Protected
I am trying to login to a website using python so that I can get some of their text from the website.
Here is my code. There always an error at the end of the code after the id and password code.
import os
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
driver = webdriver.Chrome(executable_path=r"C:\chromedriver\chromedriver.exe")
driver.get('https://www.saramin.co.kr/zf_user/auth')
driver.implicitly_wait(3)
driver.find_element_by_name('id').send_keys('<<my_id>>')
driver.find_element_by_name('password').send_keys('<<my_password>>')
driver.find_element_by_xpath('//*[#id="frmNIDLogin"]/fieldset/input').click()
HTML source of the button:
Eventually I figured it out! Thanks for your answer though.
Here is the final code.
driver = webdriver.Chrome(executable_path="C:\chromedriver\chromedriver.exe")
browser = webdriver.Chrome('C:\chromedriver\chromedriver.exe')
driver.get('https://www.saramin.co.kr/zf_user/auth')
driver.implicitly_wait(3)
driver.find_element_by_name('id').send_keys('ID') driver.find_element_by_name('password').send_keys('PW')
driver.find_element_by_xpath( '//*[#class="btn-login"]' ).click()
The xpath is incorrect, it doesn't match anything in the page. Try
driver.find_element_by_xpath('//form[#id="login_frm"]//button[#class="btn-login"]').click()
or simply use submit() function on the <form>
form = driver.find_element_by_id('login_frm')
form.submit()
In the first case you were using 'id' ('//*[#id="frmNIDLogin"]) for click button, because 'id' changes every time page loads it was giving error. But in the second case when you used class ( '//*[#class="btn-login"]' ) it worked because it remains same every time page is loaded. Also as mentioned above the value of id in first case was wrong.
All I am trying to do is: go to "http://news.google.com" and click on the "Technology" link on the side menu. Here's my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support.ui import Select
from selenium.common.exceptions import NoSuchElementException
from selenium.common.exceptions import NoAlertPresentException
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
import unittest, time, re
class GoogleTech(unittest.TestCase):
def setUp(self):
self.driver = webdriver.Firefox()
self.driver.implicitly_wait(30)
self.base_url = "https://www.google.com/"
self.verificationErrors = []
self.accept_next_alert = True
def test_googletech(self):
driver = self.driver
driver.get("http://news.google.com")
#driver.find_element_by_xpath("//div[#id='main-wrapper']/div[#id='main-pane']/div[#class='background']/div[#class='centered']/div[#id='nav-menu-wrapper']/div[#class='browse-sidebar']/ul[#id='anchorman-two-browse-nav']/li[#class='nav-item nv-en_us:tc']/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("//ul[#id='anchorman-two-browse-nav']/li[#class='nav-item nv-en_us:tc']/a[contains(text(),'Technology')]").click()
#driver.find_element_by_xpath("html/body/div[3]/div[1]/div/div/div[2]/div/ul/li[6]/a").click()
#driver.find_element_by_link_text("Technology").click()
wait = WebDriverWait(driver, 10)
link = wait.until(EC.presence_of_element_located((By.XPATH, "//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]")))
link.click()
time.sleep(10)
...
I've tried many combinations (all the commented out lines plus more!), but still cannot get it to work. I've check to make sure that the element I am using does exists (using Element Inspector) before entering it into the code. But even the "find_element_by_link_text" is giving me a NoSuchElementException exception. Can someone please tell me what I am doing wrong?
UPDATE: After doing further tests, I now have a better understanding of when this error is occuring for the news.google.com page. So, basically, the side menu, that contains the "Technology" link, is set to scrollable (I believe this error is only happening because the link is in scrollable area). The error is show up if the browser (that the Selenium script opens up when you run it), does not show the "Technology" link ie. You have to maximize the browser or scroll down to see the link. You can test this error out yourself - when the test opens up the browser, quickly resize the browser window so that the "Technology" link is not showing, and the test will fail. If your initial browser window doesn't show "Technology" link when it first opens, it will fail unless you maximize the browser window or scroll down really quickly until the "Technology" link is displayed. The error can take two forms - if you are using the wait.until method, you will get a Timeoutexception. If you are not using the wait.until method, you will see the ElementNotVisibleException. I've tested in both Chrome and Firefox and for both, I am seeing this problem. I am new to Selenium, but I believe this is not normal behavior, can someone please confirm? If this normal, then can someone please tell me how I can make sure my test runs correctly each time?
I think you want to use element_to_be_clickable as your ExpectedCondition.
That means that your driver will keep polling Selenium until it finds that the element is visible and enabled source
Try the below solution
link = wait.until(EC.element_to_be_clickable((By.XPATH, "//ul[#id='anchorman-two-browse-nav']/li/a[contains(text(),'Technology')]")))
link.click()
time.sleep(10)