I've written a script in python in combination with selenium to get some information from a webpage. To reach the content it is necessary to hurdle few steps, as in accepting condition, fill in the inputbox, click on the search button to populate results and finally click on the first grid (the first tr, more specifically) within the populated table. As soon as any click is initiated on the first tr, a new page (containing desired information) opens up.
My script can do the first three steps successfully. What I can't do are:
perform a click on the first tr
focus to the newly opened tab (containing information I'm after)
To reach the content:
This is the link to follow. There is an accept button to click first. Then there is an inputbox Name to be filled in with HMC DESIGN GROUP. Now, pressing the search button, the result should appear below within a table. From there I need to click on the first tr.
This is what I've tried so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "https://officialrecords.broward.org/AcclaimWeb/search/SearchTypeName"
def get_information(driver,url):
driver.get(url)
wait.until(EC.element_to_be_clickable((By.ID, "btnButton"))).click()
wait.until(EC.presence_of_element_located((By.ID,"SearchOnName"))).send_keys("HMC DESIGN GROUP")
wait.until(EC.presence_of_element_located((By.ID, "btnSearch"))).click()
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR,".t-grid-content table tr"))).click()
if __name__ == "__main__":
driver = webdriver.Chrome()
wait = WebDriverWait(driver,10)
try:
get_information(driver,link)
finally:
driver.quit()
Currently the script neither clicks on the grid the first tr of the newly generated table nor throws any error. It quits the browser gracefully.
There is another table under the div with the same class name "t-grid-content". It appears as "Loading...". So after you submit the search you really make a click on node with selector ".t-grid-content table tr", but it just don't make a proper effect
You just need more specific selector.
Try, for instance
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[contains(., 'HMC DESIGN GROUP')]")))
To switch to new window, try update your function body as
driver.get(url)
current = driver.current_window_handle
wait.until(EC.element_to_be_clickable((By.ID, "btnButton"))).click()
wait.until(EC.presence_of_element_located((By.ID,"SearchOnName"))).send_keys("HMC DESIGN GROUP")
wait.until(EC.presence_of_element_located((By.ID, "btnSearch"))).click()
wait.until(EC.element_to_be_clickable((By.XPATH, "//td[contains(., 'HMC DESIGN GROUP')]"))).click()
driver.switch_to.window([window for window in driver.window_handles if window != current][0])
Just replace this css selctor :
.t-grid-content table tr
to this :
.t-grid-content table tr.t-state-selected:first-child
for clicking on first tr.
Before clicking on first tr, store the window handle as :
window_before = driver.window_handles[0]
after clicking on first tr, store the window handle of newly opened window as:
window_after = driver.window_handles[1]
Now all you have to do is to switch the focus of your web driver to newly opened windows.Like this :
driver.switch_to_window(window_after)
Related
Hi Before starting Thanks for the help in advance
So I am trying to scrape google flight website : https://www.google.com/travel/flights
When scraping I have done the sending Key to the text field but I am stuck at clicking the search button it always gives the error that the field is not clickable at a point or Other elements would receive the click
the error image is
and the code is
from time import sleep
from selenium import webdriver
chromedriver_path = 'E:/chromedriver.exe'
def search(urrl):
driver = webdriver.Chrome(executable_path=chromedriver_path)
driver.get(urrl)
asd= "//div[#aria-label='Enter your destination']//div//input[#aria-label='Where else?']"
driver.find_element_by_xpath("/html/body/c-wiz[2]/div/div[2]/c-wiz/div/c-wiz/c-wiz/div[2]/div[1]/div[1]/div[1]/div[2]/div[1]/div[4]/div/div/div[1]/div/div/input").click()
sleep(2)
TextBox = driver.find_element_by_xpath(asd)
sleep(2)
TextBox.click()
sleep(2)
print(str(TextBox))
TextBox.send_keys('Karachi')
sleep(2)
search_button = driver.find_element_by_xpath('//*[#id="yDmH0d"]/c-wiz[2]/div/div[2]/c-wiz/div/c-wiz/c-wiz/div[2]/div[1]/div[1]/div[2]/div/button/div[1]')
sleep(2)
search_button.click()
print(str(search_button))
sleep(15)
print("DONE")
driver.close()
def main():
url = "https://www.google.com/travel/flights"
print(" Getitng Data ")
search(url)
if __name__ == "__main__":
main()
and i have done it by copying the Xpath using dev tools
Thanks again
The problem you are facing is that after entering the city Karachi in the text box, there is a suggestion dropdown that is displayed over the Search Button. That is the cause of the exception as the dropdown would receive the click instead of the Search button. The intended usage of the website is to select the city from the dropdown and then continue.
A quick fix would be to first look for all of the dropdowns in the source (there are a few) and look for the one that is currently active using is_displayed(). Next you would select the first element in the dropdown suggested:
.....
TextBox.send_keys('Karachi')
sleep(2)
# the attribute(role) in the dropdowns element are named 'listbox'. Warning: This could change in the future
all_dropdowns = driver.find_elements_by_css_selector('ul[role="listbox"]')
# the active dropdown
active_dropdown = [i for i in all_dropdowns if i.is_displayed()][0]
# click the first suggestion in the dropdown (Note: this is very specific to your search. It could not be desirable in other circumstances and the logic can be modified accordingly)
active_dropdown.find_element_by_tag_name('li').click()
# I recommend using the advise #cruisepandey has offered above regarding usage of relative path instead of absolute xpath
search_button = driver.find_element_by_xpath("//button[contains(.,'Search')]")
sleep(2)
search_button.click()
.....
Also suggest to head the advise provided by #cruisepandey including research more about Explicit Waits in selenium to write better performing selenium programs. All the best
Instead of this absolute xapth
//*[#id="yDmH0d"]/c-wiz[2]/div/div[2]/c-wiz/div/c-wiz/c-wiz/div[2]/div[1]/div[1]/div[2]/div/button/div[1]
I would recommend you to have a relative path:
//button[contains(.,'Search')]
Also, I would recommend you to have explicit wait when you are trying a click operation.
Code:
search_button = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[contains(.,'Search')]")))
search_button.click()
You'd need below imports as well:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Pro tip:
Launch the browser in full screen mode.
driver.maximize_window()
You should have the above line just before driver.get(urrl) command.
I am trying to click to an object that I select with Xpath, but there seems to be problem that I could not located the element. I am trying to click accept on the page's "Terms of Use" button. The code I have written is as
driver.get(link)
accept_button = driver.find_element_by_xpath('//*[#id="terms-ok"]')
accept_button.click()
prov = driver.find_element_by_id("province-region")
prov.click()
Here is the HTML code I have:
And I am getting a "NoSuchElementException". My goal is to click this "Kabul Ediyorum" button at the bottom of the HTML code. I started to think that we have some restrictions on the tasks we could do on that page. Any ideas ?
Not really sure what the issue might be.
But you could try the following:
Try to locate the element by its visible text
accept_button = driver.find_element_by_xpath("//*[text()='Kabul Ediyorum']").click()
Try with ActionChains
For that you need to import ActionChains
from selenium.webdriver.common.action_chains import ActionChains
accept_button = driver.find_element_by_xpath("//*[text()='Kabul Ediyorum']")
actions = ActionChains(driver)
actions.click(on_element=accept_button).perform()
Also make sure you have implicitly wait
# implicitly wait
driver.implicitly_wait(10)
Or explicit wait
element = WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//*[text()='Kabul Ediyorum']"))).click()
Hope this helped!
I am trying to scrape information on a website where the information is not immediately present. When you click a certain button, the page begins to load new content on the bottom of the page, and after it's done loading, red text shows up as "Assists (At Least)". I am able to find the first button "Go to Prop builder", which doesn't immediately show up on the page, but after the script clicks the button, it times out when trying to find the "Assists (At Least)" text, in spite of the script sleeping and being present on the screen.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup
driver = webdriver.Chrome()
driver.get('https://www.bovada.lv/sports/basketball/nba')
# this part succeeds
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located(
(By.XPATH, "//span[text()='Go to Prop builder']")
)
)
element.click()
time.sleep(5)
# this part fails
element2 = WebDriverWait(driver, 6).until(
EC.visibility_of_element_located(
(By.XPATH, "//*[text()='Assists (At Least)']")
)
)
time.sleep(2)
innerHTML = driver.execute_script('return document.body.innerHTML')
driver.quit()
soup = BeautifulSoup(innerHTML, 'html.parser')
The problem is the Assist element is under a frame. You need to switch to the frame like this:
frame = WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CLASS_NAME,"player-props-frame")))
driver.switch_to.frame(frame)
Increase the timeout to confirm the timeout provided is correct, You can also confirm using debug mode. If still issue persist, please check "Assists (At Least)" element do not fall under any frame.
You can also share the DOM and proper error message if issue not resolved.
I have a couple of suggestions you could try,
Make sure that the content loaded at the bottom of the is not in a frame. If it is, you need to switch to the particular frame
Check the XPath is correct, try the XPath is matching from the Developer Console
Inspect the element from the browser, once the Developer console is open, press CTRL +F and then try your XPath. if it's not highlighting check frames
Check if there is are any iframes in the page, search for iframe in the view page source, and if you find any for that field which you are looking for, then switch to that frame first.
driver.switch_to.frame("name of the iframe")
Try adding a re-try logic with timeout, and a refresh button if any on the page
st = time.time()
while st+180>time.time():
try:
element2 = WebDriverWait(driver, 6).until(
EC.visibility_of_element_located(
(By.XPATH, "//*[text()='Assists (At Least)']")
)
)
except:
pass
The content you want is in an iFrame. You can access it by switching to it first, like this:
iframe=driver.find_element_by_css_selector('iframe[class="player-props-frame"]')
driver.switch_to.frame(iframe)
Round brackets are the issue here (at least in some cases...). If possible, use .contains selector:
//*[contains(text(),'Assists ') and contains(text(),'At Least')]
I am trying to write a code that is able to auto apply on job openings on indeed.com. I have managed to reach the last stage, however, the final 2 clicks on the application form is giving me a lot of trouble. Please refer to the first page as below
Once we click on continue on the first page, for the second page I first need to scroll down a bit to reach from here...
..to here and then finally click on apply.
I am stuck on the first page, as the click function does not do anything. I have written the following code:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://in.indeed.com/jobs?q=data%20analyst&l=Delhi&vjk=5c0bd416675cf4e5")
driver.find_element_by_xpath('//*[#id="apply-button-container"]/div[1]/span[1]').click()
time.sleep(5)
frame_1 = driver.find_element_by_css_selector('iframe[title="Job application form container"')
driver.switch_to.frame(frame_1)
frame_2 = driver.find_element_by_css_selector('iframe[title="Job application form"]')
driver.switch_to.frame(frame_2)
continue_btn = driver.find_element_by_css_selector('#form-action-continue')
continue_btn.click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#id='form-action-continue']"))).click()
driver.find_element_by_xpath('//button[#id="form-action-continue"]').click()
I have tried switching the iframes again for this step but nothing happens. Even the .click() function does not do anything.
Will appreciate some help on this.
This should go through the first click if your values are inputted.
driver.get("https://in.indeed.com/jobs?q=data%20analyst&l=Delhi&vjk=5c0bd416675cf4e5")
wait=WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a.indeed-apply-button"))).click()
frame_1 = driver.find_element_by_css_selector('iframe[#title="Job application form container"')
driver.switch_to.frame(frame_1)
frame_2 = driver.find_element_by_css_selector('iframe[#title="Job application form"]')
driver.switch_to.frame(frame_2)
cont="#form-action-continue"
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, cont))).click()
This is a follow-up question to this:
WebDriver element found, but click returns nothing
I am trying to scrape data from the URL in the code after making selections in the drop-down menu. I first click on Progress Monitoring and then Physical and Financial Project Summary. Then I make the following selections: State, District, Block, Year, Batch, and Collaboration. I would also like to check the Road Wise button and then click on the view button. After the table loads, I would like to click on the save button and download the excel file. In the code below I also loop through different selections under "State" item. Here is my code:
from selenium import webdriver
from selenium.webdriver.support.ui import Select
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
import time
import os
chromedriver = r"C:\Users\yuppal\chromedriver"
os.environ["webdriver.chrome.driver"] = chromedriver
browser = webdriver.Chrome(chromedriver)
browser.implicitly_wait(10)
browser.get("http://omms.nic.in")
browser.maximize_window()
#Click on the item Progress Monitoring
progElem = browser.find_element_by_link_text("Progress Monitoring").click()
#Click on the item Physical and Financial Project Sumamry
summElem = browser.find_element_by_link_text("Physical and Financial Project Summary").click()
#Find the element for state and create a list of different selection options
stateElem = browser.find_element_by_xpath("//select[#name='StateCode']")
state_options = stateElem.find_elements_by_tag_name("option")
#delete the first option in the list
del state_options[0]
def select_option(xpath, text):
'''
This function will select the remaining dropd-down menu items.
'''
elem = browser.find_element_by_xpath(xpath)
Select(elem).select_by_visible_text(text)
#run the loop for each option in the list of states
for option in state_options:
select_state = Select(stateElem).select_by_value(option.get_attribute("value"))
# Select the district.
select_option("//select[#name='DistrictCode']","All Districts")
# Select the block.
select_option("//select[#name='BlockCode']","All Blocks")
# Select the year.
select_option("//select[#name='Year']","All Years")
# Select the batch.
select_option("//select[#name='Batch']","All Batches")
# Select the funding agency.
select_option("//select[#name='FundingAgency']","Regular PMGSY")
# Check the road wise box.
time.sleep(10)
checkElem = WebDriverWait(browser, 120).until(EC.element_to_be_clickable((By.XPATH, "//input[#title='Road Wise']")))
browser.execute_script("arguments[0].click();", checkElem)
# Click on the view button.
time.sleep(10)
browser.find_element_by_xpath("//input[#type='button']").click()
# Switch to a new frame.
time.sleep(10)
frame = browser.find_element_by_xpath("//div[#id='loadReport']/iframe")
browser.switch_to.default_content()
#browser.switch_to.frame(frame)
WebDriverWait(browser, 120).until(EC.frame_to_be_available_and_switch_to_it(frame))
#browser.switch_to.frame(browser.find_element_by_xpath("//*[#id='loadReport']/iframe"))
# click on the save button
time.sleep(10)
WebDriverWait(browser, 120).until(EC.element_to_be_clickable((By.XPATH, "//a[#title='Export drop down menu']"))).click()
# Within the save button, Click on the "Excel" option.
time.sleep(10)
WebDriverWait(browser, 20).until(EC.element_to_be_clickable((By.XPATH, "//div/a[#title='Excel']"))).click()
# Switch back to the main content.
time.sleep(20)
browser.switch_to.default_content()
My issue is the "Road Wise" checkbox gets clicked only for some states. Thus the loop proceeds without clicking the checkbox for some states. I checked the HTML code and it is the same for all checkboxes.
I thought the problem might be that the "View" button gets clicked before the road wise button is clickable. So I put some waiting period before both road wise and view buttons. But that doesn't seem to help. So I can't really understand why the checkbox button isn't clicked for some iterations in the loop.
Before clicking on the checkbox, check that is already selected or not:
# Check the road wise box.
time.sleep(10)
checkElem = WebDriverWait(browser, 120).until(EC.element_to_be_clickable((By.XPATH, "//input[#title='Road Wise']")))
if checkElem.is_selected() != True:
browser.execute_script("arguments[0].click();", checkElem)
PS: In your case, the click will be only in the first iteration of the loop.