I am attempting to scrape the Census website for ACS data. I have scripted the whole processes using Selenium except the very last click. I am using Python. I need to click a download button that is in a window that pops when the data is zipped and ready, but I can't seem to identify this button. It also seems that the button might change names based on when it was last run, for example, yui-gen2, yui-gen3, etc so I am thinking I might need to account for this someone. Although I normally only see yui-gen2.
Also, the tag seems to be in a "span" which might be adding to my difficulty honing in on the button I need to click.
Please help if you can shed any light on this for me.
code snippet:
#Refine search results to get tables
driver.find_element_by_id("prodautocomplete").send_keys("S0101")
time.sleep(2)
driver.find_element_by_id("prodsubmit").click()
driver.implicitly_wait(100)
time.sleep(2)
driver.find_element_by_id("check_all_btn_above").click()
driver.implicitly_wait(100)
time.sleep(2)
driver.find_element_by_id("dnld_btn_above").click()
driver.implicitly_wait(100)
driver.find_element_by_id("yui-gen0-button").click()
time.sleep(10)
driver.implicitly_wait(100)
driver.find_element_by_id("yui-gen2-button").click()
enter image description here
enter image description here
Instead of using the element id, which as you pointed out varies, you can use XPath as Nogoseke mentioned or CSS Selector. Be careful to not make the XPath/selector too specific or reliant on changing values, in this case the element id. Rather than using the id in XPath, try expressing the XPath in terms of the DOM structure (tags):
//*/div/div/div/span/span/span/button[contains(text(),'Download')]
TIL you can validate your XPath by using the search function, rather than by running it in Selenium. I right-clicked the webpage, "inspect element", ctrl+f, and typed in the above XPath to validate that it is the Download button.
For posterity, if the above XPath is too specific, i.e. it is reliant on too many levels of the DOM structure, you can do something shorter, like
//*button[contains(text(),'Download')]
although, this may not be specific enough and may require an additional field, since there may be multiple buttons on the page with the 'Download' text.
Given the HTML you provided, you should be able to use
driver.find_element_by_id("yui-gen2-button")
I know you said you tried it but you didn't say if it works at all or what error message you are getting. If that never works, you likely have an IFRAME that you need to switch to.
If it works sometimes but not consistently due to changing ID, you can use something like
driver.find_element_by_xpath("//button[.='Download']")
On the code inspection view on Chrome you can right click on the item you want to find and copy the xpath. You can they find your element by xpath on Selenium.
Related
HTML Source here
It seems trivial, but after hours of attempts I cannot manage to check this checkbox using Selenium. I am able to select the element no problem, and have even been able to 'highlight' (see code snippet below) it by sending Keys.RETURN to the element, but on attempting to click it, nothing happens. Perhaps someone has an idea?
What I have tried already:
Using with WebDriverWait
Using multiple and different combinations of .click()/.send_keys(Keys.RETURN)
Just about every combination of XPATH/css selector/id/name/classname/text that Selenium would accept for the element, and all of its children and most of its parents (including spans/labels, and the text inside the label itself).
Directly clicking the element/label coordinates using actions (nothing happens, even when I can 100% confirm it is clicking the right spot by using context_click()).
Using execute_script to change the "checked" attribute to True (the checkbox appears checked, but it is clear it is only client side as a box that is supposed to render when it is actually clicked doesn't).
Using execute_script to change the class to 'ng-valid ng-not-empty ng-dirty ng-valid-parse ng-touched'
Here is the code that I feel got me the closest (it is 'highlighting' the box as seen here)
browser.find_element(By.ID, "sp_formfield_none_of_the_above").send_keys(Keys.RETURN)
Managed to solve it after another while of testing. Hopefully this helps someone in the future. I had to use JS to execute .click() on the checkbox element. In retrospect, I should have tried this solution sooner. See code snippet:
cb_none = browser.find_element(By.ID, 'sp_formfield_none_of_the_above')
browser.execute_script("arguments[0].click();", cb_none)
I'm new to using Selenium, and I am having trouble figuring out how to click through all iterations of a specific element. To clarify, I can't even get it to click through one as it's a dropdown but is defined as an element.
I am trying to scrape fanduel; when clicking on a specific game you are presented with a bunch of main title bets and in order to get the information I need to click the dropdowns to get to that information. There is also another drop down that states, "See More" which is a similar problem, but assuming this gets fixed I'm assuming I will be able to figure that out.
So far, I have tried to use:
find_element_by_class_name()
find_element_by_css_selector()
I have also used them in the sense of elements, and tried to loop through and click on each index of the list, but that did not work.
If there are any ideas, they would be much appreciated.
FYI: I am using beautiful soup to scrape the website for the information, I figured Selenium would be helpful making the information that isn't currently accessible, accessible.
This image shows the dropdowns that I am trying to access, in this case the dropdown 'Win Margin'. The HTML code is shown to the left of it.
This also shows that there are multiple dropdowns, varying in amount based off the game.
You can also try using action chains from selenium
menu = driver.find_element_by_css_selector(".nav")
hidden_submenu = driver.find_element_by_css_selector(".nav # submenu1")
ActionChains(driver).move_to_element(menu).click(hidden_submenu).perform()
Source: here
I am making an automation for download data from a weather institution.
The issue is that in my effort to make it more independent I am trying to make Selenium to Tab keys to a certain spot, so the Browser focus can "Walk" to the download button. When I call the click() function it doesn't do anything. So I tried to Extract the XPath with the function get_attribute("xpath") but it returns None. How I can extract the XPath?
I am going to paste the issue down here:
Bandera=driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/div[3]/div/div/div/div[1]/div/div/div[2]/div/table/tbody/tr[1]/td[1]/div/input')
Bandera.click()
Bandera.click()
## So Here i just select and dis-select a checkbox just to be near the Download button.
actions = ActionChains(driver)
actions.send_keys(Keys.TAB * 1 )
actions.perform()
#Here i just tabed to the button
Accion=driver.switch_to.active_element
#Maybe, here is when i lost the focus of the button?
Descarga_Actual=Accion.get_attribute("xpath")
Thank you and sorry to borrow your time.
To make a click on hover I would use the following sequence:
your_dropdown_locator.click()
dropdown_option = driver.find_element_by_xpath("dropdown option locator")
actions = ActionChains(driver)
actions.move_to_element(dropdown_option)
actions.click().perform()
But, this is the approach that is usually used for dropdowns.
You use: driver.find_element_by_xpath('/html/body/div[2]/div[2]/div/div[3]/div/div/div/div[1]/div/div/div[2]/div/table/tbody/tr[1]/td[1]/div/input') This is the main problem of your code.
If you pasted html code of this dropdown (or button you need to click), people would help you to find a better locator. XPath/CSS must be unique. In your case the locator is very bad.
Also, I see no sense making Bandera.click() two times.
In your case, as I understand, you just need to click the button. So the locator is your main problem.
You need to find the correct locator, to wait till the button is clickable and then to click it.
Another problem in what you are trying to do:
get_attribute("xpath") looks like incorrect expectation of how get_attribute function works. Check at least here what this function means Python Selenium: Find object attributes using xpath
I'm relatively new to using python & selenium. I'm trying to access NexisUni to automate a loop of searches. But, once I'm in NexisUni, I struggle to locate elements -- I get a "no such element" exception. I want to locate the search bar and input my search terms.
I've read about the fact that an iFrame might be present, and I need to switch frames. But, I don't see any frames! Is there a way to identify frames easily -- and could a frame be present without the word "frame" in the HTML? I've also tried loading the page longer and having the driver wait, to no avail.
The HTML code is below, the grey part is the piece I'd like to select:
HTML Code
The code I'm writing to identify it is:
SearchBar = driver.find_element_by_xpath('/html/body/main/div/div[13]/div[2]/div[1]/header/div[3]/section/span[2]/span/textarea').send_keys('search text')
I've also tried these two options:
find_element_by_class_name, find_element_by_id
WebDriverWait(driver,10).until(EC.presence_of_element_located)
... Any suggestions would be appreciated!
I am having trouble finding elements on a customer-facing webpage that I am scraping data from, using Robot Framework + Selenium. My trouble, I think, has to do with the desired data existing in a popup/frame. The data I seek is located on a customer's invoice, which pops up when I press a button ("View Current Invoice"). I've been successful with logging into the site and navigating around, and at one point I was successful pressing the View Current Invoice button to cause the invoice to pop up - but forgot to commit that code and lost it. :-(
In any case, eve if I manually enter the popped up invoice by pressing the button when my script is expecting it to be pressed, I can't seem to scrape the subsequent data. I have tried to identify elements on the invoice using locators (from Right-Click-Inspect capability built into Firefox and Chrome; Katalaon Recorder; Selenium IDE; etc.). I get what looks like a valid locator (almost always Xpath); yet when I run my Robot script, it fails to find the element in question. I have spent a lot of time poring over the page's source code, but since I am not as savvy with HTML/JS/CSS as I should be, I haven't been successful.
Here is a screenshot of the invoice button:
And here is what I see when the button is pressed. I want to scrape all the invoice data, like Amount Due, Invoice Number, Due Date, etc.
Does anyone have any idea what I am missing here? What would you do to get the data on the invoice if you were in my shoes? I know my question probably sounds vague and naiive, but I am at the end of my rope, so to speak. I am willing to share page source code, more screenshots, whatever is required.
EDIT I used Rahul Rai's method to inspect the popup while it was popped up; then searched for "iframe". There were 10 matches; #7, when clicked on, resulted in the invoice popup being highlighted in blue:
I assume this means this is the iframe referencing the popup? If so, I should be able to find information about the "handle" to the iframe in the inspection code, but I don't see anything there that matches the locators I am used to (e.g. name, id, xpath). I even tried "Select Frame 1599252503952", but that just resulted in a
"Element with locator '1599252503952' not found" error.
As per above screen you have shared I can see your Invoice details are inside iframe. So after clicking on View Current Invoice button you can use below code to navigate inside frame and then scrape required information.
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH,"//iframe[contains(#src,'invoice_detail_container']")))
#Code to scrape data
ele = driver.find_element_by_xpath('<xpath>')
print(ele.txt)
......
......
#After your work is done in this frame to navigate back to main window
driver.switch_to.default_content()
Note: I have assumed your main frame for invoice is not in side any other iframe ( Based on screen shared). Also before elements start there is no other nested frame. If there is any other nested frame you need to navigate first into that.
I was finally able to scrape data from the Invoice popup after inspecting the HTML source, and seeing this:
<iframe frameborder="0" src="/cmc/invoice_detail_container.pyt?direction=//my.hughesnet.com/cmc/invoice_detail.pyt%3Finvnumber%1234-567890&portletId=863" name="1599391562960" class="cboxIframe" allowfullscreen="true" webkitallowfullscreen="true" mozallowfullscreen="true"></iframe>
I was then able to use the Select Frame keyword in Robot Framework, specifying the iframe locator for the popup, using the 'class' strategy. I also had to explicitly select the main body frame first. In the end, the code that allowed me to enter the iframe and scrape was:
Select Frame body
Select Frame class:cboxIframe
Big thank you to Rahul Rai for pushing me closer to the solution; and thanks to the others who answered as well.
You need to switch your site to frame/popup, you can use like below example, may be it will help you.
IList<IWebElement> textfields = new List<IWebElement>();
textfields = driver.FindElements(By.TagName("iframe"));
driver.SwitchTo().Frame(textfields[count); // number of textfields list.
please try to implement as per your scenario, let me know if any question.
You can try to do :
driver.switch_to_active_element()
and then scrape the popup to close it. Then I think it will be okay...