Ive been using selenium to scrape a site to retrieve some information.
the information is hidden behind a see more tab that is being revealed using javascript when i click it. I have tried many different methods to get the the information visible. and it does not seem to be working .
I have tried to use action chains along with the regular xpath methods to chain the functionality together but it still does not seem to be clicking all the other info is pulled and the button text is printed out to the console instead of being clicked.
def grabDetails(self):
facts = self.browser.find_elements_by_xpath("//section[#id='hdp-content']/main/div/div[4]")
for fact in facts:
details = fact.text
print(details)
def moreFeatures(self):
view_more_elements = WebDriverWait(self.browser, 20).until(EC.visibility_of_element_located((By.XPATH, "//a[contains(text(),'See More Facts and Features')]")))
# features.click()
ActionChains(view_more_elements).double_click().preform()
# self.browser.execute_script('arguments[0].scrollIntoView(true);', features)
Im trying to get the information from this page printed out !
here is the zillow page that im trying to scrape
its the see more sections part below it
You are setting view_more_elements as a WebDriverWait object rather than a WebElement. This will prevent the object from being clickable. You just need to run WebDriverWait as it's own call, then .click() the element.
view_more_xpath = '//div[#class="read-more zsg-centered"]/a'
view_more_elements = WebDriverWait(self.browser, 20).until(EC.visibility_of_element_located((By.XPATH, view_more_xpath))
self.browser.find_element_by_xpath(view_more_xpath).click()
EDIT: you don't actually need to set WebDriverWait to view_more_elements. You can just do:
view_more_xpath = '//div[#class="read-more zsg-centered"]/a'
WebDriverWait(self.browser, 20).until(EC.visibility_of_element_located((By.XPATH, view_more_xpath))
self.browser.find_element_by_xpath(view_more_xpath).click()
Related
Im trying to perform select and click action from the search box result dropdown for testing purpose. Though i dont get ant error but i'm stuck and not able to do so, search results came then disappeared immediately. Please any one help me out. Im using Python script to automate webdriver. Here is the screenshot below for reference.
. I have tried webdriverwait for same action but it gives Timeout exception error. If there is any child actions from CSS to perform let me know. Here is what i tried
search = driver.find_element_by_id('searchInput')
search.send_keys("flowers")
dropdown = WebDriverWait(driver, 4).until(
EC.presence_of_element_located((By.XPATH, "//li[text()='flowers']")))
Apart from this snippet, i want to rather just perform enter key operation, where i get query result for 'flower' on this ecomm. website.
Here is the website URL- https://paytmmall.com
Once you type flower in the input field, there are multiple options appearing based on the input provided. They are in li tags and under b tag.
Code :
driver = webdriver.Chrome(driver_path)
driver.maximize_window()
#driver.implicitly_wait(30)
wait = WebDriverWait(driver, 30)
driver.get("https://paytmmall.com/")
search = wait.until(EC.visibility_of_element_located((By.ID, "searchInput")))
search.send_keys("flowers")
time.sleep(3)
wait.until(EC.visibility_of_element_located((By.XPATH, "(//li)[4]/descendant::b[contains(text(),'flowers')]"))).click()
time.sleep is just for visibility purpose. you can remove that as well.
Also this xpath (//li)[4]/descendant::b[contains(text(),'flowers')] is based on xpath indexing , since I think you wanna select the 4th option which is flower itself. In case you wanna select a different option, you would have to write the different xpath.
In case you are looking to just select the searched item, it's better to pass enter key once you type flower in the input field.
You can use the below code for that :
search = wait.until(EC.visibility_of_element_located((By.ID, "searchInput")))
search.send_keys("flowers")
time.sleep(3)
search.send_keys(Keys.RETURN)
The suggested options are not containing the text directly in li elements, they are inside child elements inside li elements.
Try this instead:
search = driver.find_element_by_id('searchInput')
search.send_keys("flowers")
dropdown = WebDriverWait(driver, 4).until(
EC.visibility_of_element_located((By.XPATH, "//li//*[text()='flowers']")))
I am trying to extract data from this page using Python Selenium. The table is rendered by Tableau. I would need to input some data and then use the download button.
Interestingly, I can't access the elements inside the table from Selenium. I tried looking by id, class or xpath. I keep getting the NoSuchElementException. However, these elements are rendered in HTML and I can see them with the inspect tool. Does anyone know why this is, and how I can make them visible to Selenium?
EDIT1: It's not a problem of loading time. I tried with time.sleep() and I am also interacting directly with the page.
I can see your tables are inside iFrame. First go inside and then try to scrape table data.
WebDriverWait(driver, 20).until(EC.frame_to_be_available_and_switch_to_it((By.XPATH, "//iframe[contains(#src,'zika_Weekly_Agg_tben')]")))
# COde here to scrape data
driver.switch_to.default_content() # To come out of frame
You need to Import
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
This was quite challenging since it has 2 iframe followed by shadow element. And does not stop here. When you switch to the iframe you don't have iframe reference available to access the shadow element. you Can refer below code. It manage to get the tableau chart heading.
# Get first iframe and switch to it
root1 = driver.find_element_by_xpath("//div[#itemprop='articleBody']//iframe")
driver.switch_to.frame(root1)
# Grab the shadow element
shadow = driver.execute_script('return document')
# Get the iframe inside shadow element of first iframe
iframe2 = shadow.find_element_by_xpath("//body/iframe")
# switch to 2nd iframe
driver.switch_to.frame(iframe2)
print("selected 2nd iframe")
shadow_doc2 = driver.execute_script('return document')
print("second iframe")
heading = shadow_doc2.find_element_by_xpath("//div[#class='tab-textRegion-content']/span//span[text()='Cases of Zika Virus Disease']/ancestor::div[2]").text
print(heading)
Output -
I am trying to make a program in Python using Selenium which prints out the quotes from https://www.brainyquote.com/quote_of_the_day
EDIT:
I was able to access the quotes and the associated authors like so:
authors = driver.find_elements_by_css_selector("""div.col-xs-4.col-md-4 a[title="view author"]""")
for quote,author in zip(quotes,authors):
print('Quote: ', quote.text)
print('Author: ', author.text)
Not able to club topics similarly. Doing
total_topics = driver.find_elements_by_css_selector("""div.col-xs-4.col-md-4 a.qkw-btn.btn.btn-xs.oncl_list_kc""")
would make an undesired list
Earlier I was using Beautiful Soup which did the job perfectly except the fact that the requests library was able to access only the static website. However, I wanted to be able to scroll the website continuously to keep accessing new quotes. For that purpose, I'm trying to use Selenium.
This is how I did it using Soup:
for quote_data in soup.find_all('div', class_='col-xs-4 col-md-4'):
quote = quote_data.find('a',title='view quote').text
print('Quote: ',quote)
However, I am unable to find the same using Selenium.
My code in Selenium for basic testing:
driver.maximize_window()
driver.get('https://www.brainyquote.com/quote_of_the_day')
elem = driver.find_element_by_tag_name("body")
elem.send_keys(Keys.PAGE_DOWN)
time.sleep(0.2)
quote = driver.find_element_by_xpath('//div[#title="view quote"]')
I also tried CSS Selectors
print(driver.find_element_by_css_selector('div.col-xs-4 col-md-4')
The latter gave a NoSuchElementFound exception and the former is not giving any output at all. I would love to get some tips on where I am going wrong and how I would be able to tackle this.
Thanks!
quotes = driver.find_elements_by_xpath('//a[#title="view quote"]')
First scroll to bottom
You might need to write some kind of loop to scroll and click on the quotes links until there are no more elements found. Here's a bit of an outline of how I would do that:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get('https://www.brainyquote.com/quote_of_the_day')
while True:
# wait for all quote elements to appear
quote_links = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//a[#title='view quote']")))
# todo - need to check for the end condition. page has infinite scrolling
# break
# iterate the quote elements until we reach the end of this list
for quote_link in quote_links:
quote_link.click()
driver.back()
# now quote_links has gone stale because we are on a different page
quote_links = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.XPATH, "//a[#title='view quote']")))
The above code enters a loop that searches for all of the 'View more' quote links on the page. Then, we iterate the list of links and click on each one. At this point the elements in quote_links list have gone stale due to the page no longer existing, so we re-find the elements with WebDriverWait before clicking another link.
This is just a rough outline and some extra work will need to be done to determine an end case for the infinite scrolling of the page, and you will need to write in the operations to perform on the quote pages themselves, but hopefully you see the idea here.
I am using selenium in python and cannot get it to find and select the ok button. Here is the code I am trying to use that is not finding the button.
SubmitElem = driver.find_element_by_name('ctl00$PlaceHolderMain$ctl01$RptControls$btnOK')
SubmitElem.submit()
And this is the webpage code
<input name="ctl00$PlaceHolderMain$ctl01$RptControls$btnOK"
value="OK"
onclick="javascript:WebForm_DoPostBackWithOptions(new WebForm_PostBackOptions("ctl00$PlaceHolderMain$ctl01$RptControls$btnOK", "", true, "", "", false, false))"
id="ctl00_PlaceHolderMain_ctl01_RptControls_btnOK" accesskey="o"
class="ms-ButtonHeightWidth" type="submit">
Any advice on what I am doing wrong? I found some suggestions to add onclick*= but it did not seem to work.
As noted earlier, if the element is in an iframe, you need to switch contexts. Additionally, if the element takes time to show up on the DOM, due to it being dynamically rendered by JavaScript, you might need waits. You basically want to wait until the element shows up on the page before performing actions on it.
From the Selenium docs:
element = WebDriverWait(driver, 10).until(
EC.presence_of_element_located((By.ID, "myDynamicElement"))
)
Here is the Answer to your Question:
As per the HTML you have provided, you may consider the following options:
You can consider to construct a xpath based on name as follows:
SubmitElem = driver.find_element_by_xpath("//input[#name='ctl00$PlaceHolderMain$ctl01$RptControls$btnOK']")
SubmitElem.submit()
You can also try a different xpath based on id as follows:
SubmitElem = driver.find_element_by_xpath("//input[#id='ctl00_PlaceHolderMain_ctl01_RptControls_btnOK']")
SubmitElem.submit()
Alternatively, you can also consider to call click() method instead of submit() method.
SubmitElem.click()
Again, it seems to me that the id and name is dynamic due to presence of JavaScript and AJAX Calls. In that case you can consider to construct a dynamic xpath as follows:
SubmitElem = driver.find_element_by_xpath("//input[starts-with(#id, 'ctl00')]")
SubmitElem.submit()
You can consider to construct a dynamic css_selector as follows:
SubmitElem = driver.find_element_by_css_selector("input[id^='ctl00']")
SubmitElem.submit()
Finally, if you see a NoSuchElementException or ElementNotVisibleException you may consider to induce some WebDriverWait as follows:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "//input[starts-with(#name, 'ctl00')]")))
driver.find_element_by_css_selector("input[name^='ctl00']").click()
There are total 6 suggested locators to identify the element along with different other options.
Let me know if this Answers your Question.
I am trying to select a particular option within a select box on a webpage. http://www.arnoldporter.com/news.cfm.
Specifically I want to selection "FDA and Healthcare" option within the Practice/Industry selection box.
I have tried a number of things, including clicking on the select tag, and then clicking on the FDA option. I have also checked to see whether the select tag changes after being clicked on. It doesn't.
So, Nothing seems to work, I keep getting the same error: Element is not currently visible and so may not be interacted with.
The xpath I am using for the select box is: //select[#class="medium" and #name="search_practice_id"]
The xpath I am using for the option is: //option[#value="323"]
There must be a simple solution, I just can't figure it out. Any help would be appreciated.
from selenium import webdriver
from selenium.webdriver.support.ui import Select
URL = 'http://www.arnoldporter.com/news.cfm'
CSS_SELECTOR = 'select[name=search_practice_id][class=medium]'
OPTION_TEXT = 'FDA and Healthcare'
browser = webdriver.Chrome()
browser.get(URL)
select_el = browser.find_element_by_css_selector(CSS_SELECTOR)
select = Select(select_el)
select.select_by_visible_text(OPTION_TEXT)