I am pretty new in selenium and I was looking forward to execute a click() button that expands a table, and after that I was looking forward to scrape the content of the expanded table.
This is what I tried:
url = 'https://www.telegraph.co.uk/markets-hub/assets/shares/'
phantomjs_path = '/usr/local/bin/phantomjs'
driver = webdriver.PhantomJS(executable_path=phantomjs_path,service_args=['--load-images=no'])
driver.get(url)
element = driver.find_element_by_xpath("//div[#class='tmg-show-more']")
html_source = driver.page_source
element.click()
driver.quit()
soup = BeautifulSoup(html_source,"html5lib")
tables=soup.find('table',class_="table-static kurser-table")
link=pd.read_html(str(tables),flavor='html5lib',thousands='.',decimal=',',header=0)
print(link)
The current output is as it follows:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: {"errorMessage":"Element is not currently visible and may not be manipulated",
"request":{"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"81","Content-
Type":"application/json;charset=UTF-8","Host":"127.0.0.1:58447","User-Agent":"Python http auth"},
"httpVersion":"1.1","method":"POST","post":"{\"id\": \":wdc:1556893378248\", \"sessionId\": \"f223cb80-6dae-11e9-b00b-4bb158952171\"}",
"url":"/click","urlParsed":{"anchor":"","query":"","file":"click","directory":"/","path":"/click","relative":"/click","port":"","host":"","password":"","user":"",
"userInfo":"","authority":"","protocol":"","source":"/click","queryKey":{},"chunks":["click"]},"urlOriginal":"/session/f223cb80-6dae-11e9-b00b-4bb158952171/element/:wdc:1556893378248/click"}}
Screenshot: available via screen
You would need to scroll down to let the selenium web driver knows where the element is actually present.
Code :
driver.get("https://www.telegraph.co.uk/markets-hub/assets/shares/")
wait = WebDriverWait(driver, 10)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div.section-box.kurser-table-container')))
element = driver.find_element_by_xpath("//a[#ng-click='stocksShowMore()']")
actions = ActionChains(driver)
actions.move_to_element(element).perform()
element.click()
Imports :
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
You are getting ElementNotVisibleException means you are trying to click the element before it is visible.
Also, you are getting the page source before clicking the button. You need to get the source after clicking and loading the new data.
Try the following code before getting the page source.
wait = WebDriverWait(driver, 20)
element = wait.until(EC.element_to_be_clickable(driver.find_element_by_xpath("//div[#class='tmg-show-more']")))
element.click()
wait.until(EC.invisibility_of_element(element))
To use the WebDriverWait you need to import these
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Related
I am trying to click a link on a web page with Python Selenium but I am getting this exception:
no such element: Unable to locate element:
I have already tried using find_element_by_xpath, find_element_by_partial_link_text and find_element_by_link_text.
This is my code:
import time
from selenium import webdriver
driver = webdriver.Chrome('C:/Users/me/Downloads/projetos/chromedriver_win32/chromedriver.exe') # Optional argument, if not specified will search path.
driver.get('http://10.7.0.4/web/guest/br/websys/webArch/mainFrame.cgi');
time.sleep(10) # Let the user actually see something!
#elem = driver.find_element_by_xpath('//*[#id="machine"]/div[1]/div[1]/dl[2]/dt/a')
elem = driver.find_element_by_link_text('Mensagens (2item(ns))')
elem.click()
print("Fim...")
This is the element I need to click:
Mensagens (2item(ns))
You can try with explicit waits and with the customized css :
CSS_SELECTOR :
a[href*='../../websys/webArch/getStatus.cgi']
Sample code :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "a[href*='../../websys/webArch/getStatus.cgi']"))).click()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
Update 1 :
wait = WebDriverWait(driver, 10)
wait.until(EC.element_to_be_clickable((By.XPATH, "//a[contains(#href, 'ebsys/webArch/getStatus.cgi') and contains(text(), 'Mensagens')]"))).click()
I am trying to get the review of a certain product but it returns an error.
My code:
import selenium
from selenium import webdriver
chrome_path = r"C:\Users\AV\AppData\Local\Programs\Python\Python39\Scripts\chromedriver.exe"
driver = webdriver.Chrome(chrome_path)
driver.get("https://oldnavy.gapcanada.ca/browse/product.do?pid=647076053&cid=1180630&pcid=26190&vid=1&nav=meganav%3AWomen%3ADeals%3ASale&grid=pds_0_1034_1#pdp-page-content")
driver.execute_script("window.scrollTo(0, 1000)")
import time
from time import sleep
sleep(5)
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.promoDrawer__handlebar__icon"))).click()
review = driver.find_elements_by_class_name("pr-rd-description-text")
for post in review:
print(post.text)
driver.find_element_by_xpath('//*[#id="pr-review-display"]/footer/div/div/a').click()
review2 = driver.find_elements_by_class_name("pr-rd-description-text")
for post in review2:
print(post.text)
It returns: selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable
Can you please tell me what should I do?
The button is really hard to click.
I guess it could be achieved by adding some more waits and moving with ActionChains class methods.
I could click it with Javascript code with no problems.
What it does:
1 Scrolls to the Next button
2 Clicks it.
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path='/snap/bin/chromium.chromedriver')
driver.get(
"https://oldnavy.gapcanada.ca/browse/product.do?pid=647076053&cid=1180630&pcid=26190&vid=1&nav=meganav%3AWomen%3ADeals%3ASale&grid=pds_0_1034_1#pdp-page-content")
driver.execute_script("window.scrollTo(0, 1000)")
wait = WebDriverWait(driver, 20)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div.promoDrawer__handlebar__icon"))).click()
wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, ".pr-rd-description-text")))
review = driver.find_elements_by_css_selector(".pr-rd-description-text")
for post in review:
print(post.text)
# wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".pr-rd-pagination-btn"))).click()
element = driver.find_element_by_css_selector(".pr-rd-pagination-btn")
# actions = ActionChains(driver)
# actions.move_to_element(element).click().perform()
driver.execute_script("arguments[0].scrollIntoView();", element) # Scrolls to the button
driver.execute_script("arguments[0].click();", element) # Clicks it
print("clicked next")
I also rearranged your code, moved imports to the beginning of the file, got rid of unpredictable time.sleep() and used more reliable css locators. However, your locator should also work.
I left the options I tried commented out.
That element is weird. Even when I scroll into view, use actions to click it, or execute javascript to click, it doesn't work. What I would suggest is just grabbing the href attribute from the element and going to that URL, using something like this:
driver.get(driver.find_element_by_xpath('//*[#id="pr-review-display"]/footer/div/div/a').get_attribute('href'))
I'm working to automate web page where i'm unable to close the pop up. I have tried to refresh/switch to pop up window, nothing worked.
Code:
from selenium import webdriver
import time
driver = webdriver.Chrome(executable_path='F:\chromedriver_win32\chromedriver.exe')
driver.get('https://www.libertymutual.com/get-a-quote')
driver.maximize_window()
driver.find_element_by_xpath(
'//*[#id="1555468516747"]/section/div[2]/section/form/div[1]/div[3]/div/div[1]').click()
time.sleep(1)
driver.find_element_by_xpath('//*[#id="zipcode-1555468516747-1555468516747"]').click()
driver.find_element_by_xpath('//*[#id="zipcode-1555468516747-1555468516747"]').send_keys('03878')
time.sleep(1)
driver.find_element_by_xpath(
'//*[#id="1555468516747"]/section/div[2]/section/form/div[2]/div[5]/div/div/button').submit()
time.sleep(5)
x=driver.find_element_by_xpath('//*[#id="discount-marketing-modal"]/header/button/svg').click()
driver.refresh()
If you want directly go to web page, https://buy.libertymutual.com/auto?city=Somersworth&jurisdiction=NH&lob=Auto&policyType=Auto&zipCode=03878
Replace last 3 lines of your code by below lines.Used action chain to click.
time.sleep(5)
ok = driver.find_element_by_xpath('//*[#id="discount-marketing-modal"]/footer/button')
ActionChains(driver).move_to_element(ok).pause(1).click(ok).perform()
x=driver.find_element_by_xpath('//*[#id="discount-marketing-modal"]/header/button/svg').click()
This is css selector for popup close button : .lm-Icon.lm-Icon-Close
But this element can't use the .click() method, the .click() method will produce this error:
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted:
Use ActionChains to solve this problem.
You can try the following code:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver import ActionChains
driver = webdriver.Chrome(executable_path='F:\chromedriver_win32\chromedriver.exe')
driver.get('https://www.libertymutual.com/get-a-quote')
driver.maximize_window()
wait = WebDriverWait(driver, 20)
auto_element = wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#class='quotingText']//span[contains(text(), 'Auto')]")))
auto_element.click()
zip_code = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "input.zipcode")))
zip_code.click()
zip_code.send_keys("03878")
driver.find_element_by_css_selector('div.buttonWrapper button').submit()
popup_close_btn = wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".lm-Icon.lm-Icon-Close")))
action = ActionChains(driver)
action.move_to_element(popup_close_btn).click(popup_close_btn).perform()
WebDriverWait better than time.sleep(..)
I am attempting to click the "View More Results" button on the following page: http://www.chadbourne.com/search/people?az[b]=b
My code is straight forward, and I have tried a number of iterations.
driver = driver.Firefox()
driver.get("http://www.chadbourne.com/search/people?az[b]=b")
element = driver.find_element_by_partial_link_text("View more results")
or
element = driver.find_element_by_partial_link_text("view")
or
element = driver.find_element_by_partial_link_text("results")
No matter which of the above options I try, I get a NoSuchElementException.
This is odd, because the element clearly exists on the page:
View more results
Any thoughts?
It takes some time for the page to load, the element is not immediately available. Let's wait for it to be clickable and change the locator to a CSS selector:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://www.chadbourne.com/search/people?az[b]=b")
wait = WebDriverWait(driver, 20)
# get more results
more_results = wait.until(
EC.element_to_be_clickable((By.CSS_SELECTOR, ".load-more-pager-wrapper .pager-next"))
)
more_results.click()
I am trying to scrape an airbnb listing. I cant figure out a way to get the full list of amenities other than clicking on "more". I am using selenium to simulate the click, but it does nt seem to work.
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
url = 'https://www.airbnb.com/rooms/4660676'
driver = webdriver.Firefox()
driver.get(url)
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
actions.click(elem).perform()
The XPath itself is correct, but you have not defined the actions:
from selenium.webdriver.common.action_chains import ActionChains
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
actions = ActionChains(driver)
actions.click(elem).perform()
Note that you can simply use the WebElement's click() method instead:
elem = driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]')
elem.click()
Works for me.
If you are getting NoSuchElementException error, you might need to wait for the link to be clickable via WebDriverWait and element_to_be_clickable.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, '//a[#class="expandable-trigger-more"]'))
)
element.click()
A very simple way of achieving this is given below
driver.find_element_by_xpath('//a[#class="expandable-trigger-more"]').click()
It works for me hope will work for you as well.