I'm trying to scrap the list of services we have for us from this site but not able to click to the next page.
This is what I've tried so far using selenium & bs4,
#attempt1
next_pg_btn = browser.find_elements(By.CLASS_NAME, 'ui-lib-pagination_item_nav')
next_pg_btn.click() # nothing happens
#attemp2
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
#attempt3 - saw in some stackoverflow post that sometimes we need to scroll to the
#bottom of page to have the button clickable, so tried that
browser.execute_script("window.scrollTo(0,2500)")
browser.find_element(By.XPATH, "//div[#role = 'button']").click() # nothing happens
I'm not so experienced with scrapping, pls advice how to handle this and where I'm going wrong.
Thanks
Several issues with your code:
You tried wrong locators.
You probably need to wait for the element to be loaded before clicking it. But if before clicking the pagination you performing some actions on the page this is not needed since during you scraping the page content web elements are already got loaded.
Pagination button is on the buttom of the page, so you need to scroll the page to bring the pagination button into the visible screen.
After scrolling some delay should be added, as you can see in the code below.
Now pagination element can be clicked.
The following code works
import time
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.tamm.abudhabi/en/life-events/individual/HousingProperties"
driver.get(url)
pagination = wait.until(EC.presence_of_element_located((By.CLASS_NAME, "ui-lib-pagination__item_nav")))
pagination.location_once_scrolled_into_view
time.sleep(0.5)
pagination.click()
Related
I have the following code which is supposed to scroll down the page and then click a button. When I run my script, I can see that the page does scroll until the element is at the very bottom of the page, but then the script fails when it gets time to click on that button and I get this error:
selenium.common.exceptions.ElementNotInteractableException: Message: Element could not be scrolled into view
I have tried using these methods:
driver.execute_script("arguments[0].scrollIntoView();", element)
driver.execute_script("arguments[0].scrollIntoView();", driver.find_element_by_xpath(xpath of element))
actions.move_to_element(element)
All of these methods have the same result: the page will scroll to the element, but when it is time to click on the element it complains that the element could not be scrolled into view.
CODE:
hide_partial_rows_button_per_100 = driver.find_element_by_xpath('//button[#id="per_poss_toggle_partial_table"]')
#driver.execute_script("arguments[0].scrollIntoView();",driver.find_element_by_xpath('//button[#id="per_poss_toggle_partial_table"]'))
#driver.execute_script("arguments[0].scrollIntoView();",hide_partial_rows_button_per_100)
actions.move_to_element(hide_partial_rows_button_per_100)
actions.perform
hide_partial_rows_button_per_100.click()
LINK TO PAGE I'M WORKING ON: https://www.basketball-reference.com/players/v/valanjo01.html
Someone on this site had a similar question but they were using JavaScript instead of Python and they added a time.sleep(1) between scrolling to the element and clicking on it, this did not work for me.
As said above, I've tried both the driver.execute_script("arguments[0].scrollIntoView();",element) and actions.move_to_element(element) methods and both work for scrolling, but when it is time to click on the element it complains that it cannot be scrolled into view.
You can apply location_once_scrolled_into_view method to perform scrolling here.
The following code worked for me:
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
options.add_argument('--disable-notifications')
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "eager"
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, desired_capabilities=caps, service=webdriver_service)
wait = WebDriverWait(driver, 20)
url = "https://www.basketball-reference.com/players/v/valanjo01.html"
driver.get(url)
element = wait.until(EC.presence_of_element_located((By.XPATH, '//button[#id="per_poss_toggle_partial_table"]')))
element.location_once_scrolled_into_view
I am playing around with selenium on https://www.autozone.com/, trying to fill out the add vehicle form using selenium and python
first I click on the add vehicle button
URL = "https://www.autozone.com/"
ADD_VEHICLE_XPATH = "/html/body/div[1]/div/div[2]/div[2]/header/div[2]/div/div/div[2]/div/button"
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()))
driver.implicitly_wait(WAIT_TIME)
driver.get(URL)
add_vehicle_button = driver.find_element(By.XPATH, ADD_VEHICLE_XPATH)
add_vehicle_button.click()
Then a pop up window pops up, and I try to locate the button for the year dropdown
YEAR_BUTTON_XPATH = "/html/body/div[4]/div[3]/div/div[2]/div/div/div[1]/div/div[2]/div[1]/div/div/div[1]/div/button"
year_button = driver.find_element(By.XPATH, YEAR_BUTTON_XPATH)
This throws the NoSuchElementException
according to this script
def is_in_iframe():
driver.execute_script("""function iniFrame() {
if ( window.location !== window.parent.location )
{
// The page is in an iFrames
document.write("The page is in an iFrame");
}
else {
// The page is not in an iFrame
document.write("The page is not in an iFrame");
}
}
// Calling iniFrame function
iniFrame();""")
I am not in an iframe after clicking on add vehicle
I have also checked the names of all windows before and after clicking add vehicle, there is only ever 1 window and it is the same before and after clicking.
Some things to note:
I have tried adding both python sleep() and waiting in selenium
The div that contains the pop up and all buttons does not show up until I click add vehicle, and after that it shows up near the bottom
The xpath I'm using is unique to that button
What are some other ways I can try to locate the button? Please let me know if I need to add any more code or description.
You need to improve your locators.
Also you need to wait for elements to become clickable before clicking them.
The following code opens the "Add Vehicle" dialog and selects 2020 year.
from selenium import webdriver
from selenium.webdriver import DesiredCapabilities
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
options = Options()
options.add_argument("start-maximized")
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "eager"
webdriver_service = Service('C:\webdrivers\chromedriver.exe')
driver = webdriver.Chrome(options=options, desired_capabilities=caps, service=webdriver_service)
wait = WebDriverWait(driver, 10)
url = "https://www.autozone.com/"
driver.get(url)
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='deskTopVehicle-menu-lg']"))).click()
wait.until(EC.element_to_be_clickable((By.ID, "yearheader"))).click()
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, "button[data-testid='yearheader-dropdown-list-item-4']"))).click()
This is the screenshot of web page state after applying the code above:
That site containing several JavaScript scripts making the page loading time long, so I added a special settings to driver not to wait for it.
This is what
caps = DesiredCapabilities().CHROME
caps["pageLoadStrategy"] = "eager"
coming for
I want to scrape links to news articles using scrapy + selenium. The website I am using uses a 'Load more' button, so I obviously want selenium to click on this button to load all articles.
I have looked for similar questions and tried various options already such as
element = driver.find_element(By.XPATH, value='//*[#id="fusion-app"]/main/div/div/div/div/div[4]/div/div/button')
driver.execute_script("arguments[0].click();", element)
and
element = WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
ActionChains(driver).move_to_element(element).click().perform()
All to no result. I've also inserted some print statements in between to check whether it does run the code, and that seems to work fine; I think it's just a matter of the button not being located/clicked on.
This is the html of the button btw:
<button class="ais-InfiniteHits-loadMore">Load more </button>
And when I print element, this is what I get: <selenium.webdriver.remote.webelement.WebElement (session="545716eef622a12bdbeddef99e02bdef", element="551741ec-4616-4bd4-b8fd-57c2f4bffb00")>
Is someone able to help me out? Thank you in advance.
driver = webdriver.Chrome(service=Service(ChromeDriverManager().install()),options=options)
driver.maximize_window()
wait = WebDriverWait(driver, 30)
driver.get('https://www.businessoffashion.com/search/?q=Louis+Vuitton&f=Articles%2CFashion+Shows%2CNews')
wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR,'button.ab-close-button'))).click()
elem=wait.until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".ais-InfiniteHits-loadMore")))
driver.execute_script("arguments[0].click()", elem)
You hit two different errors with a pop up and an element click intereception when you can just use javascript to click that element.
Import:
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
Hello i am using selenium webdriver for chrome in python and trying to scrape some data from https://www.google.com/travel/things-to-do
I am here focused on the place decscription which can be seen here:
So in order to get to each individual description I have to press on the attraction and save the html to list for future parsing with BeautifulSoup.
Every click refresh the page so i was thinking about couting somehow all the attractions that got displayed and then in loop click every attraction with saving the description.
Anybody has any idea how to approach it?
Heres simple code that gets you to the place where i am stuck
chrome_options = webdriver.ChromeOptions()
#chrome_options.headless = True
chrome_options.add_argument('--incognito')
#chrome_options.add_argument('--headless')
s=Service(ChromeDriverManager().install())
driver = webdriver.Chrome(executable_path=r"\\chromedriver.exe", options=chrome_options, service=s)
driver.get("https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F081m_&dest_state_type=sattd&dest_src=yts&q=Warszawa#ttdm=52.227486_21.004941_13&ttdmf=%252Fm%252F0862m")
# If you are not running webdriver in incognito mode you might skip the below button since it goes through accepting cookies
button = driver.find_element_by_xpath("/html/body/c-wiz/div/div/div/div[2]/div[1]/div[4]/form/div[1]/div/button")
button.click()
time.sleep(1)
objects = driver.find_elements_by_class_name('f4hh3d')
for k in objects:
k.click()
time.sleep(5)
For each attraction index you can click it to open the details, get the details, close the details, get the list of attractions again and go for the next attraction.
Something like this should work:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver.get("https://www.google.com/travel/things-to-do/see-all?dest_mid=%2Fm%2F081m_&dest_state_type=sattd&dest_src=yts&q=Warszawa#ttdm=52.227486_21.004941_13&ttdmf=%252Fm%252F0862m")
wait = WebDriverWait(driver, 20)
wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, "div.f4hh3d")))
time.sleep(1)
attractions = driver.find_elements_by_css_selector('div.f4hh3d')
for i in range(len(attractions)):
attractions = driver.find_elements_by_css_selector('div.f4hh3d')
attractions[i].click()
description = wait.until(EC.visibility_of_element_located((By.CSS_SELECTOR, 'div[jsname="BEZjkb"] div[jsname="bN97Pc"]'))).text
#do with the description what you want
#close the attraction by clicking the button
driver.find_element_by_css_selector('div.reh1ld button')
I am trying to automate downloads from a webpage using selenium.
So far my strategy is to instantiate a firefox driver that loads the page and clicks the download button. However, to be clickable, the button needs to be visible, i.e. not covered by any banners on the page (at least in my understanding). Therefore I need to scroll down (I use scrollIntoView()). If I run button.click() immediately after scrolling down, the download doesn't start, if I hard code a sufficient timeout in between it works out fine. Can somebody help me to set a timeout conditioned on the scroll down?
Here is my code:
from selenium import webdriver
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
time.sleep(10)
button.click()
and this is the html code of the button:
<button class="hs-btn-secondary small" id="btn-download" type="button">Download</button>
I would be very thankful for any direct help or suggestions on how to tackle the problem from a different angle.
It sounds like you may need to induce WebDriverWait for the element to by clickable. You could do so like this. It will wait for the element to be both visible and enabled before clicking.
from selenium import webdriver
import time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
profilePath = '/path/to/my/firefox/profile'
profile = webdriver.FirefoxProfile(profilePath)
driver = webdriver.Firefox(firefox_profile=profile)
driver.get("https://www.happyscribe.com/public/lex-fridman-podcast-artificial-intelligence-ai/164-andrew-huberman-sleep-dreams-creativity-the-limits-of-the-human-mind")
button = driver.find_element_by_id('btn-download')
target=driver.execute_script("arguments[0].scrollIntoView({block: 'center'});", button)
WebDriverWait(driver, 10).until(EC.element_to_be_clickable(button))
button.click()