I've written a script in python in association with selenium to click on some dots available on a map in a web page. When a dot is clicked, a small box containing relevant information pops up.
Link to that site
I would like to parse the title of each box. When I execute my script, it throws an error while clicking on a dot. How can I make a go successfully?
This is the script:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
link = "replace with above link"
driver = webdriver.Chrome()
driver.get(link)
wait = WebDriverWait(driver, 10)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"#mapDiv_zoom_slider .esriSimpleSliderIncrementButton"))).click()
for item in wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR,"#NWQMC_VM_directory_June2016_3915_0_layer circle"))):
item.click()
Error I'm having:
line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Element <circle fill="rgb(237, 81, 81)" fill-opacity="1" stroke="rgb(153, 153, 153)" stroke-opacity="1" stroke-width="1" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="4" cx="720" cy="430" r="4" transform="matrix(1.00000000,0.00000000,0.00000000,1.00000000,0.00000000,0.00000000)" fill-rule="evenodd" stroke-dasharray="none" dojoGfxStrokeStyle="solid"></circle> is not clickable at point (720, 430). Other element would receive the click: <circle fill="rgb(20, 158, 206)" fill-opacity="1" stroke="rgb(153, 153, 153)" stroke-opacity="1" stroke-width="1" stroke-linecap="butt" stroke-linejoin="miter" stroke-miterlimit="4" cx="720" cy="430" r="4" transform="matrix(1.00000000,0.00000000,0.00000000,1.00000000,0.00000000,0.00000000)" fill-rule="evenodd" stroke-dasharray="none" dojoGfxStrokeStyle="solid"></circle>
(Session info: chrome=67.0.3396.99)
(Driver info: chromedriver=2.36.540470 (e522d04694c7ebea4ba8821272dbef4f9b818c91),platform=Windows NT 6.1.7601 SP1 x86)
This is how a box gets popped up:
#Andrei Suvorkov was really close (+1)
Try below code to get required output:
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome()
driver.get(link)
wait = WebDriverWait(driver, 5)
driver.maximize_window()
items = wait.until(EC.presence_of_all_elements_located((By.XPATH, "//*[name()='svg']//*[name()='circle']")))
for item in items:
ActionChains(driver).move_to_element(item).click(item).perform()
popup = wait.until(EC.visibility_of_element_located((By.CLASS_NAME, "contentPane")))
print(popup.text)
wait.until(EC.visibility_of_element_located((By.XPATH, "//div[contains(#class, 'close')]"))).click()
To answer your particular question you cannot interact with svg elements like usual. For that you have to use xPath like I have provided in the example. Also you cannot click on these elements like usual, but you can use ActionChains.
wait = WebDriverWait(driver, 5)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR,"#mapDiv_zoom_slider .esriSimpleSliderIncrementButton"))).click()
time.sleep(3)
items = driver.find_elements_by_xpath("//*[name()='svg']//*[name()='circle']")
i = 0
for item in items:
try:
time.sleep(1)
ActionChains(driver).move_to_element(item).click(item).perform()
except Exception:
print("Can't click")
This code works and every element will be clicked until the map will be zoomed in. At one of the elements the map zooms in and it doesn't work after that. Why? I didn't found out yet, but you can find it from yourself or ask another question and we will try to help you.
Note: you have to add some imports:
from selenium.webdriver.common.action_chains import ActionChains
import time
EDIT: I have found the problem, you have to close every popup after click and then it works. The working code is below:
wait = WebDriverWait(driver, 5)
wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, "#mapDiv_zoom_slider .esriSimpleSliderIncrementButton"))).click()
time.sleep(3) # wait until all elements will be ready
items = driver.find_elements_by_xpath("//*[name()='svg']//*[name()='circle']")
for item in items:
time.sleep(0.5) # small pause before each iteration
ActionChains(driver).move_to_element(item).click(item).perform()
wait.until(EC.element_to_be_clickable((By.XPATH, "//div[#title = 'Close']"))).click()
I didn't find a way to avoid time.sleep(), probably in this particular case it is not possible.
Related
I have a script that uses Selenium (Python).
I tried to make the code click a button that it acknowledges is clickable, but throws an error stating it;s not clickable.
Same thing happens again in a dropdown menu, but this time I'm not clicking, but selecting an option by value.
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
from selenium.webdriver.common.by import By
from selenium.webdriver.support.select import *
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# Getting to Chrome and website
website = 'https://www.padron.gob.ar/publica/'
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(website)
driver.maximize_window()
#"BUENOS AIRES" in "Distrito Electoral"
distritoElectoralOptions = driver.find_element(By.NAME, 'site')
Select(distritoElectoralOptions).select_by_value('02 ')
#Clicking "Consulta por Zona
WebDriverWait(driver, 35).until(EC.element_to_be_clickable((By.ID, 'lired')))
consultaPorZona = driver.find_element(By.ID, 'lired')
consultaPorZona.click()
#"SEC_8" in "Sección General Electoral"
WebDriverWait(driver, 35).until(EC.visibility_of((By.NAME, 'secg')))
seccionGeneralElectoral = driver.find_element(By.NAME, 'secg')
Select(seccionGeneralElectoral).select_by_value('00008')
I'm getting this error on line 21:
selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable: element has zero size
It works in a ipython notebook if each section is separated, but only if the option "Run all" is not used. Instead, the kernel has to be run on it's own.
I'm using VS Code.
Also, when it reaches the last line, when run in ipynb format, it throws this error:
Message: Cannot locate option with value: 00008
Thank you in advance.
When a web element is present in the HTML-DOM but it is not in the state that can be interacted. Other words, when the element is found but we can’t interact with it, it throws ElementNotInteractableException.
The element not interactable exception may occur due to various reasons.
Element is not visible
Element is present off-screen (After scrolling down it will display)
Element is present behind any other element
Element is disabled
If the element is not visible then wait until element is visible. For this we will use wait command in selenium
wait = WebDriverWait(driver, 10)
element = wait.until(EC.element_to_be_clickable((By.ID, 'someid')))
element.click()
If the element is off-screen then we need to scroll down the browser and interact with the element.
Use the execute_script() interface that helps to execute JavaScript methods through Selenium Webdriver.
browser = webdriver.Firefox()
browser.get("https://en.wikipedia.org")
browser.execute_script("window.scrollTo(0,1000)") //scroll 1000 pixel vertical
Reference this solution in to the regarding problem.
I'm pretty new to selenium. I'm working on a autocheckout scrip using selenium python. Script executes very smooth till the final checkout page. However, I'm unable to access element on the final page('Razorpay Checkout'), which is a kind of pop up/iframe window.
Any help accessing the same would be highly appreciated.
Thanks in advance!
Below is the code I'm using
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('C:/Users/hamhatre/Desktop/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
driver.implicitly_wait(20)
elem = driver.find_element_by_xpath('//*[#id="pincode_input"]')
elem.send_keys("400708")
driver.find_element_by_xpath('//*[#id="check-delivery-submit"]').click()
driver.find_element_by_xpath('/html/body/div[2]/main/div[1]/div/div/div/div[2]/div[1]/form/div[3]/div/button[2]').click()
driver.find_element_by_xpath('//*[#id="checkout_email"]').send_keys('abc#gmail.com')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_first_name"]').send_keys('abc')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_last_name"]').send_keys('xyz')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_address1"]').send_keys('Rd1, Flat no 2, Apt 1')
driver.find_element_by_xpath(('//*[#id="checkout_shipping_address_address2"]')).send_keys('Nothing')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_city"]').send_keys('Mumbai')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_province"]').send_keys('Maharashtra')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_zip"]').send_keys('400708')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_phone_custom"]').send_keys('9876543210')
driver.find_element_by_id('continue_to_shipping_button_custom').click()
driver.find_element_by_id('continue_button').click()
driver.find_element_by_xpath('/html/body/div/div/div/main/div[1]/div/form/div[3]/div[1]/button').click()
print(driver.title)
seq = driver.find_elements_by_tag_name('iframe')
print(seq)
print("No of frames present in the web page are: ", len(seq))
for index in range(len(seq)):
iframe = driver.find_elements_by_tag_name('iframe')[index]
driver.switch_to.frame(iframe)
driver.find_element_by_id('language-dropdown').click()
Below is the error
Traceback (most recent call last):
File "C:\Users\hamhatre\Desktop\Algotron\WebScrap_Sample.py", line 47, in <module>
driver.find_element_by_id('language-dropdown').click()
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\hamhatre\Anaconda3\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
NoSuchElementException: no such element: Unable to locate element: {"method":"css selector","selector":"[id="language-dropdown"]"}
(Session info: chrome=99.0.4844.84)
I am using chromedriver version 100. Your code throws a NoSuchElementException. Thats because your driver tries to find an element which does not exist. So, what you need to do is to put time.sleep(4) statements after your click() statements (Your browser needs time to open the window/s). If you are done using the driver, use driver.quit().
Followed code worked for me:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver = webdriver.Chrome('C:/Users/hamhatre/Desktop/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
driver.implicitly_wait(20)
elem = driver.find_element_by_xpath('//*[#id="pincode_input"]')
elem.send_keys("400708")
driver.find_element_by_xpath('//*[#id="check-delivery-submit"]').click()
driver.find_element_by_xpath('/html/body/div[2]/main/div[1]/div/div/div/div[2]/div[1]/form/div[3]/div/button[2]').click()
driver.find_element_by_xpath('//*[#id="checkout_email"]').send_keys('abc#gmail.com')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_first_name"]').send_keys('abc')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_last_name"]').send_keys('xyz')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_address1"]').send_keys('Rd1, Flat no 2, Apt 1')
driver.find_element_by_xpath(('//*[#id="checkout_shipping_address_address2"]')).send_keys('Nothing')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_city"]').send_keys('Mumbai')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_province"]').send_keys('Maharashtra')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_zip"]').send_keys('400708')
driver.find_element_by_xpath('//*[#id="checkout_shipping_address_phone_custom"]').send_keys('9876543210')
driver.find_element_by_id('continue_to_shipping_button_custom').click()
time.sleep(4)
driver.find_element_by_id('continue_button').click()
time.sleep(4)
driver.find_element_by_xpath('/html/body/div/div/div/main/div[1]/div/form/div[3]/div[1]/button').click()
time.sleep(4)
print(driver.title)
seq = driver.find_elements_by_tag_name('iframe')
print(seq)
print("No of frames present in the web page are: ", len(seq))
for index in range(len(seq)):
iframe = driver.find_elements_by_tag_name('iframe')[index]
driver.switch_to.frame(iframe)
driver.find_element_by_id('language-dropdown').click()
A few things you could do in your next project:
Use WebDriverWait to allow enough time for elements to be loaded
Use switch_to.default_content() before switching to another frame
Use time.sleep() while you switch frames
Try using relative xpaths (ex: //div[#id="user_address"]). It's easy to right click and copy the absolute xpath for an element from the inspect window, however if there's another element added in between, your xpath would no longer point to the desired element. Moreover, relative xpaths would empower you with a deeper understanding allowing you to easily find elements.
This code should work:
from selenium import webdriver
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.common.exceptions import StaleElementReferenceException
import time
driver = webdriver.Chrome('D://chromedriver/100/chromedriver.exe')
driver.get("https://shopatsc.com/collections/playstation-5/products/ps5-horizon-forbidden-west")
driver.maximize_window()
wait = WebDriverWait(driver, 20)
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="pincode_input"]'))).send_keys('400708')
wait.until(EC.presence_of_element_located((By.XPATH, '//span[#id="check-delivery-submit"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//button[normalize-space(text())="Buy Now"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_email"]'))).send_keys('abc#gmail.com')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_first_name"]'))).send_keys('abc')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_last_name"]'))).send_keys('xyz')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_address1"]'))).send_keys('Rd1, Flat no 2, Apt 1')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_address2"]'))).send_keys('Nothing')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_city"]'))).send_keys('Mumbai')
wait.until(EC.presence_of_element_located((By.XPATH, '//select[#id="checkout_shipping_address_province"]'))).send_keys('Maharashtra')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_zip"]'))).send_keys('400708')
wait.until(EC.presence_of_element_located((By.XPATH, '//input[#id="checkout_shipping_address_phone_custom"]'))).send_keys('9876543210')
wait.until(EC.presence_of_element_located((By.XPATH, '//button[#id="continue_to_shipping_button_custom"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//button[#id="continue_button"]'))).click()
wait.until(EC.presence_of_element_located((By.XPATH, '//span[normalize-space(text())="Complete order"]/parent::button'))).click()
print(driver.title)
# finding all iframes
iframes = wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, 'iframe')))
print(f'No. of iframes: {len(iframes)}')
print('Looping through frames to check language dropdown')
found = False
frameno = 0
tries = 1
maxtries = 5
# Attempt at least 5 times to click on the language dropdown
while not found and frameno < len(iframes) and tries <= maxtries:
try:
print(f"Frame# {frameno} \t {tries} of {maxtries} tries")
print('Switching to frame')
driver.switch_to.frame(iframes[frameno])
time.sleep(2)
wait.until(EC.presence_of_element_located((By.XPATH, '//div[#id="language-dropdown"]/div/div'))).click()
found = True
print('Found and clicked')
except StaleElementReferenceException:
print("Exception occured. Capturing iframes again")
driver.switch_to.default_content()
iframes = wait.until(EC.presence_of_all_elements_located((By.TAG_NAME, 'iframe')))
except Exception as ex:
print("Couldn't find/click on the language dropdown")
frameno += 1
finally:
print('Switching to default content')
driver.switch_to.default_content()
time.sleep(2)
tries += 1
I want to scrape news headlines from this link: https://www.newsnow.co.uk/h/Business+&+Finance?type=ln
I want to expand news by clicking (using selenium) on the button 'view more headlines' to collect the max number of news headlines possible
I created this code but failed to make the click to expand news :
import time
from selenium import webdriver
u = 'https://www.newsnow.co.uk/h/Business+&+Finance?type=ln'
driver = webdriver.Chrome(executable_path=r"C:\chromedriver.exe")
driver.get(u)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
driver.implicitly_wait(60) # seconds
elem = driver.find_element_by_css_selector('span:contains("view more headlines")')
for i in range(10):
elem.click()
time.sleep(5)
print(f'click {i} done')
returns: selenium.common.exceptions.InvalidSelectorException: Message: invalid selector: An invalid or illegal selector was specified
I tried using xpath selector:
elem = driver.find_element_by_xpath('//[#id="nn_container"]/div[2]/main/div[2]/div/div/div[3]/div/a')
returns: selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <a class="rs-button-more js-button-more btn--primary btn--primary--no-spacing" href="#">...</a> is not clickable at point (353, 551). Other element would receive the click: <div class="alerts-scroller">...</div>
The click button gets covered by an overlay element after the click. So, we use javascript to get to it after the first click. Here is the working program.
import time
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
u = 'https://www.newsnow.co.uk/h/Business+&+Finance?type=ln'
driver = webdriver.Chrome(executable_path=r"C:\bin\chromedriver.exe")
driver.maximize_window()
driver.get(u)
time.sleep(10)
driver.execute_script("window.scrollTo(0, document.body.scrollHeight)")
for i in range(10):
element =WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CLASS_NAME,'btn--primary__label')))
driver.execute_script("arguments[0].scrollIntoView();", element)
element.click()
time.sleep(5)
print(f'click {i} done')
This one is the correct XPath:
driver.find_element_by_xpath(r'//*[#id="nn_container"]/div[2]/main/div[2]/div/div/div[3]/div/a').click()
I am performing an automation bot, capable of sharing post. However, when it comes to performing the task more than 3 times, I get an error and program stops working. I am not sure why my code is able to perform the task 3 times and then stops.
Here is my code:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ECDS
import time
driver = webdriver.Chrome()
driver.get("https://www.poshmark.com") #Open webpage
Log_Field=(By.XPATH, "//a[contains(text(),'Log in')]")
Email= (By.XPATH, "//input[#placeholder='Username or Email']")
Pass= (By.XPATH, "//input[#placeholder='Password']")
Second_Log= (By.XPATH, "//button[#class='btn btn--primary']")
SF = (By.XPATH, "//img[#class='user-image user-image--s']")
MyCloset = (By.XPATH, "//a[contains(text(),'My Closet')]")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(Log_Field)).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(Email)).send_keys("xx#xx.com")
driver.find_element_by_xpath("//input[#placeholder='Password']").send_keys("xxx")
driver.find_element_by_xpath("//button[#class='btn blue btn-primary']").click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(SF)).click()
WebDriverWait(driver, 20).until(EC.element_to_be_clickable(MyCloset)).click()
for i in range(100):
driver.find_element_by_tag_name('body').send_keys(Keys.END)#Use send_keys(Keys.HOME)
driver.find_element_by_xpath("//div[6]//div[1]//div[2]//div[3]//i[1]").click()
driver.find_element_by_xpath("//div[#class='share-wrapper-container']").click()
driver.refresh()
time.sleep(20)
The error that I am getting is the following:
Traceback (most recent call last):
File "/home/pi/Documents/Bot_Poshmark.py", line 27, in <module>
driver.find_element_by_xpath("//div[6]//div[1]//div[2]//div[3]//i[1]").click()
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <i class="icon share-gray-large"></i> is not clickable at point (870, 163). Other element would receive the click: <div class="tile col-x12 col-l6 col-s8 p--2">...</div>
(Session info: chrome=78.0.3904.108)
Any ideas why my code is only working not more than 3 times?
Thank you
I had the same problem several times while testing my selenium web automation. As the exception tells, this object is not clickable. That means you have to dive deeper into the HTML tree to find an element that is always clickable. If you hover over the HTML-lines Chrome shows you the according piece of website.
However, if this is not possible, try to let your code sleep() for a bit :-)
You could do that for a certain amount of time
Or you use WebDriverWait() as in the comments described:
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "your XPATH"))).click()
(You missed one (), this should remove your error: Without: Argument 1 = self [Keep that in mind with Python!!], Argument 2 = By.XPATH and 3 = "the xpath". With the (), Argument 2 and 3 are together)
WebDriverWait() requires a timeout parameter because selenium does not know whether the element exists or not. But you could easily create your own waiting-method. Pay attention: you have to know that the element exists or you will end up with an infinite loop.
Here's the code:
from time import sleep
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
def wait_for(driver, method):
"""Calls the method provided with the driver"""
while True:
try:
element = method(driver)
if element:
return element
except:
pass
sleep(0.5)
It tries to find the element and when it's found, it returns it.
You can use it like so:
el = wait_for(driver, EC.element_to_be_clickable((By.XPATH, "//input[#class='13e44'")))
el.click()
Disclaimer: This code is not fully my creation. I adapted the selenium source-code so it fits our needs :)
I am having trouble selecting a load more button on a Linkedin page. I receive this error in finding the xpath: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element
I suspect that the issue is that the button is not visible on the page at that time. So I have tried actions.move_to_element. However, the page scrolls just below the element, so that the element is no longer visible, and the same error subsequently occurs.
I have also tried move_to_element_with_offset, but this hasn't changed where the page scrolls to.
How can I scroll to the right location on the page such that I can successfully select the element?
My relevant code:
import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('C:\\Users\\Root\\Downloads\\chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys(parameters.linkedin_username)
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys(parameters.linkedin_password)
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
After playing around with it, I seem to have figured out where the problem is stemming from. The error
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]"}
(Session info: chrome=81.0.4044.113)
always correctly states the problem its encountering and as such it's not able to find the element. The possible causes of this include:
Element not present at the time of execution
Dynamically generated
content Conflicting names
In your case, it was the second point. As the content that is displayed is loaded dynamically as you scroll down. So When it first loads your profile the skills sections aren't actually present in the DOM. So to solve this, you simply have to scroll to the section so that it gets applied in the DOM.
This line is the trick here. It will position it to the correct panel and thus loading and applying the data to the DOM.
driver.execute_script("window.scrollTo(0, 1800)")
Here's my code (Please change it as necessary)
from time import sleep
# import parameters
from selenium.webdriver.common.action_chains import ActionChains
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
ChromeOptions = webdriver.ChromeOptions()
driver = webdriver.Chrome('../chromedriver.exe')
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
sleep(0.5)
username = driver.find_element_by_name('session_key')
username.send_keys('')
sleep(0.5)
password = driver.find_element_by_name('session_password')
password.send_keys('')
sleep(0.5)
sign_in_button = driver.find_element_by_xpath('//button[#class="btn__primary--large from__button--floating"]')
sign_in_button.click()
driver.get('https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk')
sleep(3)
# driver.execute_script("window.scrollTo(0, 1800)")
sleep(3)
loadmore_skills=driver.find_element_by_xpath('//button[#class="pv-profile-section__card-action-bar pv-skills-section__additional-skills artdeco-container-card-action-bar artdeco-button artdeco-button--tertiary artdeco-button--3 artdeco-button--fluid"]')
actions = ActionChains(driver)
actions.move_to_element(loadmore_skills).perform()
#actions.move_to_element_with_offset(loadmore_skills, 0, 0).perform()
loadmore_skills.click()
Output
Update
In concerns to your newer problem, you need to implement a continuous scroll method that would enable you to dynamically update the skills section. This requires a lot of change and should ideally be asked as a another question.
I have also found a simple solution by setting the scroll to the correct threshold. For y=3200 seems to work fine for all the profiles I've checked including yours, mine and few others.
driver.execute_script("window.scrollTo(0, 3200)")
If the button is not visible on the page at the time of loading then use the until method to delay the execution
try:
myElem = WebDriverWait(browser, delay).until(EC.presence_of_element_located((By.ID, 'IdOfMyElement')))
print "Button is rdy!"
except TimeoutException:
print "Loading took too much time!"
Example is taken from here
To get the exact location of the element, you can use the following method to do so.
element = driver.find_element_by_id('some_id')
element.location_once_scrolled_into_view
This actually intends to return you the coordinates (x, y) of the element on-page, but also scroll down right to target element. You can then use the coordinates to make a click on the button. You can read more on that here.
You are getting NoSuchElementException error when the locators (i.e. id / xpath/name/class_name/css selectors etc) we mentioned in the selenium program code is unable to find the web element on the web page.
How to resolve NoSuchElementException:
Apply WebDriverWait : allow webdriver to wait for a specific time
Try catch block
so before performing action on webelement you need to take web element into view, I have removed unwated code and also avoided use of hardcoded waits as its not good practice to deal with synchronization issue. Also while clicking on show more button you have to scroll down otherwise it will not work.
from selenium import webdriver
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.common.action_chains import ActionChains
driver = webdriver.Chrome(executable_path="path of chromedriver.exe")
driver.get('https://www.linkedin.com/login?fromSignIn=true&trk=guest_homepage-basic_nav-header-signin')
driver.maximize_window()
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_key"))).send_keys("email id")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.NAME, "session_password"))).send_keys("password ")
WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//button[#class='btn__primary--large from__button--floating']"))).click()
driver.get("https://www.linkedin.com/in/kate-yun-yi-wang-054977127/?originalSubdomain=hk")
driver.maximize_window()
driver.execute_script("scroll(0, 250);")
buttonClick = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, "//span[text()='Show more']")))
ActionChains(driver).move_to_element(buttonClick).click().perform()
Output: