I have a site that has multiple input fields containing URLs with the same value (X) that I want to replace with (Y). Basically, I want to automate replacing URL X with URL Y for all occurrences of URL X. There is an edit and save button to the right of each of these input fields. And each time you click save an alert pops up for each field you update. This is in short, the entire workflow I need to automate.
I'm at the point in my code that I can do it successfully for one field, but when I use a for loop it just updates one element/field twice and not the rest. FYI - I limited my for loop to two iterations using itertools.islice intentionally since I want to limit my testing to a small subset before I do a full run.
Below is a screenshot of what the page looks like:
HTML
New error:
C:\Users\me\OneDrive\Documents\Python\Web page entry automation\preferences_v3 copy.py:21: DeprecationWarning: use options instead of chrome_options chrome_browser = webdriver.Chrome(service=service, chrome_options=chrome_options)
DevTools listening on ws://127.0.0.1:61787/devtools/browser/155908c1-8be2-40a1-85bf-27c44bf25506 Traceback (most recent call last): File "C:\Users\me\OneDrive\Documents\Python\Web page entry automation\preferences_v3 copy.py", line 90, in <module>
auto_ftp_url() File "C:\Users\me\OneDrive\Documents\Python\Web page entry automation\preferences_v3 copy.py", line 57, in auto_ftp_url
number = elem.get_attribute("id").split("_")[-1]
^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\me\AppData\Roaming\Python\Python311\site-packages\selenium\webdriver\remote\webelement.py", line 177, in get_attribute
attribute_value = self.parent.execute_script(
^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\me\AppData\Roaming\Python\Python311\site-packages\selenium\webdriver\remote\webdriver.py", line 500, in execute_script
return self.execute(command, {"script": script, "args": converted_args})["value"]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\me\AppData\Roaming\Python\Python311\site-packages\selenium\webdriver\remote\webdriver.py", line 440, in execute
self.error_handler.check_response(response) File "C:\Users\me\AppData\Roaming\Python\Python311\site-packages\selenium\webdriver\remote\errorhandler.py", line 245, in check_response
raise exception_class(message, screen, stacktrace) selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document (Session info: chrome=109.0.5414.120) Stacktrace: Backtrace:
(No symbol) [0x00506643]
(No symbol) [0x0049BE21]
(No symbol) [0x0039DA9D]
(No symbol) [0x003A09E4]
(No symbol) [0x003A08AD]
(No symbol) [0x003A12E5]
(No symbol) [0x004079F7]
(No symbol) [0x003EFD7C]
(No symbol) [0x00406B09]
(No symbol) [0x003EFB76]
(No symbol) [0x003C49C1]
(No symbol) [0x003C5E5D]
GetHandleVerifier [0x0077A142+2497106]
GetHandleVerifier [0x007A85D3+2686691]
GetHandleVerifier [0x007ABB9C+2700460]
GetHandleVerifier [0x005B3B10+635936]
(No symbol) [0x004A4A1F]
(No symbol) [0x004AA418]
(No symbol) [0x004AA505]
(No symbol) [0x004B508B]
BaseThreadInitThunk [0x771300F9+25]
RtlGetAppContainerNamedObjectPath [0x77847BBE+286]
RtlGetAppContainerNamedObjectPath [0x77847B8E+238]
PS C:\Users\me\OneDrive\Documents\Python\Web page entry automation> [15148:25476:0201/123239.502:ERROR:device_event_log_impl.cc(215)] [12:32:39.502] USB: usb_service_win.cc:415 Could not read device interface GUIDs: The system cannot find the file specified. (0x2) [15148:25476:0201/123239.557:ERROR:device_event_log_impl.cc(215)] [12:32:39.557] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F) [15148:25476:0201/123239.575:ERROR:device_event_log_impl.cc(215)] [12:32:39.575] USB: usb_device_handle_win.cc:1046 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
Below is my code:
from selenium import webdriver
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.common.by import By
from selenium.webdriver.common.alert import Alert
import itertools
def auto_ftp_url():
username, password = login()
chrome_options = Options()
chrome_options.add_experimental_option("detach", True)
service = Service(
r"C:\Users\me\OneDrive\Documents\chromedriver_win32\chromedriver.exe"
)
chrome_browser = webdriver.Chrome(service=service, chrome_options=chrome_options)
chrome_browser.get("https://somelink")
# send text to username field
userElem = chrome_browser.find_element(by="id", value="Username")
userElem.click()
userElem.send_keys(f"{username}")
# send password to password field
passwordElem = chrome_browser.find_element(by="id", value="Password")
passwordElem.click()
passwordElem.send_keys(f"{password}")
# click login
loginElem = chrome_browser.find_element(by="id", value="Button1")
loginElem.click()
# add some delay (2 seconds) to the webdriver so that the elements will load
chrome_browser.implicitly_wait(2)
# click on Admin button on left navigation pane
adminElem = chrome_browser.find_element(By.XPATH, "//*[#id='btnAdminPage']")
adminElem.click()
# click on Data FTP button in the top center
data_ftp_Elem = chrome_browser.find_element(
By.XPATH, "//*[#id='menu-outer']/div[12]/a/div"
)
data_ftp_Elem.click()
# find FTP URL with a value of 'aftersoftna.com'
ftp_url_Elem = chrome_browser.find_elements(
By.XPATH, "//*[#value='aftersoftna.com']"
)
for elem in itertools.islice(ftp_url_Elem, 2):
if elem: # if the value is 'aftersoftna.com' then do the following
# click edit button
editElem = chrome_browser.find_element(
By.XPATH, "//*[#id='ContentPlaceHolder1_gvFTP_btnSave_7']"
)
editElem.click()
# use WebDriverWait to wait for the element to be clickable
textElem = WebDriverWait(chrome_browser, 20).until(
EC.element_to_be_clickable(
(By.XPATH, "//*[#id='ContentPlaceHolder1_gvFTP_txtFtpUrl_7']")
)
)
textElem.click()
textElem.clear()
textElem.send_keys(
"catalog.com"
) # after clearing the old URL, enter the new URL
# click save
saveElem = chrome_browser.find_element(
By.XPATH, "//*[#id='ContentPlaceHolder1_gvFTP_btnSave_7']"
)
saveElem.click()
# popup or alert message comes up confirming changes saved. click ok
WebDriverWait(chrome_browser, 10).until(EC.alert_is_present())
chrome_browser.switch_to.alert.accept()
def login():
username = input("Username: ")
password = input("Password: ")
return username, password
auto_ftp_url()
Without seeing the page it's impossible to know but I'm going to guess that it's because you are using the same locators for editElem, textElem, and saveElem for each URL. Each of them end in "_7" but I would assume that there are multiple elements on the page, one set of three for each URL from what you are describing, e.g. "_8", "_9", etc. If this is true then your locators need to be relative to the URL that you want to edit.
If I were to guess even further, you should be able to get your list of ftp_url_Elem elements, grab the ID from each in the loop, and then extract the final number, e.g. 7 from ContentPlaceHolder1_gvFTP_txtFtpUrl_7, and then extrapolate that to the other ID, ContentPlaceHolder1_gvFTP_btnSave_7.
To get the magic number that corresponds to the row number, we can
Grab the first ID using elem.get_attribute("id"), ContentPlaceHolder1_gvFTP_txtFtpUrl_7
Split that ID by "_" resulting in ['ContentPlaceHolder1','gvFTP','txtFtpUrl','7']
Grab the last instance from that list which is the desired row number, '7'
Now we can append this number to the "prefix" of the ID for the other elements to get the correct element on the matching row of the table, e.g. ID prefix for the Edit/Save button, ContentPlaceHolder1_gvFTP_btnSave_, and add '7' to get ContentPlaceHolder1_gvFTP_btnSave_7
Then you just repeat that last step for all the elements on that row that you need to click.
Here's all the code as an example
for elem in itertools.islice(ftp_url_Elem, 2):
if elem: # if the value is 'aftersoftna.com' then do the following
# get row number from element ID
row_number = elem.get_attribute("id").split("_")[-1]
# click edit button
editElem = chrome_browser.find_element(
By.XPATH, "//*[#id='ContentPlaceHolder1_gvFTP_btnSave_" + row_number + "']"
)
... and so on
If this is enough to point you in the right direction to fix it yourself, great. If not, please post a few example rows of HTML that contain the edit, URL field, and save button so we can create code and relative locators.
Related
i'm trying to make a program that as the first step must login on site edostavka.by having phone and password. If I use "https://edostavka.by/login?screen=withPassword" - program still redirected on "https://edostavka.by/login?screen=phone". Luckily there is button "Войти с паролем", which should open "https://edostavka.by/login?screen=withPassword" page. But selenium can't find it.
My code here:
from selenium import webdriver
import time
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
from fake_useragent import UserAgent as ua
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
ua = ua(browsers='chrome')
driver_service = Service(executable_path=r'C:\Users\37533\Desktop\dostavka\dostavka\chromedriver\chromedriver.exe')
options = webdriver.ChromeOptions()
options.add_argument(f'user-agent={ua}')
driver = webdriver.Chrome(service=driver_service)
driver.maximize_window()
#url = 'https://edostavka.by/login?screen=withPassword'
url = 'https://edostavka.by/login?screen=phone'
try:
driver.get(url=url)
time.sleep(3)
driver.find_element(By.NAME, 'phone').send_keys('132456789')
driver.find_element(By.LINK_TEXT, 'Войти с паролем').send_keys(Keys.ENTER)
time.sleep(5)
except Exception as e:
print(e)
finally:
driver.close()
driver.quit()
It was no matter how i tried to find that button(through XPATH, CLASS_NAME, or LINK_TEXT) - none of this worked. I tried put WebDriverWait and EC instead of time.sleep, but it didn't help aswell as swapping "click" methods(I tried 'click()', 'submit()', 'send_keys(Keys.ENTER or \n).
Also, I figure that problem might be in iframe or scripts. But for iframe - there are 4 on the page and all of them seems to be empty. For scripts - I still didn't find answer: what and should I do?
Had to mention that it can find and even fill line with "phone number"
Every time i run the code i get that message
Message: no such element: Unable to locate element: {"method":"link text","selector":"Войти с паролем"}
(Session info: chrome=108.0.5359.125)
Stacktrace:
Backtrace:
(No symbol) [0x00FEF243]
(No symbol) [0x00F77FD1]
(No symbol) [0x00E6D04D]
(No symbol) [0x00E9C0B0]
(No symbol) [0x00E9C22B]
(No symbol) [0x00ECE612]
(No symbol) [0x00EB85D4]
(No symbol) [0x00ECC9EB]
(No symbol) [0x00EB8386]
(No symbol) [0x00E9163C]
(No symbol) [0x00E9269D]
GetHandleVerifier [0x01289A22+2655074]
GetHandleVerifier [0x0127CA24+2601828]
GetHandleVerifier [0x01098C0A+619850]
GetHandleVerifier [0x01097830+614768]
(No symbol) [0x00F805FC]
(No symbol) [0x00F85968]
(No symbol) [0x00F85A55]
(No symbol) [0x00F9051B]
BaseThreadInitThunk [0x771800F9+25]
RtlGetAppContainerNamedObjectPath [0x779C7BBE+286]
RtlGetAppContainerNamedObjectPath [0x779C7B8E+238]
Maybe my question easy and newbie, but i tried to fix this problem over six-seven hours and still didn't find the way of solving
First of read the selenium documentation. Then use Chrome Developer tools (you may call by pressing F12 or Ctrl+Shift+I) to inspect target page elements.
Below I provide an example solution for your case:
# accept coockies
driver.find_element(By.XPATH, "//button[#class='button button_size_medium button_type_subtle accept-cookies_accept__mok-Y']").click()
time.sleep(2)
# change login mode to with password
driver.find_element(By.XPATH, "//button[#class='button button_size_large button_type_subtle phone_button__2Z9fs phone_button__padding__e0VFd']").click()
time.sleep(2)
# type phone number
driver.find_element(By.XPATH, "//input[#class='index-module_wrapperPhone__input__KD5gF']").send_keys('132456789')
time.sleep(2)
# type password
driver.find_element(By.XPATH, "//input[#class='input__input input__input_size_medium']").send_keys('Password')
time.sleep(2)
# click to eye icon
driver.find_element(By.XPATH, "//button[#class='withPassword_login__form_password_button__3Zrhk']").click()
time.sleep(2)
This is how to use XPath
I'm writing a function that opens WhitePages, searches a person's name and location, and scrapes their phone number and address. It does this by:
Navigating to whitepages.com
Finding the name <input> and sending it keys (send_keys(persons_name))
Finding the location <input> and sending it keys (send_keys(my_city))
Finding the search button <button> and clicking it
On the search results page, finding the link <a> to the person's page
On the person's page, finding and returning the person's landline and address
When I run the function in a loop on a list of names, the function runs successfully on the first iteration, but not the second. For testing purposes, I'm running the WebDriver with a head/GUI so that I can verify what is going on, and it seems as though on the second iteration, the function successfully finds the name <input> but doesn't input the person's name via send_keys(), then successfully finds the location <input> and successfully inputs the location, then successfully finds and click()s the search button.
Since there must be a name in the name <input> for a search to be done, no search occurs and red text under the name <input> appears saying "Last name is required" (that's how I know for sure send_keys() is failing), and then I get a NoSuchElementException when the program tries to find a search result element that doesn't exist since no search results page was loaded.
(Note: by default, WhitePages denies access to the program when trying to hit search; the options.add_argument('--disable-blink-features=AutomationControlled') in the code below circumvents that.)
So, what may be happening that is causing send_keys() to fail, and how do I fix it?
Full code:
from selenium import webdriver
# for passing a URL as a service object into the Chrome webdriver initializing method
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
# for clicking buttons
from selenium.webdriver.common.action_chains import ActionChains
# raised when using find_element() and no element exists matching the given criteria
from selenium.common.exceptions import NoSuchElementException
# for specifying to run the browser headless (w/o UI) and to surpress warnings in console output
from selenium.webdriver.chrome.options import Options
# for choosing an option from a dropdown menu
from selenium.webdriver.support.select import Select
def scrape_individual_info_wp(driver, individual_name, city_state):
# FIND INDIVIDUAL ON WHITEPAGES & NAVIGATE TO THEIR INDIVIDUAL PAGE
driver.get('https://www.whitepages.com/')
# find name input
driver.find_element(By.XPATH, "//form/div/div/div/div/input").send_keys(individual_name) # attempt to find the input *relatively*
# find location input
driver.find_element(By.XPATH, "//form/div/div/following-sibling::div/div/div/input").send_keys(city_state)
# find & click search button
driver.find_element(By.XPATH, "//form/div/div/button").click()
# FIND INDIVIDUAL IN SEARCH RESULTS
# click (first) free search result link
driver.find_element(By.XPATH, "//div[#class='results-container']/a").click()
# SCRAPE PERSON'S INFO
landline = driver.find_element(By.XPATH, "//div[contains(text(),'Landlines')]/following-sibling::div/a").text.strip()
address_info = driver.find_element(By.XPATH, "//p[contains(text(),'Current Address')]/parent::div/div/div/div/a").text.strip().split('\n')
address = address_info[0]
city_state_zip = address_info[1]
return [driver.current_url, address, city_state_zip, landline]
# selenium webdriver setup
options = webdriver.ChromeOptions()
# for the webdriver; suppresses warnings in terminal
options.add_experimental_option('excludeSwitches', ['enable-logging'])
# options.add_argument("--headless")
# options.add_argument('--disable-gpu')
# options.add_argument('--no-sandbox')
options.add_argument('--start-maximized')
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
options.add_argument('--disable-blink-features=AutomationControlled')
# below, you provide the path to the WebDriver for the browser of your choice, not the path to the browser .exe itself
# the WebDriver is a browser extension that you must install in order for Selenium to work with that browser
driver = webdriver.Chrome(service=Service(r'C:\Users\Owner\OneDrive\Documents\Gray Property Group\Prospecting\Python\Selenium WebDriver for Chrome\chromedriver.exe'), options=options)
driver.implicitly_wait(10)
# driver.maximize_window()
from time import sleep
names = ['Kevin J Haggerty', 'Patricia B Halliday', 'David R Harb', 'Jeffrey E Hathway', 'Hanshin Hsieh']
for name in names:
print(name + ':')
individual_info = scrape_individual_info_wp(driver, name, 'Manchester, NH')
for field in individual_info:
print('\t' + field)
print('\n')
driver.quit()
Output:
Kevin J Haggerty:
https://www.whitepages.com/name/Kevin-J-Haggerty/Bedford-NH/PLyZ4BaGl8Q
26 Southgate Dr
Bedford, NH 03110
(603) 262-9114
Patricia B Halliday:
Traceback (most recent call last):
(...)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//div[#class='results-container']/a"}
Screenshot of browser (see arrow / red text):
Try with below code once.
Navigate to the Website once and can use driver.back() to come back to the original Page.
User Explicit wait to wait for the elements to appear. And can use good locators like ID or CLASS_NAME to locate the elements.
from selenium.webdriver.support.wait import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium import webdriver
from selenium.webdriver.chrome.service import Service
from selenium.webdriver.common.by import By
import time
def scrape_individual_info_wp(driver, individual_name, city_state):
name_field = wait.until(EC.element_to_be_clickable((By.ID, "desktopSearchBar")))
name_field.clear() # Clear the field to enter a new name
name_field.send_keys(individual_name)
state_field = wait.until(EC.element_to_be_clickable((By.CLASS_NAME, "pa-3")))
state_field.clear() # Clear the field and enter the state
state_field.send_keys(city_state)
search = wait.until(EC.element_to_be_clickable((By.ID, "wp-search")))
search.click()
time.sleep(2) # Using Sleep to make sure the data is loaded. Can use Waits to wait for the data to appear and extract the same.
# Code to scarpe data.
driver.back()
driver = webdriver.Chrome(service= Service("C:/expediaproject/Chromedriver/chromedriver.exe"))
driver.maximize_window()
wait = WebDriverWait(driver,30)
driver.get("https://www.whitepages.com/")
names = ['Kevin J Haggerty', 'Patricia B Halliday', 'David R Harb', 'Jeffrey E Hathway', 'Hanshin Hsieh']
state = "Manchester, NH"
for name in names:
scrape_individual_info_wp(driver, name, state)
driver.quit()
One of the scripts is reopening the page if any input is not empty when the script runs.
(There are too many scripts on the page; I was not able to narrow down which one it is.)
A simple solution is to add sleep(1) after driver.get(...).
driver.get('https://www.whitepages.com/')
sleep(1) # Add this
Power usage
sleep(1) significantly slows down the script, especially if looping through many names.
Instead, we can use two tabs alternately and prepare each tab for the next-next iteration.
Setup before loop:
driver.get('https://www.whitepages.com/')
driver.execute_script('window.open("https://www.whitepages.com/")')
driver.switch_to.window(driver.window_handles[0])
next_index = 1
Prepare for the next and next-next iterations after processing each name:
for name in names:
...
# Prepare this tab for next-next iteration
driver.get('https://www.whitepages.com/')
# Switch to another tab for next iteration
driver.switch_to.window(driver.window_handles[next_index])
next_index = 1 - next_index
Comment out driver.get(...) in scrape_individual_info_wp function:
# driver.get('https://www.whitepages.com/')
You may create a new driver for each person, so in each iteration you will start off from the home page and navigate to your desired pages.
I did these kinds of things using Selenium in Python, and this was my common approach when scraping multiple pages.
names = ['Kevin J Haggerty', 'Patricia B Halliday', 'David R Harb', 'Jeffrey E Hathway', 'Hanshin Hsieh']
for name in names:
print(name + ':')
driver = webdriver.Chrome(service=Service("C:/expediaproject/Chromedriver/chromedriver.exe"))
wait = WebDriverWait(driver,30)
driver.get("https://www.whitepages.com/")
individual_info = scrape_individual_info_wp(driver, name, 'Manchester, NH')
for field in individual_info:
print('\t' + field)
print('\n')
driver.quit()
I am trying to find a button in a date selector, I tried this:
driver.find_element_by_xpath("//input[#day='5']")
the element is :
<button class="active" day="5" month="10" year="2021" aria-describedby="tooltiptext_5">5</button>
the clander look like this:
clickhere
the error msg when it trying to find the element:
[17756:20660:1004/113550.831:ERROR:chrome_browser_main_extra_parts_metrics.cc(22
8)] crbug.com/1216328: Checking Bluetooth availability started. Please report if
there is no report that this ends.
[17756:20660:1004/113550.831:ERROR:chrome_browser_main_extra_parts_metrics.cc(23
1)] crbug.com/1216328: Checking Bluetooth availability ended.
[17756:20660:1004/113550.832:ERROR:chrome_browser_main_extra_parts_metrics.cc(23
4)] crbug.com/1216328: Checking default browser status started. Please report if
there is no report that this ends.
[17756:20660:1004/113550.860:ERROR:chrome_browser_main_extra_parts_metrics.cc(23
8)] crbug.com/1216328: Checking default browser status ended.
Traceback (most recent call last):
File "
", line 26, in <module>
func()
File "
", line 22, in func
driver.find_element_by_xpath(DATE)
File "
ib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_elem
ent_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "
ib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_elem
ent
return self.execute(Command.FIND_ELEMENT, {
File "
ib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "
ib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_
response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Una
ble to locate element: {"method":"xpath","selector":"//input[#day='5']"}
(Session info: chrome=94.0.4606.71)
thanks a lot Adam.
You might be getting this error since you have given an input element for the xpath even though it's a button. So try this.
btn = driver.find_element_by_xpath("//button[#day='5']")
It could be cause of lot of reason.
Element could be in an iframe.
Element is not rendered properly.
Element in not unique in HTMLDOM.
etc..
Please check in dev tool, if we have unique entry in HTML DOM or not for the below xpath.
//button[#day='5']
Steps to check:
Press F12 in Chrome -> go to element section -> do a CTRL + F -> then paste the xpath and see, if your desired element is getting highlighted with 1/1 matching node.
if it's 1/1 matching node.
Code trial 1 :
time.sleep(5)
driver.find_element_by_xpath("//button[#day='5']").click()
Code trial 2 :
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//button[#day='5']"))).click()
Code trial 3 :
time.sleep(5)
button = driver.find_element_by_xpath("//button[#day='5']")
driver.execute_script("arguments[0].click();", button)
Code trial 4 :
time.sleep(5)
button = driver.find_element_by_xpath("//button[#day='5']")
ActionChains(driver).move_to_element(button).click().perform()
Imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.action_chains import ActionChains
I am writing a script that deals with google. I have successfully searched for what I wanted using the selenium web driver however I would like to navigate to the next page of results. my code looks as follows:
import parameters
import csv
from parsel import Selector
from selenium import webdriver
from time import sleep
from selenium.webdriver.common.keys import Keys
empty=""
def validate_field(field):
if field == "":
field= 'No results'
return field
writer=csv.writer(open(parameters.file_name, 'w'))
writer.writerow(('Name','Job Title','Company','College','Location','URL'))
driver=webdriver.Chrome('/usr/local/bin/chromedriver')
driver.implicitly_wait(10)
driver.get('https://www.linkedin.com')
username=driver.find_element_by_id('session_key') #locating the email form using the class name
username.send_keys('')
sleep(0.5)
password=driver.find_element_by_id('session_password') #locating the password form using the class name
password.send_keys('')
sleep(0.5)
log_in_button=driver.find_element_by_class_name('sign-in-form__submit-button') #locating submit button by class name
log_in_button.click() #here we are mimicing a click
sleep(0.5)
driver.get('https:www.google.com') #navigating to google
sleep(3)
search_gog=driver.find_element_by_name('q')
search_gog.send_keys(parameters.search_gog)
sleep(0.5)
search_gog.send_keys(Keys.RETURN)
sleep(3)
list_links = [link.get_attribute('href') for link in driver.find_elements_by_xpath("//div[#class='g']//div[#class='r']/a[contains(#href, 'https://www.linkedin.com')]")]
for link in list_links:
driver.get(link)
sel=Selector(text=driver.page_source)
name = sel.xpath('//*[starts-with(#class, "inline t-24 t-black t-normal break-words")]/text()').extract_first()
if name:
name=name.strip()
job_title= sel.xpath('//*[starts-with(#class, "mt1 t-18 t-black t-normal break-words")]/text()').extract_first()
if job_title:
job_title=job_title.strip()
education = sel.xpath('//*[starts-with(#class, "pv-profile-section-pager ember-view")]/text()').extract_first()
if education:
education=education.strip()
name=validate_field(name)
job_title=validate_field(job_title)
education=validate_field(education)
print('\n')
print('Name: ' + name)
print('Job Title: ' + job_title)
print('Education: ' + education)
print(education)
print('URL: ' + driver.current_url)
print('\n')
driver.find_element_by_link_text("Next").click()
the parameters file contains the google search query which states:
search_gog = 'site:linkedin.com/in/ AND "data analyst" AND "California"'
upon running the above I receive the following error:
Traceback (most recent call last):
File "app2.py", line 79, in <module>
driver.find_element_by_link_text("Next").click()
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 428, in find_element_by_link_text
return self.find_element(by=By.LINK_TEXT, value=link_text)
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Users/rubenolmos/opt/anaconda3/lib/python3.8/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"link text","selector":"Next"}
(Session info: chrome=85.0.4183.102)
I have also tried using the html code for the element which is:
<a href="/search?q=site:linkedin.com/in/+AND+%22data+analyst%22+AND+%22California%22&ei=lbpbX4GhDoPs9APDmITID
A&start=10&sa=N&ved=2ahUKEwjBv57e1uHrAhUDNn0KHUMMAckQ8NMDegQICxA_"
id="pnnext" style="text-align:left"><span class="SJajHc NVbCr"
style="background:url(/images/nav_logo299_hr.webp) no-repeat;background-position:-96px
0;background-size:167px;width:71px"></span><span style="display:block;margin-
left:53px">Next</span></a>
using the above html element I have tried finding the element using the element id "pnnext' by doing the following find_element_by_id("pnnext").click and it has been unsuccessful.
Any ideas?
First, if you have driver.implicitly_wait(10), then when you do driver.find_element_by_link_text("Next").click() following search_gog.send_keys(Keys.RETURN), the driver will wait for up to 10 seconds for the next page to load and a link with the text "Next" to appear before throwing an exception. So, the call to sleep(3) that you invoke between the two calls is rather superfluous. If for whatever reason you thought 10 seconds was not enough (hard to believe for google.com), then just increase the amount of time on the call to driver.implicitly_wait.
I cannot tell what your search argument to Google is, but it just may be possible that the number of returned results isn't enough to warrant a "Next" link. Modify your code as follows:
search_gog.send_keys(Keys.RETURN)
next_links = driver.find_elements_by_link_text("Next") # note it is find_elements with an s
if len(next_links):
next_links[0].click()
else:
print('There is no "Next" link')
Update
I have run the following code on my desktop 5 times in a row and each time it has successfully navigated to the second page (that is, it has found the "Next" link and clicked on it successfully).
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
options = webdriver.ChromeOptions()
#options.add_argument("headless")
options.add_experimental_option('excludeSwitches', ['enable-logging'])
driver = webdriver.Chrome(options=options)
try:
driver.implicitly_wait(10)
driver.get('https://www.google.com') #navigating to google
search_gog=driver.find_element_by_name('q')
search_gog.send_keys('site:linkedin.com/in/ AND "data analyst" AND "California"')
search_gog.send_keys(Keys.RETURN)
next_links = driver.find_elements_by_link_text("Next") # note it is find_elements with an s
if len(next_links):
print('Found "Next" link')
next_links[0].click()
else:
print('There is no "Next" link')
finally:
input('pausing (hit enter to terminate) ...')
driver.quit()
selenium.common.exceptions.NoSuchElementException means it can't find the element it may be caused by elements takings too long to load or a wrong XPath. There might be an iframe as well.
elem=WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.ID('pnnext')))
elem.click()
Import these.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
I am trying to scrape cart items and get their respective price of a shopping website.
So far this is my code but getting an error. I think I am not getting the proper XPath
from selenium import webdriver
chrome_path =r"C:\Users\aq4'july\Desktop\chromedriver.exe"
driver = webdriver.Chrome()
driver.get("https://www.myntra.com/")
#User Login
driver.find_element_by_xpath("""//*[#id="desktop-header-cnt"]/div[2]/div[2]/div/div[2]/div[2]/div[2]/a[2]""")
The error I am getting is
Traceback (most recent call last):
File "<pyshell#16>", line 1, in <module>
driver.find_element_by_xpath("""//*[#id="desktop-header-cnt"]/div[2]/div[2]/div/div[2]/div[2]/div[2]/a[2]""").click()
File "C:\Users\aq4'july\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py", line 80, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Users\aq4'july\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "C:\Users\aq4'july\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\webdriver.py", line 311, in execute
self.error_handler.check_response(response)
File "C:\Users\aq4'july\AppData\Roaming\Python\Python36\site-packages\selenium\webdriver\remote\errorhandler.py", line 237, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotVisibleException: Message: element not visible
(Session info: chrome=63.0.3239.84)
(Driver info: chromedriver=2.31.488763 (092de99f48a300323ecf8c2a4e2e7cab51de5ba8),platform=Windows NT 10.0.15063 x86_64)
Please feel free to Use username and Password (sarwarhayatt#outlook.com, sunset#paris ) respectively for test purpose.
Note: I am using python 3.6 and please highlight I missed anything while asking the question.
After you logged in you can go to cart directly
driver.get('https://www.myntra.com/checkout/cart')
or you can click "bag" button
item = driver.find_element_by_css_selector('.desktop-cart')
item.click()
Full code tested with Firefox()
EDIT: sometimes it has problem to login but it seems sleep() resolves this problem.
from selenium import webdriver
import time
LOGIN = 'xxx#xxx.com'
PASSWORD = 'PaSwOrD'
# --- start ---
chrome_path =r"C:\Users\aq4'july\Desktop\chromedriver.exe"
driver = webdriver.Chrome()
#driver = webdriver.Firefox()
# resize window so all elements are visible
# and the is no problem to click them
driver.maximize_window()
#driver.set_window_size(1920, 1080)
#driver.execute_script("window.resizeTo(1920,1080)") # doesn't work for me
# --- main page ---
#driver.get("https://www.myntra.com/")
# --- login ---
driver.get('https://www.myntra.com/login?referer=https://www.myntra.com/')
time.sleep(1)
item = driver.find_element_by_css_selector('.login-user-input-email')
item.send_keys(LOGIN)
item = driver.find_element_by_css_selector('.login-user-input-password')
item.send_keys(PASSWORD)
item = driver.find_element_by_css_selector('.login-login-button')
item.click()
time.sleep(1)
# --- main page ---
#driver.get("https://www.myntra.com/")
# --- cart ---
item = driver.find_element_by_css_selector('.desktop-cart')
item.click()
# or
#driver.get('https://www.myntra.com/checkout/cart')
Resizing: How do I resize the window in Chrome and Firefox when testing with Selenium?
Scrolling: How can I scroll a web page using selenium webdriver in python?
Bro if you want to login just go to login page
and do stuff....
do count double quotes" and escape sequences
from selenium import webdriver
import time
chrome_path ="chromedriver.exe"
driver = webdriver.Chrome()
driver.get("https://www.myntra.com/login")
time.sleep(2)
driver.find_element_by_xpath("//*[#id=\"mountRoot\"]/div/div/div/form/fieldset[1]/div[1]/input").send_keys("sarwarhayatt#outlook.com")
driver.find_element_by_xpath("//*[#id=\"mountRoot\"]/div/div/div/form/fieldset[1]/div[2]/input").send_keys("sunset#paris")
driver.find_element_by_xpath("//*[#id=\"mountRoot\"]/div/div/div/form/fieldset[2]/button").click()
time.sleep(2)
driver.find_element_by_xpath("//*[#id=\"desktop-header-cnt\"]/div[2]/div[2]/a").click()
time.sleep(2)
print (driver.find_element_by_xpath("//*[#id=\"prod-item-1514127198\"]/div[2]/div[4]/div/div").text)
time.sleep(10)
driver.quit()
your error is right here:
3 double quotes and "desktop-header-cnt" double quotes here is breaking the complete string
driver.find_element_by_xpath("""//*[#id="desktop-header-cnt"]/div[2]/div[2]/div/div[2]/div[2]/div[2]/a[2]""")