I've recently encountered something I've never seen before while using selenium.
The code (quite simple and straightforward):
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.drugs.com/drug-class/laxatives.html?condition_id=&generic=0&sort=rating&order=desc")
print driver.find_element_by_tag_name("title").text
Here is a stack trace of the error I'm getting:
Traceback (most recent call last):
File "/Users/a/p/SO/selenium_scripts/test.py", line 6, in <module>
print driver.find_element_by_tag_name("title").text
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 328, in find_element_by_tag_name
return self.find_element(by=By.TAG_NAME, value=name)
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
{'using': by, 'value': value})['value']
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
self.error_handler.check_response(response)
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: h is null
Using Firefox 37.0 and selenium 2.45.0.
Observations:
if I switch to webdriver.Chrome() - I don't see any errors
if I use a different URL, e.g. https://google.com - I don't see any errors
I've tried to explicitly wait for search results to be visible before making any further actions, but I still get the same error, code I've used:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://www.drugs.com/drug-class/laxatives.html?condition_id=&generic=0&sort=rating&order=desc")
# wait for the table list to load
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.data-list")))
it is not easily googleable which probably means it is web-site specific, but, as noted before, no errors in Chrome
Where is the error coming from and what can I do to prevent/fix it? Does this mean I cannot browser/locate elements on this particular web page using selenium+firefox?
Looks like selenium 2.45.0 does not support ff 37.
The change log shows selenium 2.44 supported FF33. Selenium 2.45 was released around Feb 26th 2015, while FF37 was released on March 31st 2015.
Looks like by the time the page loads it is executing this line of code and throwing an error message.
try to wait till the page loads by using
time.sleep(200)
and keep this line of code under try block as:
try:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.data-list")))
except:
print("not found")
Related
This is my first post here!
I am using Selenium's Chrome Driver to send WhatsApp attachment to some people.
Here is my code:
from selenium import webdriver
import os
from time import sleep
link="https://wa.me/91xxxxxxxxxx"
phnno="91xxxxxxxxxx"
driver=webdriver.Chrome(executable_path=f"{os.getcwd()}\\chromedriver.exe")
#driver.get(link)
#button=driver.find_element_by_xpath('//a[#title="Share on WhatsApp"]')
#button.click()
driver.get(f"https://web.whatsapp.com/send?phone={phnno}&text&app_absent=0")
#This above line opens up the sender window in whatsapp
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]') #This is line 15
#The above line is the one that is giving me the error
attachbutt.click()
sleep(10)
forpdf=driver.find_element_by_xpath('//input[#accept="*"]')
path="C:\\Users\\Deven\\Desktop\\test_file.pdf"
forpdf.send_keys(path) #Attaches the file
sleep(5)
sendbutt=driver.find_element_by_xpath('//span[#data-icon="send"]')
sendbutt.click()
ERROR:
DevTools listening on ws://127.0.0.1:56230/devtools/browser/1a8a2adb-37ee-4b0c-bedc-5cfb58559c24
Traceback (most recent call last):
File "d:\Coding\Python Scripts\Dr Nikhil Prescription App\Prescription Generator\WA-testing.py", line 15, in <module>
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]')
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[#data-icon="clip"]"}
(Session info: chrome=90.0.4430.212)
PS D:\Coding\Python Scripts\Dr Nikhil Prescription App\Prescription Generator> [16176:15752:0523/212201.236:ERROR:device_event_log_impl.cc(214)] [21:22:01.236] USB: usb_device_handle_win.cc:1054 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
It says it is unable to locate the element but I have bene very careful in inspecting the website and then writing the code. Still I wonder why it does not work. Can anyone please help me with what is it that I am doing wrnong? Thank You!
Looks like you are missing a wait / delay before clicking that element.
The simplest solution is to put
sleep(5)
before
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]')
However, it's much better to use expected conditions there. Like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
attachbutt = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, '//span[#data-icon="clip"]')))
My python script is running on a Debian Google Cloud Server. I am trying to scrape news from
Forexfactory.com/news. I tested my code on my macOS and it worked beautifully. After uploading onto the server I had to add chrome option arguments to my code to help find the chromedriver in the VM. It was after debugging this that selenium was unable to find the elements required.
This is the python code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import os
opt = Options()
opt.add_argument("--no-sandbox")
opt.add_argument("--disable-dev-shm-usage")
opt.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=opt, executable_path=r'/home/rehanmahmood38/chromedriver')
URL = 'https://www.forexfactory.com/news'
driver.get(URL)
driver.implicitly_wait(5) # wait for seconds
uiOuter = driver.find_element_by_id('ui-outer')
aHref = driver.find_elements_by_css_selector('div.flexposts__story-title a')
span = driver.find_elements_by_css_selector('div.flexposts__storydisplay-info')
This is the error message
Traceback (most recent call last):
File "ffmain.py", line 16, in <module>
uiOuter = driver.find_element_by_id('ui-outer')
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
'value': value})['value']
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"ui-outer"}
(Session info: headless chrome=83.0.4103.116)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.19.0-12-cloud-amd64 x86_64)
Any idea why selenium is unable to locate the elements
Use this for your custom window size. Probably it differs on MacOS and VM.
driver.manage().window().setSize(new Dimension(1024,768));
Instead of implicitly waiting, use selenium's explicit wait.
Seems like the element may have not loaded properly - hence it was not found.
Try using this.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
uiOuter = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "ui-outer"))
The docs on waits is helpful if you would like to look into it further.
https://selenium-python.readthedocs.io/waits.html
The bot is supposed to send message on whatsApp WEB but unfortunately is stopping and giving error when asked find the user through X-path.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver=webdriver.Chrome(executable_path="C:\drivers\chromedriver.exe")
driver.get("https://web.whatsapp.com/")
time.sleep(5)
name= input("Enter name")
input("Enter anything after scanning")
time.sleep(2)
user=driver.find_element_by_xpath("//span[#title='{}']".format(name))
The program is stopping exactly after this line, and giving the following error,
Traceback (most recent call last):
File "C:/Users/myName/PycharmProjects/firstpro/whatsAppBot.py", line 17, in <module>
user=driver.find_element_by_xpath("//span[#title='{}']".format(name))
File "C:\Users\myName\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\myName\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\myName\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\myName\AppData\Local\Programs\Python\Python38-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[#title='Jaden']"}
(Session info: chrome=81.0.4044.138)
Process finished with exit code 1
python version:3.8
Traceback (most recent call last):
File "C:/Users/myName/PycharmProjects/firstpro/whatsAppBot.py", line 17, in <module>
user=driver.find_element_by_xpath("//span[#title='{}']".format(name))
The final line of the traceback output tells you what type of exception was raised along with some relevant information about that exception. The previous lines of the traceback point out the code that resulted in the exception being raised.
here we see that it was unable to locate the path you specified
raison why it stopped
I feel the issue is with the element not being in the viewport, THe user you are trying to access must be not in the view, also try to debug the issue manually, by below steps
List item
try to locate the xpath //span[#title='Jaden'] and see if you are able to locate it in the dev tools without scrolling the page down.(If it is able to locate after scrolling the you will have to scroll programmatically using javascript executor.)
Try to see if there is any load time issue and try to implement appropriate explicit wait for the element
To send a message to a user through WhatsApp web https://web.whatsapp.com/ using Selenium you have to click on the username inducing WebDriverWait for the element_to_be_clickable() and you can use either of the following Locator Strategies:
Using CSS_SELECTOR:
driver.get("https://web.whatsapp.com/")
name= input("Enter name:")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.CSS_SELECTOR, "div[role='option'] span[title='{}']".format(name)))).click()
Using XPATH:
driver.get("https://web.whatsapp.com/")
name= input("Enter name:")
WebDriverWait(driver, 20).until(EC.element_to_be_clickable((By.XPATH, "//div[#role='option']//span[#title='{}']".format(name)))).click()
Note : You have to add the following imports :
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.by import By
from selenium.webdriver.support import expected_conditions as EC
I'm trying to get a login script to select a user name input to enter in my user name. After this popup is done there will be another one asking for the password. I'm new to python and web interfaces so I'm having trouble identifying what element of the website I need to select to get this to work. Here is the code I have so far.
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
DynamoForum = webdriver.Chrome()
DynamoForum.get("https://forum.dynamobim.com/")
login = DynamoForum.find_element_by_class_name("header-buttons").click()
#DynamoForum.switch_to_frame(DynamoForum.find_element_by_
#wait(DynamoForum,10).until(EC.frame_to_be_available_and_switch_to_it(
DynamoForum.find_element_by_xpath("//title[1]")))
wait(DynamoForum,10).until(EC.frame_to_be_available_and_switch_to_it(
DynamoForum.find_element_by_xpath(
"//iframe[#id='destination_publishing_iframe_autodesk_0']")))
DynamoForum.find_element_by_id("userName").send_heys("xxx")
The website is opening and the popup is starting but no text is being entered. Here is what my getting as a result:
Traceback (most recent call last):
File "C:/Users/cjr/PycharmProjects/DynamoForum/DynamoForum.py", line 17, in <module>
wait(DynamoForum, 10).until(EC.frame_to_be_available_and_switch_to_it(DynamoForum.find_element_by_xpath("//iframe[#id='destination_publishing_iframe_autodesk_0']")))
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//iframe[#id='destination_publishing_iframe_autodesk_0']"}
(Session info: chrome=72.0.3626.119)
(Driver info: chromedriver=73.0.3683.20 (8e2b610813e167eee3619ac4ce6e42e3ec622017),platform=Windows NT 10.0.17134 x86_64)
Basically when you are clicking on login button, you are moving to another window and to access the element in new window you need to switch it from parent window to access this.Try the below code it should work.
from selenium import webdriver
DynamoForum = webdriver.Chrome()
DynamoForum.get("https://forum.dynamobim.com/")
Parent_window = DynamoForum.window_handles[0]
login = DynamoForum.find_element_by_class_name("header-buttons").click()
window_child= DynamoForum.window_handles[1]
DynamoForum.switch_to.window(window_child)
DynamoForum.find_element_by_id("userName").send_keys("xyz#gmail.com")
DynamoForum.find_element_by_id("verify_user_btn").click()
wait=WebDriverWait(DynamoForum,20)
wait.until(EC.visibility_of_element_located((By.ID,"password"))).send_keys("xxx")
DynamoForum.find_element_by_id("btnSubmit").click()
You need to switch to the iframe.
e.g.
iframe = driver.find_element_by_id('destination_publishing_iframe_autodesk_0')
driver.switch_to.frame(iframe)
driver.find_element_by_name('userName').send_keys('xxx')
See the switch_to function here : https://selenium-python.readthedocs.io/api.html?highlight=iframe
For reference:
python selenium cant find iframe xpath
https://seleniumwithjavapython.wordpress.com/selenium-with-python/intermediate-topics/handling-iframes-in-a-webpage/
With all installation prerequisite of PhantomJS and Selenium on my Ubuntu machine I am running below code snippet:
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
driver.get("https://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit()
On execution I am getting below error:
$ python duck.py
Traceback (most recent call last):
File "duck.py", line 5, in <module>
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 208, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Error Message => 'Unable to find element with id 'search_form_input_homepage''
caused by Request => {"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"107","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50789","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"26560250-fec9-11e4-b2ee-2dada5838664\", \"value\": \"search_form_input_homepage\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/26560250-fec9-11e4-b2ee-2dada5838664/element"}
Screenshot: available via screen
Setting --ssl-protocol=any service argument and using Explicit Waits made it work for me:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS(service_args=['--ssl-protocol=any'])
driver.maximize_window()
driver.get("https://duckduckgo.com/")
wait = WebDriverWait(driver, 10)
search = wait.until(EC.presence_of_element_located((By.ID, "search_form_input_homepage")))
search.send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit()
Prints https://duckduckgo.com/?q=realpython.
Note that without --ssl-protocol=any PhantomJS hasn't even loaded the page and the current url stayed as about:blank.
I think whats happening to you is that you try to find an element which is not loaded yet on the page. So what I can recommend is the insert a WaitForElementDisplayed(by.ID("search_form_input_homepage")); right before you try to type in the search field. So you will be sure that the element is there before trying to interact with it.
I can't really give you a code example because I'm not really familiar with the Python bindings.