With all installation prerequisite of PhantomJS and Selenium on my Ubuntu machine I am running below code snippet:
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.set_window_size(1120, 550)
driver.get("https://duckduckgo.com/")
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit()
On execution I am getting below error:
$ python duck.py
Traceback (most recent call last):
File "duck.py", line 5, in <module>
driver.find_element_by_id('search_form_input_homepage').send_keys("realpython")
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 208, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
{'using': by, 'value': value})['value']
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: Error Message => 'Unable to find element with id 'search_form_input_homepage''
caused by Request => {"headers":{"Accept":"application/json","Accept-Encoding":"identity","Connection":"close","Content-Length":"107","Content-Type":"application/json;charset=UTF-8","Host":"127.0.0.1:50789","User-Agent":"Python-urllib/2.7"},"httpVersion":"1.1","method":"POST","post":"{\"using\": \"id\", \"sessionId\": \"26560250-fec9-11e4-b2ee-2dada5838664\", \"value\": \"search_form_input_homepage\"}","url":"/element","urlParsed":{"anchor":"","query":"","file":"element","directory":"/","path":"/element","relative":"/element","port":"","host":"","password":"","user":"","userInfo":"","authority":"","protocol":"","source":"/element","queryKey":{},"chunks":["element"]},"urlOriginal":"/session/26560250-fec9-11e4-b2ee-2dada5838664/element"}
Screenshot: available via screen
Setting --ssl-protocol=any service argument and using Explicit Waits made it work for me:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.PhantomJS(service_args=['--ssl-protocol=any'])
driver.maximize_window()
driver.get("https://duckduckgo.com/")
wait = WebDriverWait(driver, 10)
search = wait.until(EC.presence_of_element_located((By.ID, "search_form_input_homepage")))
search.send_keys("realpython")
driver.find_element_by_id("search_button_homepage").click()
print driver.current_url
driver.quit()
Prints https://duckduckgo.com/?q=realpython.
Note that without --ssl-protocol=any PhantomJS hasn't even loaded the page and the current url stayed as about:blank.
I think whats happening to you is that you try to find an element which is not loaded yet on the page. So what I can recommend is the insert a WaitForElementDisplayed(by.ID("search_form_input_homepage")); right before you try to type in the search field. So you will be sure that the element is there before trying to interact with it.
I can't really give you a code example because I'm not really familiar with the Python bindings.
Related
This is my first post here!
I am using Selenium's Chrome Driver to send WhatsApp attachment to some people.
Here is my code:
from selenium import webdriver
import os
from time import sleep
link="https://wa.me/91xxxxxxxxxx"
phnno="91xxxxxxxxxx"
driver=webdriver.Chrome(executable_path=f"{os.getcwd()}\\chromedriver.exe")
#driver.get(link)
#button=driver.find_element_by_xpath('//a[#title="Share on WhatsApp"]')
#button.click()
driver.get(f"https://web.whatsapp.com/send?phone={phnno}&text&app_absent=0")
#This above line opens up the sender window in whatsapp
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]') #This is line 15
#The above line is the one that is giving me the error
attachbutt.click()
sleep(10)
forpdf=driver.find_element_by_xpath('//input[#accept="*"]')
path="C:\\Users\\Deven\\Desktop\\test_file.pdf"
forpdf.send_keys(path) #Attaches the file
sleep(5)
sendbutt=driver.find_element_by_xpath('//span[#data-icon="send"]')
sendbutt.click()
ERROR:
DevTools listening on ws://127.0.0.1:56230/devtools/browser/1a8a2adb-37ee-4b0c-bedc-5cfb58559c24
Traceback (most recent call last):
File "d:\Coding\Python Scripts\Dr Nikhil Prescription App\Prescription Generator\WA-testing.py", line 15, in <module>
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]')
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 976, in find_element
return self.execute(Command.FIND_ELEMENT, {
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\Deven\AppData\Local\Programs\Python\Python39\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//span[#data-icon="clip"]"}
(Session info: chrome=90.0.4430.212)
PS D:\Coding\Python Scripts\Dr Nikhil Prescription App\Prescription Generator> [16176:15752:0523/212201.236:ERROR:device_event_log_impl.cc(214)] [21:22:01.236] USB: usb_device_handle_win.cc:1054 Failed to read descriptor from node connection: A device attached to the system is not functioning. (0x1F)
It says it is unable to locate the element but I have bene very careful in inspecting the website and then writing the code. Still I wonder why it does not work. Can anyone please help me with what is it that I am doing wrnong? Thank You!
Looks like you are missing a wait / delay before clicking that element.
The simplest solution is to put
sleep(5)
before
attachbutt=driver.find_element_by_xpath('//span[#data-icon="clip"]')
However, it's much better to use expected conditions there. Like this:
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
attachbutt = WebDriverWait(driver, 20).until(
EC.element_to_be_clickable((By.XPATH, '//span[#data-icon="clip"]')))
My python script is running on a Debian Google Cloud Server. I am trying to scrape news from
Forexfactory.com/news. I tested my code on my macOS and it worked beautifully. After uploading onto the server I had to add chrome option arguments to my code to help find the chromedriver in the VM. It was after debugging this that selenium was unable to find the elements required.
This is the python code:
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
import os
opt = Options()
opt.add_argument("--no-sandbox")
opt.add_argument("--disable-dev-shm-usage")
opt.add_argument("--headless")
driver = webdriver.Chrome(chrome_options=opt, executable_path=r'/home/rehanmahmood38/chromedriver')
URL = 'https://www.forexfactory.com/news'
driver.get(URL)
driver.implicitly_wait(5) # wait for seconds
uiOuter = driver.find_element_by_id('ui-outer')
aHref = driver.find_elements_by_css_selector('div.flexposts__story-title a')
span = driver.find_elements_by_css_selector('div.flexposts__storydisplay-info')
This is the error message
Traceback (most recent call last):
File "ffmain.py", line 16, in <module>
uiOuter = driver.find_element_by_id('ui-outer')
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 978, in find_element
'value': value})['value']
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/usr/lib/python3/dist-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"ui-outer"}
(Session info: headless chrome=83.0.4103.116)
(Driver info: chromedriver=2.40.565383 (76257d1ab79276b2d53ee976b2c3e3b9f335cde7),platform=Linux 4.19.0-12-cloud-amd64 x86_64)
Any idea why selenium is unable to locate the elements
Use this for your custom window size. Probably it differs on MacOS and VM.
driver.manage().window().setSize(new Dimension(1024,768));
Instead of implicitly waiting, use selenium's explicit wait.
Seems like the element may have not loaded properly - hence it was not found.
Try using this.
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
uiOuter = WebDriverWait(driver, 10).until( EC.presence_of_element_located((By.ID, "ui-outer"))
The docs on waits is helpful if you would like to look into it further.
https://selenium-python.readthedocs.io/waits.html
I'm trying to get a login script to select a user name input to enter in my user name. After this popup is done there will be another one asking for the password. I'm new to python and web interfaces so I'm having trouble identifying what element of the website I need to select to get this to work. Here is the code I have so far.
import selenium
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.support.ui import WebDriverWait as wait
DynamoForum = webdriver.Chrome()
DynamoForum.get("https://forum.dynamobim.com/")
login = DynamoForum.find_element_by_class_name("header-buttons").click()
#DynamoForum.switch_to_frame(DynamoForum.find_element_by_
#wait(DynamoForum,10).until(EC.frame_to_be_available_and_switch_to_it(
DynamoForum.find_element_by_xpath("//title[1]")))
wait(DynamoForum,10).until(EC.frame_to_be_available_and_switch_to_it(
DynamoForum.find_element_by_xpath(
"//iframe[#id='destination_publishing_iframe_autodesk_0']")))
DynamoForum.find_element_by_id("userName").send_heys("xxx")
The website is opening and the popup is starting but no text is being entered. Here is what my getting as a result:
Traceback (most recent call last):
File "C:/Users/cjr/PycharmProjects/DynamoForum/DynamoForum.py", line 17, in <module>
wait(DynamoForum, 10).until(EC.frame_to_be_available_and_switch_to_it(DynamoForum.find_element_by_xpath("//iframe[#id='destination_publishing_iframe_autodesk_0']")))
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 394, in find_element_by_xpath
return self.find_element(by=By.XPATH, value=xpath)
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\cjr\PycharmProjects\DynamoForum\venv\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//iframe[#id='destination_publishing_iframe_autodesk_0']"}
(Session info: chrome=72.0.3626.119)
(Driver info: chromedriver=73.0.3683.20 (8e2b610813e167eee3619ac4ce6e42e3ec622017),platform=Windows NT 10.0.17134 x86_64)
Basically when you are clicking on login button, you are moving to another window and to access the element in new window you need to switch it from parent window to access this.Try the below code it should work.
from selenium import webdriver
DynamoForum = webdriver.Chrome()
DynamoForum.get("https://forum.dynamobim.com/")
Parent_window = DynamoForum.window_handles[0]
login = DynamoForum.find_element_by_class_name("header-buttons").click()
window_child= DynamoForum.window_handles[1]
DynamoForum.switch_to.window(window_child)
DynamoForum.find_element_by_id("userName").send_keys("xyz#gmail.com")
DynamoForum.find_element_by_id("verify_user_btn").click()
wait=WebDriverWait(DynamoForum,20)
wait.until(EC.visibility_of_element_located((By.ID,"password"))).send_keys("xxx")
DynamoForum.find_element_by_id("btnSubmit").click()
You need to switch to the iframe.
e.g.
iframe = driver.find_element_by_id('destination_publishing_iframe_autodesk_0')
driver.switch_to.frame(iframe)
driver.find_element_by_name('userName').send_keys('xxx')
See the switch_to function here : https://selenium-python.readthedocs.io/api.html?highlight=iframe
For reference:
python selenium cant find iframe xpath
https://seleniumwithjavapython.wordpress.com/selenium-with-python/intermediate-topics/handling-iframes-in-a-webpage/
I'd like to scroll to the bottom of a page using Selenium with the Firefox driver.
Here is what I have so far:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser=webdriver.Firefox()
browser.get('http://nostarch.com')
htmlElem=browser.find_element_by_tag_name('html')
htmlElem.send_keys(Keys.END) #scrolls to bottom
Unfortunately, my code fails with the following exception:
Traceback (most recent call last):
File "<pyshell#26>", line 1, in <module>
htmlElem.send_keys(Keys.END) #scrolls to bottom
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webelement.py", line 352, in send_keys
'value': keys_to_typing(value)})
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webelement.py", line 501, in _execute
return self._parent.execute(command, params)
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\webdriver.py", line 308, in execute
self.error_handler.check_response(response)
File "C:\Users\academy\AppData\Local\Programs\Python\Python35\lib\site-
packages\selenium\webdriver\remote\errorhandler.py", line 194, in
check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.ElementNotInteractableException: Message:
Element is not visible
Use WebDriverWait (explicit wait) till element to be visible i dont know too much about python so give just general syntax
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser=webdriver.Firefox()
browser.get('http://nostarch.com')
htmlElem=WebDriverWait(browser, 5).until(
EC.presence_of_element_located((By.find_element_by_tag_name, "html"))
htmlElem.send_keys(Keys.END)
To scroll to the bottom of a page using Selenium you can use :
Python
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
Java
((JavascriptExecutor)driver).executeScript("window.scrollTo(0, document.body.scrollHeight);");
I've recently encountered something I've never seen before while using selenium.
The code (quite simple and straightforward):
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("http://www.drugs.com/drug-class/laxatives.html?condition_id=&generic=0&sort=rating&order=desc")
print driver.find_element_by_tag_name("title").text
Here is a stack trace of the error I'm getting:
Traceback (most recent call last):
File "/Users/a/p/SO/selenium_scripts/test.py", line 6, in <module>
print driver.find_element_by_tag_name("title").text
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 328, in find_element_by_tag_name
return self.find_element(by=By.TAG_NAME, value=name)
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 664, in find_element
{'using': by, 'value': value})['value']
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/webdriver.py", line 175, in execute
self.error_handler.check_response(response)
File "/Users/a/.virtualenvs/so/lib/python2.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 166, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: h is null
Using Firefox 37.0 and selenium 2.45.0.
Observations:
if I switch to webdriver.Chrome() - I don't see any errors
if I use a different URL, e.g. https://google.com - I don't see any errors
I've tried to explicitly wait for search results to be visible before making any further actions, but I still get the same error, code I've used:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get("http://www.drugs.com/drug-class/laxatives.html?condition_id=&generic=0&sort=rating&order=desc")
# wait for the table list to load
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.data-list")))
it is not easily googleable which probably means it is web-site specific, but, as noted before, no errors in Chrome
Where is the error coming from and what can I do to prevent/fix it? Does this mean I cannot browser/locate elements on this particular web page using selenium+firefox?
Looks like selenium 2.45.0 does not support ff 37.
The change log shows selenium 2.44 supported FF33. Selenium 2.45 was released around Feb 26th 2015, while FF37 was released on March 31st 2015.
Looks like by the time the page loads it is executing this line of code and throwing an error message.
try to wait till the page loads by using
time.sleep(200)
and keep this line of code under try block as:
try:
WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "table.data-list")))
except:
print("not found")