Send keys doesn't work on mac os x, or maybe I'm doing something wrong. maybe I am referencing the keys wrong.
I'm trying to click each link to open in a new tab
Any suggestions?
Main.py
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
driver.get('https://orlando.craigslist.org/search/cta')
owl = driver.find_element_by_xpath('//*[#id="sortable-results"]/ul/li/p/a')
res = 1
size = len(driver.find_elements_by_xpath('//*[#id="sortable-results"]/ul/li/p/a'))
def run():
for i in range(0, size):
owl = driver.find_elements_by_xpath('//*[#id="sortable-results"]/ul/li/p/a')
owl[i].click().send_keys(Keys.COMMAND + 't')
driver.find_element_by_xpath('/html/body/section/header/nav/ul/li[3]/p/a').click()
if i == 1:
break
if __name__ == '__main__':
run()
Here is the error
Traceback (most recent call last):
File "main.py", line 24, in <module>
run()
File "main.py", line 17, in run
owl[i].click().send_keys(Keys.COMMAND + 't')
AttributeError: 'NoneType' object has no attribute 'send_keys'
Based on my experience using Selenium WebDriver on .Net, the .click() does not return an element. It returns nothing. It causes an element to be clicked which in this case would cause a new page to be loaded in the current instance of driver. I think this is what is happening here.
I suggest as an alternative you should collect all of the "href" attributes in all of those < a > elements in an array, then open a new tab for each and switch to the new tab using the answer here how to open a link in new tab (chrome) using selenium webdriver? . Once you've opened a new tab and switched to it, use driver.get() to load the page.
Related
I am using Python3.9+Selenium to write a small script that fills an online form for me.
A bit of context: the webpage contains a field (locationField) expecting a street address as input, and located on top of some sort of "google maps wrapper".
When typing in the field, it loads a drop-down list of one element (locationField_sugg) with the compatible complete address, and when this option is selected the map zooms-in on the chosen part of the city.
Other than this, there is a descriptionField to be filled with some random text, and then a submitButton to be clicked in order to send the form.
I noticed that if I use actionChains.move_to_element(locationField_sugg).click().perform() to click on the address in the drop-down list, then submitButton throws a StaleElementReference exception, while if I just use locationField_sugg.click() this is not the case, and the code proceeds as it should.
I've been reading through many Q/A about this notorious exception handling, but none of them seemed to explain the reason why this happens in my code.
For me it seems to be related to the behaviour of move_to_element() in combination with the "map wrapper" (?) but I do not understand why, since this function is just supposed to move the mouse in the middle of a given element.
No reload nor other changes in the webpage seem to happen (I verified that if I query the button at the beginning and re-query at the end of the script, I get the same exact instance representation string).
Besides, I query the submitButton right before performing an action on it, and I assume it is properly found since it can be printed.
Below there is a snippet of my code and of the output I get (Note: the code works properly if I use the alternative commented option, but I am curious to understand what I am missing)
CODE SNIPPET
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
websiteUrl = "https://mywebsite"
option = webdriver.ChromeOptions()
option.add_argument("-incognito")
browser = webdriver.Chrome(executable_path="/Applications/chromedriver", options=option)
browser.get(websiteUrl)
actionChains = ActionChains(browser)
# Write and select complete address
locationField = browser.find_element_by_id("location")
print("locationField = ", locationField)
locationField.send_keys("my location")
locationField_sugg = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.ID, "as-listbox")))
print("locationField_sugg = ", locationField_sugg)
#
# this throws stale element reference exception:
actionChains.move_to_element(locationField_sugg).click().perform()
#
# this does not:
# locationField_sugg.click()
# Write description
descriptionField = browser.find_element_by_id("description")
print("descriptionField = ", descriptionField)
descriptionField.send_keys("my description")
# Submit form
submitButton = WebDriverWait(browser, 10).until(EC.visibility_of_element_located((By.CSS_SELECTOR, "span.value")))
print("submitButton = ", submitButton)
actionChains.move_to_element(submitButton).click().perform()
OUTPUT
locationField = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="e75539a7-ebd0-4090-9c04-0c4994afe03f")>
locationField_sugg = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="53ef0269-aceb-451a-b559-d9e5e7aa7851")>
descriptionField = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="dce478de-a23d-4ca2-817c-81ee5ce0c232")>
nowButton = <selenium.webdriver.remote.webelement.WebElement (session="e10dc716790c61a0c160599624dd30c6", element="f2036f1c-d164-4211-a37c-2ee50e5c55c1")>
Traceback (most recent call last):
File "/Users/alice/Desktop/wasteComplaints_selenium.py", line 44, in <module>
actionChains.move_to_element(nowButton).click().perform()
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/action_chains.py", line 80, in perform
self.w3c_actions.perform()
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/common/actions/action_builder.py", line 76, in perform
self.driver.execute(Command.W3C_ACTIONS, enc)
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/Users/alice/miniconda3/lib/python3.9/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: stale element reference: element is not attached to the page document
(Session info: chrome=92.0.4515.131)
I'm not sure I know why this occurs without debugging it. Maybe the suggested address is disappearing at the moment when focus is removed from the input field? If so, when you insert the address string to the input field and then instantly clicking on the suggested address it works correct, but if after inserting the input address string and the suggested address appears you are applying actionChains.move_to_element this moves the mouse from it's initial position to the suggested address element. So the focus is moved from input field to the mouse cursor. This causes the suggested address to disappear and this is why StaleElementReference exception is thrown.
EDITED: I incorporated the final lines suggested by Sushil. At the end, I am copying the output in my terminal. I still do not get the zipfile.
SOLVED: My error was due to an incompatibility between the driver and chrome versions. I fixed by following the instructions here: unknown error: call function result missing 'value' for Selenium Send Keys even after chromedriver upgrade
I am trying to use Selenium to fill out a form and download a zipfile.
After extensively googling, I have written a Python code, but I am currently unable to download the file. A browser opens, but nothing is filled out.
I am very new at Python, so I am guessing I am missing something very trivial since the website I am trying to get info from is super simple.
This is what I have tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path='/home/miranda/webscrap_Python/chromedriver')
#driver.wait = WebDriverWait(driver,5)
driver.get("http://www.cis.es/cis/opencms/EN/formulario.jsp?dwld=/Microdatos/MD3288.zip")
Name = '//*[#id="Nombre"]'
LastName = '//*[#id="Apellidos"]'
University = '//*[#id="profesion"]'
email = '//*[#id="Email"]'
ob_req = '//*[#id="objeto1"]'
terms = '//*[#id="Terminos"]'
download = '//*[#id="mediomicrodatos"]/form/div[3]/input'
driver.find_element_by_xpath(Name).send_keys("Miranda")
driver.find_element_by_xpath(LastName).send_keys("MyLastName")
driver.find_element_by_xpath(University).send_keys("MySchool")
driver.find_element_by_xpath(email).send_keys("my_email#gmail.com")
#lines added by Sushil:
ob_req_element = driver.find_element_by_xpath(ob_req) #Finds the ob_req element
driver.execute_script("arguments[0].click();", ob_req_element) #Scrolls down to the element and clicks on it
terms_element = driver.find_element_by_xpath(terms) #The same is repeated here
driver.execute_script("arguments[0].click();", terms_element)
driver.find_element_by_xpath(download).click() #Scrolling down is not needed for the download button as it would already be in view. Only when an element is not in view should we scroll down to it in order to click on it.
Output in my terminal:
Traceback (most recent call last):
File "myCode.py", line 18, in <module>
driver.find_element_by_xpath(Name).send_keys("Miranda")
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 479, in send_keys
'value': keys_to_typing(value)})
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
(Session info: chrome=86.0.4240.75)
(Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 5.4.0-45-generic x86_64)
For each and every element below the email element, you have to scroll down to click them. Here is the full code to do it:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
#driver.wait = WebDriverWait(driver,5)
driver.get("http://www.cis.es/cis/opencms/EN/formulario.jsp?dwld=/Microdatos/MD3288.zip")
Name = '//*[#id="Nombre"]'
LastName = '//*[#id="Apellidos"]'
University = '//*[#id="profesion"]'
email = '//*[#id="Email"]'
ob_req = '//*[#id="objeto1"]'
terms = '//*[#id="Terminos"]'
download = '//*[#id="mediomicrodatos"]/form/div[3]/input'
driver.find_element_by_xpath(Name).send_keys("Miranda")
driver.find_element_by_xpath(LastName).send_keys("MyLastName")
driver.find_element_by_xpath(University).send_keys("MySchool")
driver.find_element_by_xpath(email).send_keys("my_email#gmail.com")
#All the lines below were added by me
ob_req_element = driver.find_element_by_xpath(ob_req) #Finds the ob_req element
driver.execute_script("arguments[0].click();", ob_req_element) #Scrolls down to the element and clicks on it
terms_element = driver.find_element_by_xpath(terms) #The same is repeated here
driver.execute_script("arguments[0].click();", terms_element)
driver.find_element_by_xpath(download).click() #Scrolling down is not needed for the download button as it would already be in view. Only when an element is not in view should we scroll down to it in order to click on it.
I want to download and save images using selenium in python2.7
I've tried:
from selenium import webdriver
from selenium.webdriver.common.action_chains import ActionChains
from selenium.webdriver.common.keys import Keys
url= "https://in.images.search.yahoo.com/images/view;_ylt=A2oKiHPRis1VplIALaEO9olQ;_ylu=X3oDMTIyN2I2OHZkBHNlYwNzcgRzbGsDaW1nBG9pZANjN2U1ZjU4NjAwMDQ1MDA0OGExZGMxY2Y0MzMyMDk0MwRncG9zAzEEaXQDYmluZw--?.origin=&back=https%3A%2F%2Fin.images.search.yahoo.com%2Fyhs%2Fsearch%3Fp%3D%2522Eiffel%2BGreens%2522%2BBalewadi%2509Pune%26n%3D60%26ei%3DUTF-8%26y%3DSearch%26type%3Dff.40.w81.hp.04-01.in.avg._.0715av%26fr%3Dyhs-avg-fh_lsonsw%26fr2%3Dsb-top-in.images.search.yahoo.com%26hsimp%3Dyhs-fh_lsonsw%26hspart%3Davg%26tab%3Dorganic%26ri%3D1&w=556&h=309&imgurl=www.propertyonepune.com%2Fimg%2Fgallery%2F0becda3e53f8db646a699e54b1333a4c.jpg&rurl=http%3A%2F%2Fwww.propertyonepune.com%2Fproperties%2F46%2FBalewadi&size=49.8KB&name=...+bungalows+by+Eiffel+Developers+%26+Realtors+Ltd.+at+%3Cb%3EBalewadi%3C%2Fb%3E%2C+%3Cb%3EPune%3C%2Fb%3E&p=%22Eiffel+Greens%22+Balewadi%09Pune&oid=c7e5f586000450048a1dc1cf43320943&fr2=sb-top-in.images.search.yahoo.com&fr=yhs-avg-fh_lsonsw&tt=...+bungalows+by+Eiffel+Developers+%26+Realtors+Ltd.+at+%3Cb%3EBalewadi%3C%2Fb%3E%2C+%3Cb%3EPune%3C%2Fb%3E&b=0&ni=21&no=1&ts=&tab=organic&sigr=11lu74lc1&sigb=17t67hvmu&sigi=1284god0v&sigt=12i2gtekb&sign=12i2gtekb&.crumb=wZ3uTmSmDfL&fr=yhs-avg-fh_lsonsw&fr2=sb-top-in.images.search.yahoo.com&hsimp=yhs-fh_lsonsw&hspart=avg&type=ff.40.w81.hp.04-01.in.avg._.0715av"
driver = webdriver.Firefox()
driver.get(url)
path = '//div[#class="iholder"]//img[#src]'
for k in driver.find_elements_by_xpath(path):
items = []
src = (k.get_attribute('src')).encode('utf8')
items.append(src)
print items
for lm in items:
driver.get(lm)
driver.sendKeys(Keys.Control + "s")
driver.send_keys(Keys.Enter)
It's giving me error:
Traceback (most recent call last):
File "C:/Users/Heypillow/Desktop/download.py", line 17, in <module>
driver.sendKeys(Keys.Control + "s")
AttributeError: 'WebDriver' object has no attribute 'sendKeys'
I've tried with:
driver.send_keys(Keys.CONTROL + "s")
Same error is showing
What should I do to save the images? Thanks in advance
Actually, op's first attempt is more correct than the selected answer. If you're not sending keys to an element for typing then you're sending them to the browser for shortcuts, etc.
ActionChains(driver).key_down(Keys.Control).send_keys("s").key_up(Keys.Control).perform()
It looks like you want to save the html for each picture, so you could use actions to get the context-menu of firefox -> "p" is shortcut for save page:
for lm in items:
driver.get(lm)
body = driver.find_element(By.tagName("body"));
ActionChains(driver).move_to_element(body).context_click(htmlElement).send_keys("p").send_keys(Keys.RETURN).perform();
I'm usually using Java, so there might be some typos in this python code of mine ;-)
The error you are getting is because .send_keys does not hang off of webdriver, it hangs off of webelement. You need to get a webelement first before trying to use .send_keys. For example,
for lm in items:
lm.sendKeys(Keys.Control + "s")
This isn't going to answer your main question but it does explain why you are getting the error message.
To answer your main question, google it and you will find many responses such as this one that already has answers.
The Accepted Answer has had one change since 2015.
Instead of
Keys.Control
It has now changed to
Keys.CONTROL
and the snippet changes to
ActionChains(browser).key_down(Keys.CONTROL).send_keys("s").key_up(Keys.CONTROL).perform()
I'm trying to take look at several pages on one web with Selenium - PhantomJS().
The problem is that it started freezing and I can't figure out why. It is probably something with Timeout.
Here is the__init__ method of a class.
self.driver = webdriver.PhantomJS(service_args=["--load-images=false"])
self.wait = WebDriverWait(self.driver, 2)
And here is the method:
def click_next_page(self):
log('click_next_page : '+self.driver.current_url) # THIS LINE RUNS
rep = 0
while 1:
try:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a'))) # IT MAY FREEZE HERE
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()# IT MAY FREEZE HERE
print 'NEXT' # DOESNT PRINT ANY TEXT SO THIS LINE NOT EXECUTED
log('NEXT PAGE')
return True
except Exception as e:
log('click next page EXCEPTION') # DONT HAVE THIS TEXT IN MY LOG SO IT DOES NOT RAISES ANY EXCEPTION
self.driver.save_screenshot('click_next_page_exception.png')
self.driver.back()
self.driver.forward()
rep += 1
log('REPEAT '+str(rep))
if rep>4:
break
sleep(4)
return False
The problem is that it does not raises any exception or any message.
The line log('click_next_page : '+self.driver.current_url) is working and then it freezes, I know it because I have click_next_page : http://.... in my log as a last line.
The problem is definitely somewhere here:
self.wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
self.driver.find_element_by_css_selector('li.arr-rgt.active a').click()
But I can't realize where because it does not raise any Exception.
Could you give me an advice?
I don't have any idea about how Selenium works in PhantomJS. But, I am not seeing any issues within your code. To help you in knowing the exact problem, I would suggest you to debug it in smaller chunks and using one line at a time in console (not by running the python file).
So check with this :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> time.sleep(5)
>>> driver.find_element_by_css_selector('li.arr-rgt.active a')
So, this should return you the selenium webdriver instance for the object you are searching using the css selector. If, the element itself is not found then it will throw error.
If the above code runs then re-run the above code with following modifications :-
>>> from selenium import webdriver
>>> driver = webdriver.PhantomJS(service_args=["--load-images=false"])
>>> wait = WebDriverWait(driver, 2)
>>> code for clicking next page
>>> wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, 'li.arr-rgt.active a')))
>>> driver.find_element_by_css_selector('li.arr-rgt.active a').click()
Here you will be able to check whether there is actually problem with wait_until(). If there is any error, you can point it out by running it one by one. Hope this helps...
I'd like to get the data from inspect element using Python. I'm able to download the source code using BeautifulSoup but now I need the text from inspect element of a webpage. I'd truly appreciate if you could advise me how to do it.
Edit:
By inspect element I mean, in google chrome, right click gives us an option called inspect element which has code related to each element of that particular page. I'd like to extract that code/ just its text strings.
If you want to automatically fetch a web page from Python in a way that runs Javascript, you should look into Selenium. It can automatically drive a web browser (even a headless web browser such as PhantomJS, so you don't have to have a window open).
In order to get the HTML, you'll need to evaluate some javascript. Simple sample code, alter to suit:
from selenium import webdriver
driver = webdriver.PhantomJS()
driver.get("http://google.com")
# This will get the initial html - before javascript
html1 = driver.page_source
# This will get the html after on-load javascript
html2 = driver.execute_script("return document.documentElement.innerHTML;")
Note 1: If you want a specific element or elements, you actually have a couple of options -- parse the HTML in Python, or write more specific JavaScript that returns what you want.
Note 2: if you actually need specific information from Chrome's tools that is not just dynamically generated HTML, you'll need a way to hook into Chrome itself. No way around that.
I would like to update answer from Jason S. I wasn't able to start phantomjs on OS X
driver = webdriver.PhantomJS()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/phantomjs/webdriver.py", line 50, in __init__
self.service.start()
File "/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/selenium/webdriver/phantomjs/service.py", line 74, in start
raise WebDriverException("Unable to start phantomjs with ghostdriver.", e)
selenium.common.exceptions.WebDriverException: Message: Unable to start phantomjs with ghostdriver.
Resolved by answer here by downloading executables
driver = webdriver.PhantomJS("phantomjs-2.0.0-macosx/bin/phantomjs")
Inspect element shows all the HTML of the page which is the same as fetching the html using urllib
do something like this
import urllib
from bs4 import BeautifulSoup as BS
html = urllib.urlopen(URL).read()
soup = BS(html)
print soup.findAll(tag_name).get_text()
BeautifulSoup could be used to parse the html document, and extract anything you want. It's not designed for downloading. You could find the elements you want by it's class and id.