We Keep Coding

Fill out form and download zipfile using Selenium in Python

EDITED: I incorporated the final lines suggested by Sushil. At the end, I am copying the output in my terminal. I still do not get the zipfile.
SOLVED: My error was due to an incompatibility between the driver and chrome versions. I fixed by following the instructions here: unknown error: call function result missing 'value' for Selenium Send Keys even after chromedriver upgrade
I am trying to use Selenium to fill out a form and download a zipfile.
After extensively googling, I have written a Python code, but I am currently unable to download the file. A browser opens, but nothing is filled out.
I am very new at Python, so I am guessing I am missing something very trivial since the website I am trying to get info from is super simple.
This is what I have tried:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome(executable_path='/home/miranda/webscrap_Python/chromedriver')
#driver.wait = WebDriverWait(driver,5)
driver.get("http://www.cis.es/cis/opencms/EN/formulario.jsp?dwld=/Microdatos/MD3288.zip")
Name = '//*[#id="Nombre"]'
LastName = '//*[#id="Apellidos"]'
University = '//*[#id="profesion"]'
email = '//*[#id="Email"]'
ob_req = '//*[#id="objeto1"]'
terms = '//*[#id="Terminos"]'
download = '//*[#id="mediomicrodatos"]/form/div[3]/input'
driver.find_element_by_xpath(Name).send_keys("Miranda")
driver.find_element_by_xpath(LastName).send_keys("MyLastName")
driver.find_element_by_xpath(University).send_keys("MySchool")
driver.find_element_by_xpath(email).send_keys("my_email#gmail.com")
#lines added by Sushil:
ob_req_element = driver.find_element_by_xpath(ob_req) #Finds the ob_req element
driver.execute_script("arguments[0].click();", ob_req_element) #Scrolls down to the element and clicks on it
terms_element = driver.find_element_by_xpath(terms) #The same is repeated here
driver.execute_script("arguments[0].click();", terms_element)
driver.find_element_by_xpath(download).click() #Scrolling down is not needed for the download button as it would already be in view. Only when an element is not in view should we scroll down to it in order to click on it.
Output in my terminal:
Traceback (most recent call last):
File "myCode.py", line 18, in <module>
driver.find_element_by_xpath(Name).send_keys("Miranda")
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 479, in send_keys
'value': keys_to_typing(value)})
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webelement.py", line 633, in _execute
return self._parent.execute(command, params)
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "/home/miranda/anaconda3/lib/python3.7/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: call function result missing 'value'
(Session info: chrome=86.0.4240.75)
(Driver info: chromedriver=2.29.461571 (8a88bbe0775e2a23afda0ceaf2ef7ee74e822cc5),platform=Linux 5.4.0-45-generic x86_64)

For each and every element below the email element, you have to scroll down to click them. Here is the full code to do it:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Chrome()
#driver.wait = WebDriverWait(driver,5)
driver.get("http://www.cis.es/cis/opencms/EN/formulario.jsp?dwld=/Microdatos/MD3288.zip")
Name = '//*[#id="Nombre"]'
LastName = '//*[#id="Apellidos"]'
University = '//*[#id="profesion"]'
email = '//*[#id="Email"]'
ob_req = '//*[#id="objeto1"]'
terms = '//*[#id="Terminos"]'
download = '//*[#id="mediomicrodatos"]/form/div[3]/input'
driver.find_element_by_xpath(Name).send_keys("Miranda")
driver.find_element_by_xpath(LastName).send_keys("MyLastName")
driver.find_element_by_xpath(University).send_keys("MySchool")
driver.find_element_by_xpath(email).send_keys("my_email#gmail.com")
#All the lines below were added by me
ob_req_element = driver.find_element_by_xpath(ob_req) #Finds the ob_req element
driver.execute_script("arguments[0].click();", ob_req_element) #Scrolls down to the element and clicks on it
terms_element = driver.find_element_by_xpath(terms) #The same is repeated here
driver.execute_script("arguments[0].click();", terms_element)
driver.find_element_by_xpath(download).click() #Scrolling down is not needed for the download button as it would already be in view. Only when an element is not in view should we scroll down to it in order to click on it.

Python with Selenium Google Translate Error

I dont know where the problem is
I used theese sources
https://medium.com/python/python-selenium-mod%C3%BCl%C3%BC-kullan%C4%B1m%C4%B1-ders-1-36983185164c
https://www.youtube.com/watch?v=olOswIyeRlY
Code
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time
driver_path = r"C:\Users\ruzgaricatsirketi\Desktop\chromedriver_win32\chromedriver.exe"
driver = webdriver.Chrome(executable_path=driver_path)
driver.get("https://translate.google.com/#tr/tr")
dinle = driver.find_element_by_id('gt-speech')
dinle.click()
time.sleep(2)
dinle.click()
time.sleep(3)
getir = driver.find_element_by_id('result_box').text
print(getir)
Error
DevTools listening on ws://127.0.0.1:12239/devtools/browser/76eb55d7-8aef-4b19-9d6e-fda8f4f1aee4
Traceback (most recent call last):
File "C:\Users\ruzgaricatsirketi\Desktop\test.py", line 13, in <module>
getir = driver.find_element_by_id('result_box').text
File "C:\Users\ruzgaricatsirketi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 360, in find_element_by_id
return self.find_element(by=By.ID, value=id_)
File "C:\Users\ruzgaricatsirketi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 978, in find_element
'value': value})['value']
File "C:\Users\ruzgaricatsirketi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 321, in execute
self.error_handler.check_response(response)
File "C:\Users\ruzgaricatsirketi\AppData\Local\Programs\Python\Python37-32\lib\site-packages\selenium\webdriver\remote\errorhandler.py", line 242, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"result_box"}
(Session info: chrome=78.0.3904.97)
(Driver info: chromedriver=2.35.528161 (5b82f2d2aae0ca24b877009200ced9065a772e73),platform=Windows NT 10.0.18362 x86_64)

In order to work with Python correctly, it is crucial to learn to read the stack trace. In your case, these lines
File "C:\Users\ruzgaricatsirketi\Desktop\test.py", line 13, in <module>
getir = driver.find_element_by_id('result_box').text
say that the exception has its origin on line 13 in your file test.py, where you are trying to search for web element with id attribute set to value "result_box". Exception you have got is NoSuchElementException:
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"id","selector":"result_box"}
As you can find in official documentation or many other places (and I bet there are thousands of them), this exception is raised when the element you are searching for is not found.
Try it yourself — open your tested Google Translator page and try to search for element with any existing id. E.g. on line 8 in your test file you are trying to locate element with id "gt-speech". Write document.getElementById("gt-speech") to your browsers console and whoala! It works, this node matches your search:
<div id="gt-speech" class="speech-button goog-toolbar-button"...>...</div>
Note especially the id="gt-speech" attribute, it is the one selenium will use to identify the object. However, if you try the same with id "result_box", you will get nothing (null is returned if using JS console to search for element), because there is no such element on the page (thus NoSuchElementException...).
My advice for you:
inspect the element you are trying to locate (on the most of the popular web browsers on Windows/Linux systems it is as easy as right-clicking the element and selecting the inspect option)
select some unique properties, you can use also other properties than id (preferably not inner text; IDs are best, but they are not always present...)
use those properties to find the element using selenium
And please, next time try to search a bit on your own before you ask, because this is some really really trivial problem you should be able to handle after first 5 minutes of any selenium course (in other words, UTFG :P)
EDIT
P.S. I can see you have got that invalid id from the video you have marked as your source. The fact that element with such id was present one year ago does not mean it will be there forever (an for every platform). If you need to get that certain element, you have to find another way how to define it using selenium locators I have mentioned earlier.

Selenium "Wait Until Presence of Element Located By" not behaving as expected

I'm running a Python script that uses Selenium + Chromedriver to automate a series of actions on a web app. Several of these actions causes the page or frame to reload. Since Selenium does not have a built-in "wait for page load" feature and I would rather not use time.sleep(x) if I can avoid it, I have used this combo frequently:
wait = WebDriverWait(driver, 10)
driver.find_element_by_id("sysverb_update_and_stay").click() #causes form to submit and data to load
element = wait.until(EC.presence_of_element_located((By.ID, "view.change_task.parent")))
element = wait.until(EC.element_to_be_clickable((By.ID, "view.change_task.parent")))
Based on the documentation I have read (and the name of the attribute itself), I expected that the script would wait until an element with that ID was present before continuing, and that if it did not find the element within the script's built-in timeout, only then would the script crash out and throw an exception. However, as soon as the script hits the the first wait line, it crashes and throws the following exception:
Traceback (most recent call last):
File "servicenow.py", line 137, in test_change_control
element = wait.until(EC.presence_of_element_located((By.ID, "view.change_task.parent")
))
...
selenium.common.exceptions.WebDriverException: Message: unknown error: unhandled inspec
tor error: {"code":-32000,"message":"Cannot find context with specified id"}
If it can't find that element by the ID, isn't it supposed to, you know, wait for it to be present? What am I missing here?
EDIT: Added wait in code above, and HTML for element below:
<a id="view.change_task.parent" name="view.change_task.parent" style="" class="btn btn-default btn-ref icon icon-info" data-type="reference_popup" data-table="change_task" data-form="task.do" data-ref="change_task.parent" data-ref-key="null" data-view="" tabindex="-1"><span class="sr-only">View - Opens reference record in current window</span></a>

This appears to be an issue with earlier versions of Chromedriver -- updating to the latest version appears to have solved this.

Selenium with Python: Stale Element Reference Exception

Working from Test Driven Development with Python, and I'm currently encountering a 'StaleElementReferenceException' when running the functional test immediately after migration. Here's the full text of the error:
ERROR: test_start_and_retrieve_list (__main__.NewVisitorTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "functional_tests.py", line 53, in test_start_and_retrieve_list
rows = table.find_elements_by_tag_name('tr')
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 237, in find_elements_by_tag_name
return self.find_elements(by=By.TAG_NAME, value=name)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 527, in find_elements
{"using": by, "value": value})['value']
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webelement.py", line 493, in _execute
return self._parent.execute(command, params)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/webdriver.py", line 256, in execute
self.error_handler.check_response(response)
File "/usr/local/lib/python3.5/dist-packages/selenium/webdriver/remote/errorhandler.py", line 194, in check_response
raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.StaleElementReferenceException: Message: The element reference of <table id="id_list_table"> stale: either the element is no longer attached to the DOM or the page has been refreshed
----------------------------------------------------------------------
Ran 1 test in 8.735s
FAILED (errors=1)
Here's the test:
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import unittest
class NewVisitorTest(unittest.TestCase):
def setUp(self):
self.browser = webdriver.Firefox()
self.browser.implicitly_wait(3)
def tearDown(self):
self.browser.close()
def check_for_row(self, row_text):
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.assertIn(row_text, [row.text for row in rows])
def test_start_and_retrieve_list(self):
self.browser.get('http://localhost:8000')
self.assertIn('To-Do', self.browser.title)
header_text = self.browser.find_element_by_tag_name('h1').text
self.assertIn('To-Do', header_text)
inputbox = self.browser.find_element_by_id('id_new_item')
self.assertEqual(
inputbox.get_attribute('placeholder'),
'Enter a to-do item'
)
inputbox.send_keys('Buy peacock feathers')
inputbox.send_keys(Keys.ENTER)
self.check_for_row('1: Buy peacock feathers')
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
table = self.browser.find_element_by_id('id_list_table')
rows = table.find_elements_by_tag_name('tr')
self.check_for_row('1: Buy peacock feathers')
self.check_for_row('2: Use peacock feathers to make a fly')
self.fail('Finish the test!')
if __name__ == '__main__':
unittest.main(warnings='ignore')
How do I configure the test to prevent this? Selenium's own page says this issue can occur when the page refreshes, but this is a necessary part of the application logic as it's configured so far.

Add these imports:
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions
Change these lines
inputbox.send_keys(Keys.ENTER)
self.check_for_row('1: Buy peacock feathers')
to:
inputbox.send_keys(Keys.ENTER)
WebDriverWait(self.browser, 10).until(
expected_conditions.text_to_be_present_in_element(
(By.ID, 'id_list_table'), 'Buy peacock feathers'))
self.check_for_row('1: Buy peacock feathers')
This replaces the time.sleep(1) with something more "reasonable"

I have been using selenium for a while now so I understand the struggles of the Stale Element Exception. While not perfect, selenium provides a series of "wait" commands to allow for the website to load complete. Unfortunately, its not perfect as loading can take different time on each run, but these are the tools provided by selenium.

I haven't worked in python but have worked on java/selenium. But,I can give you the idea to overcome staleness.
Generally we will be getting the Stale Exception if the element attributes or something is changed after initiating the webelement. For example, in some cases if user tries to click on the same element on the same page but after page refresh, gets staleelement exception.
To overcome this, we can create the fresh webelement in case if the page is changed or refreshed. Below code can give you some idea.(It's in java but the concept will be same)
Example:
webElement element = driver.findElement(by.xpath("//*[#id='StackOverflow']"));
element.click();
//page is refreshed
element.click();//This will obviously throw stale exception
To overcome this, we can store the xpath in some string and use it create a fresh webelement as we go.
String xpath = "//*[#id='StackOverflow']";
driver.findElement(by.xpath(xpath)).click();
//page has been refreshed. Now create a new element and work on it
driver.fineElement(by.xpath(xpath)).click(); //This works
Another example:
for(int i = 0; i<5; i++)
{
String value = driver.findElement(by.xpath("//.....["+i+"]")).getText);
System.out.println(value);
}
Hope this helps you. Thanks

To prevent an element to become stale, place a new element on the current page, hit the link and wait until the element is not available anymore. Then wait for an element on the new page to appear
script_to_execute = """
var new_element = document.createElement('span');
new_element.setAttribute('id', 'wait4me');
document.body.appendChild(new_element);"""
self.driver.execute_script(script_to_execute)
self.driver.find_element_by_xpath("{}".format(locator)).click()
WebDriverWait(self.driver, self.time_out).until (
lambda x: not x.find_elements_by_id("wait4me"))
This issue happens when the loop starts before an updated page has fully loaded. Especially when you update a page in an application or a form.
One workaround is to place an element on the current page, then update and use the WebDriverWait statement until the element is not found anymore.
Then start your loop. (Otherwise the reload happens during the loop...)

I read the same book as you do and encountered the same problem (solutions from this page didn't work for me).
Here's how I resolved it.
Problem
Exception is thrown whenever you try to access a stale object. So we have to wait for situation when this exception is NOT thrown anymore.
My solution
I created method that waits for my actions until they pass
from selenium.common.exceptions import StaleElementReferenceException
[...]
def stale_aware_for_action(self, action):
while(True):
try:
action()
break
except StaleElementReferenceException:
continue
And in test method I defined actions that I want to wait to finish:
def test_can_start_a_list_and_retrieve_it_later(self):
[...]
def insert_second_item_to_inputbox():
inputbox = self.browser.find_element_by_id('id_new_item')
inputbox.send_keys('Use peacock feathers to make a fly')
inputbox.send_keys(Keys.ENTER)
self.stale_aware_for_action(insert_second_item_to_inputbox)
def check_for_first_item():
self.check_for_row_in_list_table('1: Buy peacock feathers')
def check_for_second_item():
self.check_for_row_in_list_table('2: Use peacock feathers to make a fly')
self.stale_aware_for_action(check_for_first_item)
self.stale_aware_for_action(check_for_second_item)

Click on a dropdown element menu with Selenium Webdriver

I'm trying to automate an administration task, so far I have made selenium to click on an element to show a dropdown menu.
When it comes the time to click on one of those menu elements I've got an error saying that the element must be displayed.
Code:
driver = webdriver.Chrome()
driver.implicitly_wait(10)
driver.get(url)
doc = driver.find_element_by_css_selector('td.ms-vb-title > table')
try:
doc.click()
time.sleep(4)
menu = driver.find_element_by_xpath('//menu/span[5]')
time.sleep(4)
print dir(menu)
menu.click()
except:
traceback.print_exc()
driver.quit()
Error:
Traceback (most recent call last):
File "aprobar_docs.py", line 22, in main
menu.click()
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 52, in click
self._execute(Command.CLICK_ELEMENT)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webelement.py",
line 205, in _execute
return self._parent.execute(command, params)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", l
ine 156, in execute
self.error_handler.check_response(response)
File "C:\Python27\lib\site-packages\selenium\webdriver\remote\errorhandler.py"
, line 147, in check_response
raise exception_class(message, screen, stacktrace)
ElementNotVisibleException: Message: u'Element must be displayed to click'
As you can see the code waits a lot to get the element loaded. I've also tried to set the element's is_displayed property to True but didn't work neither.
Note: the element that's not displayed is the one on the xpath search, it is present because I've logged it with dir(menu)
Edit:
The menu variable is not the menu itself it's one of the spans that are elements of the menu, doc is the Perfil html element getting clicked to show the dropdown.
Edit 2:
Inspecting the DOM on chrome tools, when you click a doc a new menu gets created on the tree, I don't know if it's because of an ajax call or vanilla js, I don't think it's really that important how it's created. I can't retrieve it from the page and make a python object from it, it's just not being displayed at least on code.
Final Edit:
I ended up executing some JavaScript to make it work. Apparently when Selenium finds the menu item the first element that triggers the menu drop down loses the focus and it makes the menu invisible again, if you don't select a menu item and wait for some time the menu dropdown still is shown, if you try to select one element from the menu the menu disappears.

Why don't you select an option like this
el = driver.find_element_by_id('id_of_select')
for option in el.find_elements_by_tag_name('option'):
if option.text == 'The Options I Am Looking For':
option.click() # select() in earlier versions of webdriver
If your click is not firing an ajax call to populate your list, you don't actually need to execute the click.

You need to find the link of the target. You don't really click elements, you click links... (or rather, you click elements with links inside them). That being said, the most sure fire way to click a link is to isolate the link element.
frame = driver.find_element_by_id('this_is_your_frame_name')
links = frame.find_elements_by_xpath('.//a')
links[1].click()
or alternatively,
for link in links:
if link.text() == "Administratar Permisos":
link.click()