We Keep Coding

Why can't SafariDriver click on a specific element when Chrome, FF, Edge can?

I have a trouble when running Safari UI tests on MAC. I have an element:
wd.find_element_by_xpath("//div[#id='ZoomBundle_people_search__searchTab-innerCt']//fieldset//span[text()='Company Information']")
It collapses/uncollapses a hidden block with filters.
When I perform a click() action in Chrome, FF or Edge, this action performed. But when I run the same code for Safari, I get the exception:
selenium.common.exceptions.ElementNotInteractableException: Message:
C:\python\lib\site-packages\selenium\webdriver\remote\errorhandler.py:242: ElementNotInteractableException
I tried to add WebDriverWait, but the result is the same. It sees the element, but can't interact with it.

It's a SafariDriver issue.
Got a suggestion to try clicking via JS:
wd.execute_script("arguments[0].click();", elem)
instead
elem.click()
And it's working fine now (=

If you don't want to execute javascript, try actual simulation of pressing the keyboard Enter key as follows.
Below sample corresponds to Python code
from selenium.webdriver.common.keys import Keys
driver = webdriver.Safari()
element = driver.find_element_by_xpath("//div[#id='ZoomBundle_people_search__searchTab-innerCt']//fieldset//span[text()='Company Information']")
element.send_keys(Keys.RETURN)
P.S: I am still wondering, if its an issue with SafariDriver then why Apple has not looked into it yet or is it due to something else. While I continue to look out an answer you can try the above method.

If you believe this is a bug in safaridriver, please file a bug report at https://feedbackassistant.apple.com/ with a test case and test script. This behavior is well specified, so any divergence between implementations is probably a bug.

How to check for webpages' popups?

Is there a possibility if I code a program in python that allows to automatically browse a given website using mechanize to detect if there are popup windows (suggesting advertisements or downloading actions ...) using Python ?. I would appreciate any hint (for example, if you give me a library that fulfills this task I would be very happy)

Mechanize cannot handle javascript and popup windows:
How do I use Mechanize to process JavaScript?
Mechanize and Javascript
To accomplish the goal, you need to utilize a real browser, headless or not. This is where selenium would help. It has a built-in support for popup dialogs:
Selenium WebDriver has built-in support for handling popup dialog
boxes. After you’ve triggerd and action that would open a popup, you
can access the alert with the following:
alert = driver.switch_to_alert()
Example (using this jsfiddle):
from selenium import webdriver
url = "http://fiddle.jshell.net/ebkXh/show/"
driver = webdriver.Firefox()
driver.get(url)
button = driver.find_element_by_xpath('//button[#type="submit"]')
# dismiss
button.click()
driver.switch_to.alert.dismiss()
# accept
button.click()
driver.switch_to.alert.accept()
See also:
Handle Popup Windows
Click the javascript popup through webdriver

Unfortunately, Mechanize's browser seems to skip the pop-ups so the title, URL, and HTML are identical for both pop-ups and normal pages.
Frankly, Python is not the right tool for this job and is lagging behind in this respect IMHO. Having spent months doing web crawling, for sites that use Javascript extensively (the number of which is greatly increasing nowadays), I find that using Javascript-Based environments like PhantomJS or SlimerJS are simply better for what you're trying to do.
If you have the luxury to use Javascript-Based environments, I'd say go right ahead. However, you can still use python. PhantomJS embeds Ghost Driver. You can use Ghost.py to utilize the power of PhantomJS. Or you can use Selenium with Python as illustrated here.

Selenium Webdriver - NoSuchElementExceptions

I am using the python unit testing library (unittest) with selenium webdriver. I am trying to find an element by it's name. About half of the time, the tests throw a NoSuchElementException and the other time it does not throw the exception.
I was wondering if it had to do with the selenium webdriver not waiting long enough for the page to load.

driver = webdriver.WhatEverBrowser()
driver.implicitly_wait(60) # This line will cause it to search for 60 seconds
it only needs to be inserted in your code once ( i usually do it right after creating webdriver object)
for example if your page for some reason takes 30 seconds to load ( buy a new server), and the element is one of the last things to show up on the page, it pretty much just keeps checking over and over and over again if the element is there for 60 seconds, THEN if it doesnt find it, it throws the exception.
also make sure your scope is correct, ie: if you are focused on a frame, and the element you are looking for is NOT in that frame, it will NOT find it.

I see that too. What I do is just wait it out...
you could try:
while True:
try:
x = driver.find_by_name('some_name')
break
except NoSuchElementException:
time.sleep(1)
# possibly use driver.get() again if needed
Also, try updating your selenium to the newest version with pip install --update selenium

I put my money on frame as I had similar issue before :)
Check your html again and check if you are dealing with frame. If it is then switching to correct frame will be able to locate the element.
Python
driver.switch_to_frame("frameName")
Then search for element.
If not, try put wait time as others suggested.

One way to handle waiting for an element to appear is like this:
import selenium.webdriver.support.ui as ui
wait = ui.WebDriverWait(driver,10)
wait.until(lambda driver: driver.find_by_name('some_name') )
elem = driver.find_by_name('some_name')
You are correct that the webdriver is not waiting for the page to load, there is no built-in default wait for driver.get().

To resolve this query you have to define explicit wait. so that till the time when page is loading it will not search any WebElement.
below url help on this.
http://docs.seleniumhq.org/docs/04_webdriver_advanced.jsp

You need to have a waitUntil (your element loads). If you are sure that your element will eventually appear on the page, this will ensure that what ever validations will only occur after your expected element is loaded.

I feel it might be synchronisation issue (i.e webdriver speed and application speed is mismatch )
Use Implicit wait:
driver.manage.timeouts.implicitlyWait(9000 TIMEUNITS.miliseconds)
Reference

Python - Automating form entry on a .aspx website and storing output in a file (using Selenium?)

I've just started to learn coding this month and started with Python. I would like to automate a simple task (my first project) - visit a company's career website, retrieve all the jobs posted for the day and store them in a file. So this is what I would like to do, in sequence:
Go to http://www.nov.com/careers/jobsearch.aspx
Select the option - 25 Jobs per page
Select the date option - Today
Click on Search for Jobs
Store results in a file (just the job titles)
I looked around and found that Selenium is the best way to go about handling .aspx pages.
I have done steps 1-4 using Selenium. However, there are two issues:
I do not want the browser opening up. I just need the output saved to a file.
Even if I am ok with the browser popping up, using the Python code (exported from Selenium as Web Driver) on IDLE (i have windows OS) results in errors. When I run the Python code, the browser opens up and the link is loaded. But none of the form selections happen and I get the foll error message (link below), before the browser closes. So what does the error message mean?
http://i.stack.imgur.com/lmcDz.png
Any help/guidance will be appreciated...Thanks!

First about the error you've got, I should say that according to the expression NoSuchElementException and the message Unable to locate element, the selector you provided for the web-driver is wrong and web-driver can't find the element.
Well, since you did not post your code and I can't open the link of the website you entered, I can just give you a sample code and I will count as much details as I can.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("url")
number_option = driver.find_element_by_id("id_for_25_option_indicator")
number_option.click()
date_option = driver.find_element_by_id("id_for_today_option_indicator")
date_option.click()
search_button = driver.find_element_by_id("id_for_search_button")
search_button.click()
all_results = driver.find_elements_by_xpath("some_xpath_that_is_common_between_all_job_results")
result_file = open("result_file.txt", "w")
for result in all_results:
result_file.write(result.text + "\n")
driver.close()
result_file.close()
Since you said you just started to learn coding recently, I think I have to give some explanations:
I recommend you to use driver.find_element_by_id in all cases that elements have ID property. It's more robust.
Instead of result.text, you can use result.get_attribute("value") or result.get_attribute("innerHTML").
That's all came into my mind by now; but it's better if you post your code and we see what is wrong with that. Additionally, it would be great if you gave me a new link to the website, so I can add more details to the code; your current link is broken.

Concerning the first issue, you can simply use a headless browser. This is possible with Chrome as well as Firefox.
Check Grey Li's answer here for example: Python - Firefox Headless
from selenium import webdriver
options = webdriver.FirefoxOptions()
options.add_argument('headless')
driver = webdriver.Firefox(options=options)

Selenium Webdriver for Python: get page, enter values, click submit, get source

Alright, I'm confused. So I want to scrape a page using Selenium Webdriver and Python. I've recorded a test case in the Selenium IDE. It has stuff like
Command Taget
click link=14
But I don't see how to run that in Python. The desirable end result is that I have the source of the final page.
Is there a run_test_case command? Or do I have to write individual command lines? I'm rather missing the link between the test case and the actual automation. Every site tells me how to load the initial page and how to get stuff from that page, but how do I enter values and click on stuff and get the source?
I've seen:
submitButton=driver.find_element_by_xpath("....")
submitButton.click()
Ok. And enter values? And get the source once I've submitted a page? I'm sorry that this is so general, but I really have looked around and haven't found a good tutorial that actually shows me how to do what I thought was the whole point of Selenium Webdriver.

I've never used the IDE. I just write my tests or site automation by hand.
from selenium import webdriver
browser = webdriver.Firefox()
browser.get("http://www.google.com")
print browser.page_source
You could put that in a script and just do python wd_script.py or you could open up a Python shell and type it in by hand, watch the browser open up, watch it get driven by each line. For this to work you will obviously need Firefox installed as well. Not all versions of Firefox work with all versions of Selenium. The current latest versions of each (Firefox 19, Selenium 2.31) do though.
An example showing logging into a form might look like this:
username_field = browser.find_element_by_css_selector("input[type=text]")
username_field.send_keys("my_username")
password_field = browser.find_element_by_css_selector("input[type=password]")
password_field.send_keys("sekretz")
browser.find_element_by_css_selector("input[type=submit]").click()
print browser.page_source
This kind of stuff is much easier to write if you know css well. Weird errors can be caused by trying to find elements that are being generated in JavaScript. You might be looking for them before they exist for instance. It's easy enough to tell if this is the case by putting in a time.sleep for a little while and seeing if that fixes the problem. More elegantly you can abstract some kind of general wait for element function.
If you want to run Webdriver sessions as part of a suite of integration tests then I would suggest using Python's unittest to create them. You drive the browser to the site under test, and make assertions that the actions you are taking leave the page in a state you expect. I can share some examples of how that might work as well if you are interested.