We Keep Coding

How to use Selenium to click a button in a popup modal box

I am trying to use Selenium in Python to pull some data from https://www.seekingalpha.com. The front page has a "Sign-in/Join now" link. I used Selenium to click it, which brought up a popup asking for sign-in information with another "Sign in" button. It seems my code below can enter my username and password, but my attempt to click the "sign in" button didn't get the right response (it clicked on the ad below the popup box.)
I am using Python 3.5.
Here is my code:
driver = webdriver.Chrome()
url = "https://seekingalpha.com"
driver.get(url)
sleep(5)
driver.find_element_by_xpath('//*[#id ="sign-in"]').click()
sleep(5)
driver.find_element_by_xpath('//*[#id ="authentication_login_email"]').send_keys("xxxx#gmail.com")
driver.find_element_by_xpath('//*[#id ="authentication_login_password"]').send_keys("xxxxxxxxx")
driver.find_element_by_xpath('//*[#id="log-btn"]').click()
Any advice/suggestion is greatly appreciated.

EDIT: previous 'answer' was wrong so I have updated it.
Got you man, this is what you need to do:
1.) grab the latest firefox
2.) grab the latest geckodriver
3.) use a firefox driver
driver = webdriver.Firefox(executable_path=r'd:\Python_projects\geckodriver.exe')
url = "https://seekingalpha.com"
driver.get(url)
sign_in = driver.find_element_by_xpath('//*[#id ="sign-in"]')
driver.execute_script('arguments[0].click()', sign_in)
time.sleep(1)
email = driver.find_element_by_xpath('//*[#id ="authentication_login_email"]')
email.send_keys("xxxx#gmail.com")
pw = driver.find_element_by_xpath('//*[#id ="authentication_login_password"]')
pw.send_keys("xxxxxxxxx")
pw.send_keys(Keys.ENTER)
Explanation:
It is easy to detect if selenium is used or not if the browser tells that information (and it seems this page does not want to be scraped):
The webdriver read-only property of the navigator interface indicates whether the user agent is controlled by automation.
I have looked for an answer how to bypass detection and found this article.
Your best of avoiding detection when using Selenium would require you to use one of the latest builds of Firefox which don’t appear to give off any obvious sign that you are using Firefox.
Gave a shot and after launch the correct page design loaded and the login attempt resulted the same like the manual attempt.
Also with a bit more searching found that if you modify your chromedriver, you have a chance to bypass detection even with chromedriver.
Learned something new today too. \o/
An additional idea:
I have made a little experiment using embedded chromium (CEF). If you open a chrome window via selenium and you open the console and check navigator.webdriver the result will be True. If you open a CEF window however and then remote debug it, the flag will be False. I did not check edge cases with it but non-edge-case scenarios should be fine with CEF.
So what you may want to check out in the future:
1.) in command line: pip install cefpython3
2.) git clone https://github.com/cztomczak/cefpython.git
3.) open your CEF project and find hello.pyin the examples
4.) update the startup to cef.Initialize(settings={"remote_debugging_port":9222})
5.) run hello.py
(this was the initial, one time setup, you may customize it in the future, but the main thing is done, you have a browser with a debug port open)
6.) modify chrome startup to:
from selenium.webdriver.chrome.options import Options
chrome_options = Options()
chrome_options.debugger_address = "127.0.0.1:9222"
driver = webdriver.Chrome(chrome_options=chrome_options, executable_path=chrome_driver_executable)
7.) now you have a driver without 'automated' signature in the browser. There may be some drawbacks like:
CEF is not super very latest, right now the latest released chrome is v76, CEF is v66.
also "some stuff" may not work, like window.Notification is not a thing in CEF

I tried code you provided and it works fine. i added selenium wait just to check other options and those also worked well i changed 2 lines instead of sleeps
driver.get(url)
wait = WebDriverWait(driver, 10)
signin = wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id ='sign-in']")))
#sleep(5)
signin.click()
#driver.find_element_by_xpath('//*[#id ="sign-in"]').click()
#sleep(5)
wait.until(EC.element_to_be_clickable((By.XPATH, "//*[#id ='authentication_login_email']")))
driver.find_element_by_xpath('//*[#id ="authentication_login_email"]').send_keys("xxxx#gmail.com")
and it does click on Sign in button. and what i found is there is captcha handling on the site when i checked console after clicked on sign in button it tell the story. I went ahead and added user agent to your script but it did not worked as well. Notice the blockscript parameter in response of login API and console errors in below screenshots. However there is no captcha on the ui -

Is there a way around cookie and privacy popus in selenium browser?

So I'm trying to make a python script using selenium and bs4 to automatically buy shoes for from adidas.com. It's just that whenever selenium browser starts the adidas site it shows a popup concerning cookies and privacy. I can't click on the accept button using selenium(can't find the element to click on) and I've tried starting the selenium browser with my firefox profile containing cookies and what not. But it still shows that damn popup and it's stopping the script.
Is there anyway to fix this?
Tried starting selenium with firefox cookies like this:
ffprofile = webdriver.FirefoxProfile(r'C:\Users\chico\AppData\Local\Mozilla\Firefox\Profiles\e5108gza.default')
driver = webdriver.Firefox(firefox_profile=ffprofile)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()
The popup from the adidas site just keeps poping up and stopping my script from continuing. Sometimes it will fill in email and password before being stopped, sometimes it will be stopped before that.

I ran into this same issue recently, and it was SO frustrating, because it can't be handled with a "switch to alert" and "accept" it approach sometimes.
Here's how I got around my issue. First, and this was most important, you need to "inspect" the page you are writing selenium code for in a way that the cookie acceptance popup will show up.
For me, I was using the Chrome driver, and so for inspecting the page, I had to run incognito, so that the page would not know I had already accepted cookies. Use the method for Firefox that will achieve the same effect.
Then you can find the element ID (or class name) for the acceptance button and add a click to the end of your command. My code to do this looks as follows:
from selenium import webdriver
import time
URL = "http:some_site.com"
driver = webdriver.Chrome()
driver.get(URL)
time.sleep(5)
driver.find_element_by_id(the_id_name_for_cookie_acceptance_as_a_string).click()
# Do more stuff ...
Now, if you run into a cookie dialog that won't act as an alert, you can handle it.

I tried the code you posted and didn't receive any alert pop up.But if there is a Alert windows it needs to be handled by switching to it and accepting it
from selenium import webdriver
driver = webdriver.Firefox()
driver.set_page_load_timeout(30)
driver.get('https://www.adidas.nl/on/demandware.store/Sites-adidas-NL-Site/nl_NL/MyAccount-CreateOrLogin')
try:
WebDriverWait(browser, 3).until(EC.alert_is_present(),'Timed out waiting for alert popup')
alert = driver.switch_to.alert
alert.accept()
except:
print "no alert"
driver.find_element_by_xpath("//input[#id='dwfrm_login_username']").send_keys('email')
driver.find_element_by_xpath("//input[#id='dwfrm_login_password']").send_keys('password')
driver.find_element_by_xpath("//button[#value='Inloggen']").click()

How to send CTRL-P to a web browser in Python

The aim of this is to open a browser window and save the site as PDF.
I'm writing Python code that:
1) Opens a web page
2) Does a control-p to bring up the print dialog box
NOTE: I will have pre-configured the browser to save as PDF instead of defaulting as printing to a printer)
3) Does "return"
4) Enters the file name
5) Does "return" again
NOTE: In my full code, I'll be doing these steps hundreds of times
I'm having a problem early on with control-p. As a test, I'm able to send dummy text to Google's search, but I can't seem to be able to send a control-p (no error messages). I'm using Google as an easy example, but my final code will use various other sites.
Obviously I'm missing something but just can't figure out what.
I tried an alternate method of using javascript instead of ActionChains:
driver.execute_script('window.print();')
This worked in getting the print dialog but I wasn't able to feed anything else in that dialog box (like , file name and location for the pdf).
I tried PDFkit, to convert the web page into pdf. It worked on some sites, but it crashed often (depending on what the site returned), the page was sometimes poorly formatted and some sites (pinterest) just didn't render at all. For this reason, I changed method and decided to use selenium and Chrome in order for the pdf to render just like it shows in the browser.
I thought about using "element.send_keys("some text")" instead of ActionChains, but since I'm going across multiple different web sites, I don't necessarily know what element to look for.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.action_chains import ActionChains
import time
DRIVER = 'chromedriver'
driver = webdriver.Chrome(DRIVER)
URL = "http://www.google.com"
driver.get(URL)
actions = ActionChains(driver)
time.sleep(5) #Give the page time to load
actions.key_down(Keys.CONTROL)
actions.send_keys('p')
actions.key_up(Keys.CONTROL)
actions.perform()
time.sleep(5) #Sleep to see if the print dialog came up
driver.quit()

You can use autoit to achieve your requirement.
First do pip install -U pyautoit
from selenium import webdriver
import autoit
import time
DRIVER = 'chromedriver'
driver = webdriver.Chrome(DRIVER)
driver.get('http://google.com')
driver.maximize_window()
time.sleep(10)
autoit.send("^p")
time.sleep(10) # Pause to allow you to inspect the browser.
driver.quit()
Please let me know if it's working.

try this:
webdriver.ActionChains(driver).key_down(Keys.CONTROL).send_keys('P').key_up
(Keys.CONTROL).perform()

check this out :
robot.keyPress(KeyEvent.VK_CONTROL)
robot.keyPress(KeyEvent.VK_P)
// CTRL+P is now pressed
robot.keyRelease(KeyEvent.VK_P)
robot.keyRelease(KeyEvent.VK_CONTROL)

logging into a website and downloading a file

I would like to log into a website and download a file. I'm using selenium and the chromedriver. Would like to know if there is a better way. It currently opens up a chrome browser window and sends the info. I don't want to see the browser window opened up and the data being sent. Just want to send it and return the data into a variable.
from selenium import webdriver
driver = webdriver.Chrome()
def site_login(URL,ID_username,ID_password,ID_submit,name,pas):
driver.get(URL)
driver.find_element_by_id(ID_username).send_keys(name)
driver.find_element_by_id(ID_password).send_keys(pas)
driver.find_element_by_id(ID_submit).click()
URL = "www.mywebsite.com/login"
ID_username = "name"
ID_password = "password"
ID_submit = "submit"
name = "myemail#mail.com"
pas = "mypassword"
resp=site_login(URL,ID_username,ID_password,ID_submit,name,pas)

You can run chrome in headless mode. In which case, the chrome UI won't show up and still performing the task you were doing. Some article I found on this https://intoli.com/blog/running-selenium-with-headless-chrome/. Hope this helps.

First option: If you are able to change the driver, you can use phantom-js as driver. That was a headless browser and you can use it with selenium.
Second option: If the site are not dynamic (easily called it SPA) or you are able to trace packet (which can be done in chrome dev tools), you can directly use request with the help of beautifulsoup if you need to get some data on the page.

Just add this two lines
chrome_options = Options()
chrome_options.add_argument("--headless")
This should make chrome run in the background.

Handling "Authentication Required" alert box with Python 2.7 + Selenium Webdriver

I am having an issue with a secure URL:
Opening the URL creates an "Authentication Required" alert box with username and password fields.
I am fairly new to Selenium Webdriver and Python. I am not familiar with handling alerts and am currently manually typing in credentials until I can get this figured out. I have already tried adding my username/password into the URL. This does not work for me.
Could someone please point me in the direction of entering keys into username and password fields in an alertbox?

Could you try using Keys to tab within the alert?
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.alert import Alert
from selenium.webdriver.support.ui import WebDriverWait as wait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Firefox()
driver.get('http://www.url.com/')
wait(driver, 5).until(EC.alert_is_present())
alert = driver.switch_to_alert()
alert.send_keys('username')
alert.send_keys(Keys.TAB)
alert.send_keys('password')
alert.accept()

In case of such authentication, you need pass username and password to server while accessing page to avoid authentication window(which is out of reach of selenium)
Suppose the url you're trying to access is: http://example.com
you'll have to access this url with credentials like following:
driver.get('http://username:password#example.com')
where username is your username and password is your password for the site.

Thanks for all of the responses. Unfortunately, none of these solutions worked for me. I suspect it may have something to do with the creation of a new profile every time firefox was opened by webdriver.
My workaround:
I changed the driver from Firefox to IE, after installing the 32bit IE driver(http://selenium-release.storage.googleapis.com/index.html?path=2.44/). This solved my issue by no longer creating the alertbox, and allowing me to continue with my unittest.

I was having similar issues where adding my username/password into the URL did not work for me. This was because Firefox was alerting me with a confirmation box requiring me to confirm that I wanted to log into the site with the username provided. The following solved this issue:
from selenium import webdriver
driver = webdriver.Firefox()
driver.get('https://<username>:<password>#<site-needing-auth>.com')
alert = driver.switch_to_alert()
alert.accept()

None of the answer before helped with my situation. The website I am authenticating to uses single sign on which was posing issues when using username:password#website.com.
In the Authentication window two fields needed to be entered both User Name and Password.
To solve this send the user name and password at one time to the alert box.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.common.alert import Alert
def login(self):
self.browser = webdriver.Firefox()
self.browser.get(r'websitelogin.com')
wait(self.browser, 1).until(expected_conditions.alert_is_present())
# "Switch" to the Alert browser
alert = Alert(self.browser)
# Send the username, TAB then password all in one go using a python f string
alert.send_keys(f'username{Keys.TAB}password')
alert.accept()

I was having the exact same problems as you until I noticed, that I simply forgot to write: 'https' instead of just http. If you add the 's', for me that did it!
So in code maybe you want to try:
driver.get('https://username:password#domain-name.org')

The below Python code can help to load such a URL prompting for authentication within a JavaScript Popup alert, I was also stuck here for a long time. It's because of Chrome driver will not allow such authentication techniques after the update 59 (probably). There are still backdoors via Selenium using the JavaScript engine in the browser to load such URLs.
driver.get("https://www.google.com")
URL = "https://{username}:{password}#www.example.com"
driver.executeScript("window.open('"+URL+"')")