We Keep Coding

How can I access the same website twice without losing the settings, using Selenium?

I access a website, login and then instead of going through the process of finding and writing into the website's search field, I thought I'd simply re-access the website through a URL with the search query I want.
The problem is that when I access the website with the second "driver.get" (last line of code in the code below), it's as though it forgets that I logged in previously; as though it was a totally new session that I opened.
I have this code structure:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.chrome.service import Service
path = Service("C://chromedriver.exe")
driver = webdriver.Chrome(service=path)
driver.get('https://testwebsite.com/')
login_email_button = driver.find_element(By.XPATH,'XXXXX')
login_email_button.click()
username = driver.find_element(By.ID, 'email')
password = driver.find_element(By.ID, 'password')
username.send_keys('myuser')
password.send_keys('mypassword')
driver.get('https://testwebsite.com/search?=televisions')

when you do
driver.get('https://testwebsite.com/search?=televisions')
you're opening new session with no cookie or data of previous session. You can try to duplicate tab instead, to keep you logged in. You can do with:
Driver.execute_script
url = driver.current_url
driver.execute_script(f"window.open('{url}');")
driver.switch_to.window(window_handles[1])
# if you want give a name to tab, pass it as second param like
driver.execute_script(f"window.open('{url}', 'second_tab_name');")
driver.switch_to.window('second_tab_name')
remember to use the switch if you want go back to the main tab

Selenium can't find 99% of elements, why?

I'm using Selenium with Python and Firefox on Windows 7.
Running through my script, at some point Selenium can't find any elements beside and after I navigated to a certain part of a website.
The website belongs to an E-Mail provider called freemail (https://web.de/). I have set up a test account to complete a training project from chapter 11 of the book "Automate the Boring Stuff with Python".
Python, Selenium and Firefox are all up to date. And before this certain part of the website everything was working fine (including finding tags, class names, id's etc.).
I tried like 10 different ways to click one of the ways to navigate to the inbox of the E-Mail provider but had no luck in making it work.
Here is the HTMLcode with the highlighted element I want Selenium to click.
Here another image of the Website itself with the button highlighted in the upper left corner.
And this is the code for my script:
## This program navigates through the website of an
## e-mail provider and sends a mail automatically.
# Import necessary modules
from selenium import webdriver
# Mail account info
mailLink = 'https://web.de/'
mailAddress = 'testing2772#web.de' # This is just a test account
mailPw = 'ewaa11ewaa11'
# Recipient % content of the e-mail
recipient = 'pradicradr#matra.top'
mailContent = 'Test this awesome string.'
# Set up browser
browser = webdriver.Firefox()
browser.get(mailLink)
# Log into account
browser.find_element_by_class_name('icon-freemail').click()
browser.find_element_by_id('freemailLoginUsername').send_keys(mailAddress)
pwElem = browser.find_element_by_id('freemailLoginPassword')
pwElem.send_keys(mailPw)
pwElem.submit()
# Close notification overlay if it pops up
try:
browser.find_element_by_class_name('layerCloser').click()
except:
pass
## At this point Selenium can't Find 99% of elements including the one I need
## Testing if Selenium finds basic elements
testElem1 = browser.find_element_by_tag_name('html') # works
testElem2 = browser.find_element_by_tag_name('body') # works
testElem3 = browser.find_element_by_tag_name('div') # works
## Click link to inbox (all of these variants won't work somehow)
linkToInbox = browser.find_element_by_class_name('nx-track-sec-click-communication-newmail') # try by class_name
linkToInbox = browser.find_element_by_xpath("/html/body/div[3]/div/div[3]/div[1]/ul/li[3]/a") # try by XPath
linkToInbox = browser.find_element_by_link_text('E-Mail schreiben') # try by link_text
linkToInbox = browser.find_element_by_css_selector('a.nx-track-sec-click-communication-newmail') # try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a.newmail') # try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a.nx-track nx-track-sec-click-communication-newmail newmail') # another try by CSS selector
linkToInbox = browser.find_element_by_css_selector('a[title="E-Mail schreiben"]') # another try by CSS selector
## TODO: Click link to write a new mail
## TODO: Fill in recipient and mail content
## TODO: Send mail and close program
I guess it has something to do with an iframe? But I'm a beginner and I don't know how to handle this :/
Thanks in advance!
EDIT:
I solved the problem by finding another button that linked to the inbox with
browser.find_element_by_xpath("/html/body/atlas-app/atl-navbar/atl-actions-menu/div[1]/div[1]/atl-menu-item[2]/pos-icon-item/span/span").click(). This button is not within an iframe, so no problem here.
However, it is still strange that I can't enter any iframes. Here is another image to make it more clear. Does anybody know how to switch to that iframe?
browser.switch_to.default_content() and then browser.switch_to.frame('home') doesn't do the trick...
SECOND EDIT:
KunduK's answer did it.
The problem was, that the script needs to wait for the iframe to become available and then for the button to become available after that. See his post further down for the exact approach. Thanks everyone!

To click on the E-Mail schreiben link you need to switched to iframe first and then click on link.Use WebDriverWait and frame_to_be_available_and_switch_to_it and then use element_to_be_clickable
WebDriverWait(browser,20).until(EC.frame_to_be_available_and_switch_to_it((By.NAME,'home')))
WebDriverWait(browser,10).until(EC.element_to_be_clickable((By.XPATH,'//div[#id="navigation"]//ul//li[#class="item"]//a[#title="E-Mail schreiben"]'))).click()
You need to use following imports to execute above code.
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

I guess it has something to do with an iframe?
Since the elements you are trying to find are in an iframe, you must first switch to the iframe before finding the elements. Your iframe has a name=home attribute, so you should be able to add:
driver.switch_to.frame('home')
... then find the elements.
Once you are done interacting with the iframe and want to switch back to the top level content, use:
driver.switch_to.default_content()

Option 1
linkToInbox = browser.find_element_by_css_selector('a.nx-track.nx-track-sec-click-communication-newmail.newmail') # try by CSS selector
Option 2
use the title property of the anchor link
linkToInbox = browser.find_element_by_css_selector('a[title="E-Mail schreiben"]')
option 3
re-read the docs: https://selenium-python.readthedocs.io/locating-elements.html

How can I click on a checkbox in a webpage using selenium mimicking manual approach?

I've written a script in python using selenium to tick a checkbox and hit the submit button. When I follow the steps manually, I can do it without solving any captcha. In fact, I do not face any captcha challenge. However, the site throws captchas as soon as I initiate a click on that checkbox using the script below.
website address
This is what I've tried so far:
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
driver = webdriver.Chrome()
wait = WebDriverWait(driver, 10)
driver.get('https://www.truepeoplesearch.com/results?name=John%20Smithers')
WebDriverWait(driver,10).until(EC.frame_to_be_available_and_switch_to_it(driver.find_element_by_css_selector("iframe")))
WebDriverWait(driver,10).until(EC.presence_of_element_located((By.CSS_SELECTOR, "span#recaptcha-anchor"))).click()
How can I click on a checkbox in a webpage using selenium without triggering captchas?

You can use the PyMouse package (python package here) to move to the (x,y) position of the object on the webpage and simulate a mouse click.
from pymouse import PyMouse
mouse = PyMouse()
def click(self, x,y):
"""Mouse event click for webdriver"""
global mouse
mouse.click(x,y,1)

CAPTCHA is used to stop website automation & that's why it can not be automated using selenium. Adn for same reason, your not able to select CAPTCHA tick box. Please refer these link for more info: https://sqa.stackexchange.com/questions/17022/how-to-fill-captcha-using-test-automation

Here is the sample code to select the check box that will trigger the recaptcha images.
url = "https://www.google.com/recaptcha/api2/demo"
driver.get(url)
driver.switch_to.frame(driver.find_element_by_xpath("//iframe[starts-with(#name,'a-')]"))
# update the class name based on the UAT implementation (if it's different)
driver.find_element_by_class_name("recaptcha-checkbox-border").click()
But still you have to complete either image selection/use voice-to-text api to resolve the captcha.
The possible options are using 3rd party APIs or check you have the APIs available in truepeoplesearch where you can get the required information as response.
Edit 1: Using the API and html parser.
url = "https://www.truepeoplesearch.com/results?name=John%20Smithers"
payload = {}
headers= {}
response = requests.request("GET", url, headers=headers, data = payload)
html_content = response.text.encode('utf8')
# now you can load this content into the lxml.html parser and get the information
html_content = response.text.encode('utf8')
root=lxml.html.document_fromstring(html_content)
content=root.xpath("//div[#class='h4']") # here I am get the names
for name in content:
print(name.text_content() + '\n')

If you are working on the team that develops this site, you can agree with the developers about an efficient way to work around the captcha.
For example, they could made a case in the code, captcha to not be shown if there is a cookie with hard to guess name, known only to you and them. Potentially someone can guess that cookie, but if you have no other choice, this is an option.
You can also use a separate key for testing environments as explained here.

Selenium Webdriver: window determined as alert (HTTP Basic Access Authentication), how to login (Python)

Sript click a link like <a href="/loginform.do">
and it's open a new pop up window with login/password fields (OK/Cancel buttons).
And webdriver for some reason determine this pop up as alert not window...
if try
for handle in browser.window_handles:
print(handle)
it's return exception: UnexpectedAlertPresentException: Alert Text: Message: Modal dialog present
if try
alert=browser.switch_to_alert()
alert.send_keys('userlogin')
it's working (userlogin inserted into first field).
And then if trying to move cursor to next field (password) by TAB key
alert.send_keys(Keys.TAB)
it's replace text at the first field to bullet symbol instead of tabbing to next field...
Not only TAB, even ENTER working in the same way
alert.send_keys(Keys.ENTER)
it's paste the same bullet symbol instead of sending request to a server
And
alert.accept()
doing nothing (no sending request to a server)
What can be done here?
It's an alert, so find_element_by_... methods are not applicable.
TAB by send_keys is not working (no tabbing).
Is the only solution to click by coordinates (using pyautogui module as an option)?
Or is it possible to do more with webdriver?
BTW using IE (IEDriverServer)
And some searching for close questions
31152912 close question, but answer is about find_element_by_name (no such attribute for 'Alert' object)
31152912, 29516740, 27322871 answers about using 'http://username:password#', unfortunatelly it's not working
So an example with login form of .htaccess password protection (advancedhtml.co.uk/password.html)
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
browser=webdriver.Ie('c:\\...\\IEDriverServer.exe')
page='http://www.advancedhtml.co.uk/password/'
browser.get(page)
alert=browser.switch_to_alert()
#alert=browser.switch_to.alert() # this should work, but it doesn't: TypeError: 'Alert' object is not callable
alert.send_keys('user')
alert.send_keys(Keys.TAB) # no tabbing, bullet symbol instead...
#alert.send_keys('password')

So this kind of login page is HTTP Basic Access Authentication. And looks the best way to handle such authentication is to sent post request (Requests package)
Need to pass browser session from Selenium to Requests
(29563335)
The code may look like this:
from selenium import webdriver
import requests
startURL='http://some_url'
browser=webdriver.Ie('c:\\path\\IEDriverServer.exe')
browser.get(startURL)
# an example of finding login link and get the url
loginLink=browser.find_element_by_xpath("//*[contains(text(), 'Log In')]")
loginURL=loginLink.get_attribute('href')
# load cookies from Selenium to Requests (so we could authenticate in current session)
cookies=browser.get_cookies()
s=requests.Session()
for cookie in cookies:
s.cookies.set(cookie['name'], cookie['value'])
# and send the post request
auth=('login', 'password')
r=s.post(url=loginURL, auth=auth)
browser.refresh() # refresh page to see the result

And for .htaccess password protection (example: advancedhtml.co.uk/password.html) it may be handled with additional click at coordinates
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import pyautogui
page='http://www.advancedhtml.co.uk/password/'
browser=webdriver.Ie('c:\\path\\IEDriverServer.exe')
browser.get(page)
alert=browser.switch_to_alert()
alert.send_keys('demo')
#alert.send_keys(Keys.TAB) # tabbing is not working
pyautogui.moveTo(620, 550) # so move mouse cursor to second field
pyautogui.click()
pyautogui.typewrite('password')
alert.accept()

#Litvin, Your approach is correct but you needed to know one more thing in this.
When you do:- obj = driver.switch_to.alert
You can send the keys in one more way as below, which I understood after walking through the Python selenium source code (here).
obj.send_keys(keysToSend="username\ue004password")
Here \ue004 is the value for TAB.

Python - Selenium - Print Webpage

How do I print a webpage using selenium please.
import time
from selenium import webdriver
# Initialise the webdriver
chromeOps=webdriver.ChromeOptions()
chromeOps._binary_location = "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe"
chromeOps._arguments = ["--enable-internal-flash"]
browser = webdriver.Chrome("C:\\Program Files\\Google\\Chrome\\Application\\chromedriver.exe", port=4445, chrome_options=chromeOps)
time.sleep(3)
# Login to Webpage
browser.get('www.webpage.com')
Note: I am using the, at present, current version of Google Chrome: Version 32.0.1700.107 m

While it's not directly printing the webpage, it is easy to take a screenshot of the entire current page:
browser.save_screenshot("screenshot.png")
Then the image can be printed using any image printing library. I haven't personally used any such library so I can't necessarily vouch for it, but a quick search turned up win32print which looks promising.

The key "trick" is that we can execute JavaScript in the selenium browser window using the "execute_script" method of the selenium webdriver, and if you execute the JavaScript command "window.print();" it will activate the browsers print function.
Now, getting it to work elegantly requires setting a few preferences to print silently, remove print progress reporting, etc. Here is a small but functional example that loads up and prints whatever website you put in the last line (where 'http://www.cnn.com/' is now):
import time
from selenium import webdriver
import os
class printing_browser(object):
def __init__(self):
self.profile = webdriver.FirefoxProfile()
self.profile.set_preference("services.sync.prefs.sync.browser.download.manager.showWhenStarting", False)
self.profile.set_preference("pdfjs.disabled", True)
self.profile.set_preference("print.always_print_silent", True)
self.profile.set_preference("print.show_print_progress", False)
self.profile.set_preference("browser.download.show_plugins_in_list",False)
self.driver = webdriver.Firefox(self.profile)
time.sleep(5)
def get_page_and_print(self, page):
self.driver.get(page)
time.sleep(5)
self.driver.execute_script("window.print();")
if __name__ == "__main__":
browser_that_prints = printing_browser()
browser_that_prints.get_page_and_print('http://www.cnn.com/')
The key command you were probably missing was "self.driver.execute_script("window.print();")" but one needs some of that setup in init to make it run smooth so I thought I'd give a fuller example. I think the trick alone is in a comment above so some credit should go there too.