I'm trying to log in into Quora website.
On my local machine it runs perfectly.
But on SSH server (droplet on DigitalOcean) - no, I'm getting InvalidElementStateException
I've tried to focus on the element by send_keys(Keys.NULL), and got ElementNotVisibleException
Here is the code:
driver.get("https://www.quora.com/")
print("Logging...")
# gets email and password from json
with open('config.json') as f:
login_data = json.load(f)
email = login_data['email']
password = login_data['pass']
time.sleep(3)
email_field_xpath = "//div[#class='form_column']/input[#name='email']"
password_field_xpath = "//div[#class='form_column']/input[#name='password']"
# webdriver's going to wait max 10 seconds for email's field, password field, login button to display
email_field_element = WebDriverWait(driver, 10).until(
lambda driver: driver.find_element_by_xpath(email_field_xpath))
password_field_element = WebDriverWait(driver, 10).until(
lambda driver: driver.find_element_by_xpath(password_field_xpath))
email_field_element.send_keys(Keys.NULL)
email_field_element.clear()
email_field_element.send_keys(email)
password_field_element.send_keys(Keys.NULL)
password_field_element.clear()
password_field_element.send_keys(password)
login_button_xpath = "//input[#value='Login']"
# wait till element is clickable
login_button_element = WebDriverWait(driver, 10).until(
EC.element_to_be_clickable((By.XPATH, login_button_xpath)))
login_button_element.click()
print("Logged In.")
It may possible that screen resolution you are using at your local machine and on SSH server are different. May be the size of opened browser windows are different at both at local and on SSH server.
Try to change your local settings as SSH server's than you can figure out what is actual reason or just try the possible solutions given below.
Possible solutions by focusing on element first. Here, your element can be email_field_element or password_field_element:
1) First move to that element using ActionChains:
from selenium.webdriver.common.action_chains import ActionChains
actions = ActionChains(driver)
actions.move_to_element(element)
actions.perform()
OR
2) You can also use scroll up to that element. Like:
driver.execute_script("arguments[0].scrollIntoView(true);", element)
OR
3)Input data using java script.
driver.execute_script("document.getElementById('elementID').setAttribute('value', 'new value for element')" # you can use xpath as well
Hope it may help you.
Related
I am trying to use Selenium to sign up an email account automatically whenever I need to. It's just a fun learning project for me. For the life of me I don't understand why it can't find the element. This code works fine on the sign-in page but not the sign-up page. I have tried all different Selenium commands and even tried using the ID and class name. Either is says it can't locate the element or that it is not reachable by keyboard.
from selenium import webdriver
from selenium.webdriver.firefox.options import Options
import time
options = Options()
driver = webdriver.Firefox(options=options, executable_path=r'geckodriver.exe')
driver.get("https://mail.protonmail.com/create/new?language=en")
time.sleep(10)
username_input = driver.find_element_by_id("username").send_keys("testusername")
Also here is the HTML code: https://i.imgur.com/ZaBMTzG.png
The username field is in iframe, you need to switch to iframe to make this work.
Below is the code that works fine :
driver.get("https://mail.protonmail.com/create/new?language=en")
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='top']"))
driver.find_element_by_id("username").send_keys("some string")
read more about iframe here
learn more about how to switch to iframe/frame/framset using Python
selenium Bindings here
Update :
wait = WebDriverWait(driver, 30)
driver.get("https://mail.protonmail.com/create/new?language=en")
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='top']"))
driver.find_element_by_id("username").send_keys("some string")
driver.switch_to.default_content()
driver.execute_script("window.scrollTo(0, document.body.scrollHeight);")
sleep(5)
driver.switch_to.frame(driver.find_element_by_css_selector("iframe[title='Registration form'][class='bottom']"))
wait.until(EC.element_to_be_clickable((By.NAME, "submitBtn"))).click()
I'm not sure if I've seen enough code to diagnose, but I think the way you are defining username_input seems problematic. driver.find_element_by_id("username").send_keys("testusername") doesn't actually return anything so it seems like you are setting username_input = null.
I'm very new to Selenium and scraping in general and I need some help please. I'm trying to automate some of my work through a website that provides data and queries using Selenium.
The problem is that after the Login, the DOM gets refreshed and apparently that disconects my WebElement, as I no longer am able of doing any find_element, get or any type of function after the login is done. I am even trying to do it manually so that any refresh, wait time load, etc. is not needed because it is under human supervision :(
The same issues happens if instead of doing driver.click() I use driver.submit()
The code is as following:
# import web driver
from selenium import webdriver
import time as time
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
# specifies the path to the chromedriver.exe
driver = webdriver.Chrome('/Users/user/bin/chromedriver')
# driver.get method() will navigate to a page given by the URL address
driver.get('https://app.dnbhoovers.com/')
# locate email form by_class_name
username = driver.find_element_by_xpath('//*[#id="username"]')
# send_keys() to simulate key strokes
username.send_keys(username)
#Sleep some time
time.sleep(0.5)
# locate submit button by_class_name
log_in_button = driver.find_element_by_class_name('continue-btn')
# .click() to mimic button click
log_in_button.click()
#Wait for some seconds as password loads after clicking the contiunue bottom
#try:
driver = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "password"))
)
time.sleep(3)
# locate password form by_class_name
password = driver.find_element_by_xpath('//*[#id="password"]')
time.sleep(0.7)
# send_keys() to simulate key strokes
password.send_keys(password)
#Sleep some time
time.sleep(0.7)
# .click() to mimic button click and Sign in
log_in_button.click()
After the login is where everything goes down, and if I try to type the following functions I get these errors (even after waiting for the page to load):
search = driver.find_element_by_xpath("//input[#class='search-query']")
StaleElementReferenceException: stale element reference: element is not attached to the page document
Or if I even try this simple task I get the following:
driver.get("http://www.google.com")
AttributeError: 'WebElement' object has no attribute 'get'
And this goes on, if I try to refresh or go down with the keyboard I get:
webdriver.ActionChains(driver).send_keys(Keys.DOWN).perform()
AttributeError: 'WebElement' object has no attribute 'w3c'
Even if I add this lines after the .click() command to tell the driver to wait for the search input to appear, I get errors
# ... previous code...#
log_in_button.click()
#New lines of code
WebDriverWait(driver, 20).until(
EC.presence_of_element_located((By.XPATH, "//input[#class='search-query']"))
)
StaleElementReferenceException: stale element reference: element is not attached to the page document
Refreshing the browser will not make any difference either.
If I inspect the element of the webiste, after the login the changes completely and I see some ""<meta name="robots" content="noindex, nofollow>"" also.
I'm really stuck right now so any help is deeply appreciated!
Thanks for your time and for reading this long post! You can have a potato :)
I think the problem is this
driver = WebDriverWait(driver, 5).until(
EC.presence_of_element_located((By.ID, "password"))
)
WebDriverWait() returns a WebElement and you don't want to define driver as such.
driver.get("http://www.google.com")
AttributeError: 'WebElement' object has no attribute 'get'
This is telling you that driver is a WebElement, which you don't want.
Try to wait till it is clickable:
elem = wait.until(EC.element_to_be_clickable((By.XPATH, "//input[#class='search-query']")))
elem.click()
Try 2:
If there is a running JQuery try running this script before click:
wait.until(lambda driver: driver.execute_script('return jQuery.active') == 0)
wait.until(lambda driver: driver.execute_script('return document.readyState') == 'complete')
So I have the following Selenium Chrome Webdriver process that is working correctly locally. Basically, when the below code is run, a purchase is made through Best Buy, and an SMS message is sent to me from my Twilio phone number indicating whether or not the purchase was successful. The variables in all-caps are already defined. I also did not include the imports. Again, the below code is making the purchase SUCCESSFULLY when run locally. This means that the SMS message that I get from my Twilio phone number contains "Success!" every time the following code is run locally:
client = Client(ACCOUNT_SID, AUTH_TOKEN)
def runBestBuyBotLocal():
driver = webdriver.Chrome()
driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[#class='size-l']/div[#class='pl-page-content']//div[#class='container-v2']/div[#class='row v-m-bottom-g']/div[2]//div[#class='col-xs-12']/div[6]/div[#class='None']/div/div/div/button[#type='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#class='c-portal']/div[#role='dialog']/div[1]/div[#role='dialog']/div[#role='document']//a[#role='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='cartApp']/div[#class='page-spinner page-spinner--out']/div[#class='large-view size-l']//div[#class='fluid-large-view']//section[#class='fluid-large-view__sidebar']//button[#type='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='fld-e']")))
element.send_keys(EMAIL)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='fld-p1']")))
element.send_keys(PASSWORD)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#class='cia-app-container']/div[#class='cia-actual-full-page-wrapper lv']/section/main[#class='cia-wrapper container']//form/div[3]/button")))
element.send_keys(Keys.RETURN)
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='credit-card-cvv']"))))
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='credit-card-cvv']")))
element.send_keys(CVV)
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='checkoutApp']/div[#class='page-spinner page-spinner--out']/div[1]/div[1]//div[#class='checkout__container checkout__container-fast-track']/div[#class='checkout__col checkout__col--primary']//div[#class='button--place-order-fast-track']/button"))))
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='checkoutApp']/div[#class='page-spinner page-spinner--out']/div[1]/div[1]//div[#class='checkout__container checkout__container-fast-track']/div[#class='checkout__col checkout__col--primary']//div[#class='button--place-order-fast-track']/button")))
element.send_keys(Keys.RETURN)
try:
runBestBuyBotLocal()
message = client.messages \
.create(
body='Success!',
from_=TWILIONUMBER,
to=MYNUMBER
)
except:
message = client.messages \
.create(
body='Fail! ' + traceback.format_exc(),
from_=TWILIONUMBER,
to=MYNUMBER
)
The following code is the basically the same process but adapted slightly to function on Heroku remote dyno. It is a part of my Heroku web app. The below code is NOT making the purchase successfully when run on Heroku remote dyno. It is instead throwing a Timeout error on the line element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#class='c-portal']/div[#role='dialog']/div[1]/div[#role='dialog']/div[#role='document']//a[#role='button']")))
This is really weird because I have the same line in runBestBuyBotLocal, and when runBestBuyBotLocal gets is executed locally, no timeout exception gets thrown on that line, and the purchase is made successfully.
So, because of the timeout exception being thrown, every time I run the following code on a Heroku remote dyno, I get an SMS message from my Twilio phone number containing "Fail!":
def runBestBuyBotRemote():
chrome_options = webdriver.ChromeOptions()
chrome_options.binary_location = os.environ.get("GOOGLE_CHROME_BIN")
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument("--no-sandbox")
driver = webdriver.Chrome(executable_path=os.environ.get("CHROMEDRIVER_PATH"), chrome_options=chrome_options)
driver.get("https://www.bestbuy.com/site/spongebob-squarepants-mini-plush-styles-may-vary/6404213.p?skuId=6404213")
wait = WebDriverWait(driver, 10)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html/body[#class='size-l']/div[#class='pl-page-content']//div[#class='container-v2']/div[#class='row v-m-bottom-g']/div[2]//div[#class='col-xs-12']/div[6]/div[#class='None']/div/div/div/button[#type='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#class='c-portal']/div[#role='dialog']/div[1]/div[#role='dialog']/div[#role='document']//a[#role='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='cartApp']/div[#class='page-spinner page-spinner--out']/div[#class='large-view size-l']//div[#class='fluid-large-view']//section[#class='fluid-large-view__sidebar']//button[#type='button']")))
element.send_keys(Keys.RETURN)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='fld-e']")))
element.send_keys(EMAIL)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='fld-p1']")))
element.send_keys(PASSWORD)
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#class='cia-app-container']/div[#class='cia-actual-full-page-wrapper lv']/section/main[#class='cia-wrapper container']//form/div[3]/button")))
element.send_keys(Keys.RETURN)
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='credit-card-cvv']"))))
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//input[#id='credit-card-cvv']")))
element.send_keys(CVV)
driver.execute_script("arguments[0].scrollIntoView(true);", WebDriverWait(driver, 10).until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='checkoutApp']/div[#class='page-spinner page-spinner--out']/div[1]/div[1]//div[#class='checkout__container checkout__container-fast-track']/div[#class='checkout__col checkout__col--primary']//div[#class='button--place-order-fast-track']/button"))))
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='checkoutApp']/div[#class='page-spinner page-spinner--out']/div[1]/div[1]//div[#class='checkout__container checkout__container-fast-track']/div[#class='checkout__col checkout__col--primary']//div[#class='button--place-order-fast-track']/button")))
element.send_keys(Keys.RETURN)
client = Client(ACCOUNT_SID, AUTH_TOKEN)
try:
runBestBuyBotRemote()
message = client.messages \
.create(
body='Success!',
from_=TWILIONUMBER,
to=MYNUMBER
)
except:
message = client.messages \
.create(
body='Fail!' + traceback.format_exc(),
from_=TWILIONUMBER,
to=MYNUMBER
)
So, the question is: does anyone have an idea as to why runBestBuyBotLocal is NOT throwing a timeout exception when run locally and why runBestBuyBotRemote IS throwing a timeout exception when run on my Heroku remote dyno? In other words, why am I getting a "Success!" SMS message when the process is run locally and a "Fail!" message when it is run on my Heroku remote dyno?
I'm confused why the process is succeeding when run locally and failing when run on Heroku remote dyno. The only difference between runBestBuyBoyLocal and runBestBuyBotRemote functions is how the webdriver is instantiated. So the reason for one failing and one succeeding could be due to this difference, but I'm not sure.
Let me know if I need to clear up anything!
import traceback
try:
WebDriverWait(driver, 2).until(EC.element_to_be_clickable(
(By.XPATH, '//b[#id = "country_england"]/preceding-sibling::input'))).click()
except:
traceback.print_exc()
use traceback to print the full exception trace .
In my opinion, you should try:
Set a screen resolution: chrome_options.add_argument('window-size=1920x1080'); Websites can have different layout on different resolutions...
Not use full XPaths, for example:
"/html//div[#class='c-portal']/div[#role='dialog']/div[1]/div[#role='dialog']/div[#role='document']//a[#role='button']"
Instead use: "//a[#role='button']", if multiple elements exist in DOM with this property, try to expand condition i.e. "//a[#role='button' and contains(#id, 'foo')]" Look: How to Find XPath?
You need to consider a couple of things as follows:
ChromeDriver initiated Headless Browsing Context i.e. google-chrome-headless opens with a reduced Viewport. So you need to set the window size maximized.
Example:
options = webdriver.ChromeOptions()
options.add_argument("--headless")
options.add_argument("window-size=1400,900")
You can find a detailed discussion in Not able to maximize Chrome Window in headless mode.
If your test step is to interact with the element instead of presence_of_element_located() you need to induce WebDriverWait for element_to_be_clickable().
Example:
WebDriverWait(driver, 10).until(EC.element_to_be_clickable((By.XPATH, "/html/body[#class='size-l']/div[#class='pl-page-content']//div[#class='container-v2']/div[#class='row v-m-bottom-g']/div[2]//div[#class='col-xs-12']/div[6]/div[#class='None']/div/div/div/button[#type='button']"))).send_keys(Keys.RETURN)
You can find a detailed discussion in How to insert a value in a field Text using Selenium?.
Absolute xpath makes your tests brittle. You may like to replace them with relative xpath. As an example, replace:
element = wait.until(EC.presence_of_element_located((By.XPATH, "/html//div[#id='cartApp']/div[#class='page-spinner page-spinner--out']/div[#class='large-view size-l']//div[#class='fluid-large-view']//section[#class='fluid-large-view__sidebar']//button[#type='button']")))
with:
element = wait.until(EC.presence_of_element_located((By.XPATH, "//button[#type='button'][#attribute2='value'][#attribute3='value']")))
All the answers above helped me figure it out. It was all about the flags. Assuming you have the build packs installed, and config vars added (many other answers already mention these), then you may still need the correct flags added as options.
Even thought the Chrome Buildpack says it adds the flags automatically, for me, it did not. I also needed others. If you are scraping a page you need a window size, else you might see unknown error: session deleted because of page crash , and apparently there are sizes issues re: heroku v chrome (or something) here
chrome_options = Options()
chrome_options.add_argument("--headless")
chrome_options.add_argument("--disable-dev-shm-usage")
chrome_options.add_argument('--no-sandbox')
chrome_options.add_argument("--disable-gpu")
## might not be needed
chrome_options.add_argument("--remote-debugging-port=9222")
chrome_options.add_argument('--window-size=1920x1480')
I have some selenium code to log in to jupyterlab (running locally). Without waits, it fails as it tries to find the password input textbox before it exists. So I tried to use an explicit wait as that seems like the cleanest solution, but it works erratically. Implicit waits never work, it seems to block the webserver for 10 seconds before loading the page so always fails. time.sleep always works, however it loads the page and then waits the 10 seconds before entering the password which is inefficient and less clean than the selenium wait methods which as I understand wait up to 10 seconds but stop waiting as soon as the element becomes available. What have I done wrong?
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.common.keys import Keys
from selenium.webdriver.support import expected_conditions as ec
from selenium.webdriver.support.wait import WebDriverWait
Explicit wait - sometimes works
driver = webdriver.Firefox()
driver.get(f"http://localhost:8888")
wait = WebDriverWait(driver, 10)
password_input = wait.until(ec.presence_of_element_located((By.ID, "password_input")))
password = "my_password"
password_input.send_keys(password + Keys.RETURN)
sometimes I get the error:
selenium.common.exceptions.ElementNotInteractableException: Message: Element is not reachable by keyboard
Implicit wait - sometime errors
driver = webdriver.Firefox()
driver.get(f"http://localhost:8888")
driver.implicitly_wait(10)
password_input = driver.find_element_by_css_selector("password_input")
password = "my_password"
password_input.send_keys(password + Keys.RETURN)
sometimes I get the error:
selenium.common.exceptions.ElementNotInteractableException: Message: Element is not reachable by keyboard
time.sleep - always works
driver = webdriver.Firefox()
driver.get(f"http://localhost:8888")
time.sleep(10)
password_input = driver.find_element_by_id("password_input")
password = "my_password"
password_input.send_keys(password + Keys.RETURN)
While this always works, it wastes time for no reason. And the selenium wait method really should work.
What am I doing wrong?
While How to resolve ElementNotInteractableException: Element is not visible in Selenium webdriver? is technically a duplicate, it solves this for Java, and it always annoys me when a duplicate is for a different language, so I'll write this answer for Python.
The ec.presence_of_element_located(...) method just tests for the presence of an element within the document object model. It does not ensure that element is something the user can interact with. Another element might overlap it, or the element might be hidden from view for a brief moment before you call password_input.send_keys(...).
Waiting until the element is "clickable" is usually the best solution:
driver = webdriver.Firefox()
driver.get(f"http://localhost:8888")
wait = WebDriverWait(driver, 10)
# waiting until element is "clickable" means user can interact with
# it, and thus send_keys(...) can simulate keyboard interaction.
password_input = wait.until(ec.element_to_be_clickable((By.ID, "password_input")))
password = "my_password"
password_input.send_keys(password + Keys.RETURN)
I have done a little research and could not find any answer specific to Selenium WebDriver with Python.
I can successfully sign into the page but I cannot find way(s) to verify that the login was successful. Page title does not work for me since it does not change.
Python Selenium documentation does not have any good explanation or examples.
All I want to do after this code is to put a line and assert that the username "Tuto" is visible on the page
LoginButtonLocator = "//a[contains(text(), 'Login')]"
facebookConnectButtonLocator = "//a[contains(text(), 'Connect with Facebook')]"
facebookLoginLocatorID = "email"
facebookPasswordLocatorID = "pass"
facebookLoginButtonLocatorID = "loginbutton"
LoginButtonElement = WebDriverWait(driver, 20).until(lambda driver: driver.find_element_by_xpath(LoginButtonLocator))
LoginButtonElement.click()
facebookConnectButtonElement = WebDriverWait(driver, 20).until(lambda driver: driver.find_element_by_xpath(facebookConnectButtonLocator))
facebookConnectButtonElement.click()
facebookLoginElement = WebDriverWait(driver, 20).until(lambda driver: driver.find_element_by_id(facebookLoginLocatorID))
facebookLoginElement.send_keys(facebookID)
facebookPasswordElement = WebDriverWait(driver, 20).until(lambda driver: driver.find_element_by_id(facebookPasswordLocatorID))
facebookPasswordElement.send_keys(facebookPW)
facebookLoginButtonElement = WebDriverWait(driver, 20).until(lambda driver: driver.find_element_by_id(facebookLoginButtonLocatorID))
facebookLoginButtonElement.click()
I am working with the Javascript API instead of the Python version, so the syntax is different, but here is how I would get about it (using mocha as a test framework):
facebookLoginButtonElement.click().then(function(_) {
driver.findElement(By.xpath('//a[text() = "Tuto"]')).then(function(userLink) {
assert.ok(userLink);
});
});
In the javascript version, if <a ...>Tuto</a> can't be found there would be an error before the callback is called, so the assertion would be redundant (would only get there if the link was found), but I find it as self-documenting so I add the assert.
Just try to find an element using xPath:
wait = WebDriverWait(browser, 10)
find_username = wait.until(EC.presence_of_element_located((By.XPATH,'//span[contains(text(), "Alex")]')))
assert find_username
Hope, it will help you.
I'm writing it considering that you are using webdriver_manager for Chrome, though it doesn't create many effects, You can use the code in both scenarios with webdriver_manager or without as well provided that you have made significant changes in adding executable path for webdriver if not using webdriver_manager. Here you go...
from selenium import webdriver
// this was for importing webdriver
from webdriver_manager.chrome import ChromeDrivermanager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get("https://somedummy.domain.com/login")
driver.find_element_by_name("username").send_keys("somedummyvalidated#email.com")
driver.find_element_by_name("password").send_keys("someDummyPass")
driver.find_element_by_id("btnLogin").click()
this block was about sending keys, now if you are logged in correctly then the page title must change. Just use the below code to fetch the title and use assert or match the titles before and after login.
driver.title