I'm trying to connect to a school url and automate the process with selenium. Originally I tried using splinter, but ran into similar problems. I can't seem to be able to interact with the username and password fields. I realized a little ways in that it is an iframe that I need to interact with. Currently I have:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/")
driver.switch_to.frame('Content') #I tried contentFrame and content as well
loginid = driver.find_elements_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_elements_by_id('password')
passwd.send_keys("***")
sub = driver.find_elements_by_id('login')
sub.click()
time.sleep(5)
driver.close()
Here is the HTML that I am trying to interact with:
The Website: https://my.oregonstate.edu/webapps/portal/frameset.jsp
The iframe:
<iframe id="contentFrame" style="height: 593px;" name="content" title="Content" src="/webapps/portal/execute/tabs/tabAction?tab_tab_group_id=_1_1" frameborder="0"></iframe>
The forms:
Username:
<input name="user_id" id="user_id" size="25" maxlength="50" type="text">
Password:
<input size="25" name="password" id="password" autocomplete="off" type="password">
It seems that selenium can locate the elements just find, but I am unable to input any information into these fields, I got the error 'List object has no attribute'. When I realized it was the iframe I tried to navigate into that but it says 'Unable to locate frame: Content'. Is there another iframe that I am missing? Or something obvious? This is my first time here so sorry if I messed something up with the code linking.
Thanks for the help.
driver.switch_to.frame() takes frame's id or name, where your frame have id = contentFrame and name = content. (The reason they didn't work is probably because of a different issue, read through please)
First, please try use either one of them, not Content (which has upper case C).
Once you have fixed the issue above, there will be another error in your code.
loginid = driver.find_elements_by_id('user_id')
loginid.send_keys("***")
driver.find_elements_by_id finds all matching elements, which is a list. So you can't use send_keys. Please use driver.find_element_by_id('user_id').
Here is the code I tested working.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/")
driver.switch_to.frame('content') # all lower case to match your actual frame name
loginid = driver.find_element_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_element_by_id('password')
passwd.send_keys("***")
Regarding issue in your following comments
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/?action=relogin")
loginid = driver.find_element_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_element_by_id('password')
passwd.send_keys("***")
driver.find_element_by_css_selector('.submit.button-1').click()
Related
I am trying to collect email addresses from a form on a website that has readonly inside of it.
<input name="email" id="email" type="text" class="form-control" value="example#gmail.com" readonly="">
I want to be able to get the email address (example#gmail.com) but everything I try returns "unable to locate element".
Everything is configured properly as the rest of the script is working fine, which I have left out.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import re
import pandas as pd
import os
x = 0
all_volunteers = driver.find_elements_by_xpath('//*[#title="View volunteer record"]')
for volunteer in all_volunteers:
volunteer.click()
driver.implicitly_wait(3)
# email_add = driver.find_element_by_id('emaillabel')
#email_add = driver.switch_to_frame(driver.find_element_by_name('email'))
#print(email_add.get_attribute('email'))
#email_add = driver.find_element_by_css_selector('input id')
#email_add = driver.find_element_by_xpath('//input [#name="email"]')
#email_add = driver.find_element_by_tag_name('Email Address')
email_add = driver.find_element_by_xpath('//*[#id="email"]')
print(email_add.get_attribute('value'))
# back button
driver.execute_script("window.history.go(-1)")
#increase counter by 1
x += 1
Everything commented out (followed by #) is what I have tried.
Is any one able to tell me what I am doing wrong or missing?
I have a debugging solution to locate the element.
In the browser, open the web page containing the email input
Open developer tools (F12)
Open console tab in the developer tools
Type $x('//input[#id="email"]') and see if the element is located. This is the native xpath locator
You can also try document.getElementById('email') in the console
If the element is not found still, try the iFrame selector marked in the screenshot to identify iframes and switch to it.
If more than one element is returned, it means that you might have to modify the selector to find unique element.
I'm trying to figure out a way to automatically log in / enter text into a given text field on a particular web page. I've already don't this before, but this particular page isn't responding to anything I've thrown at it yet.
The default page load already has the auto-focus on the necessary text box. I'm currently using Python to write the Selenium code. My current script includes prior processes that lead to the page in question, where my current problem lies. Additionally, I've been running this code in a Google-Chrome browser, but with the user-agent selected to Edge - Mobile (but that probably won't matter here).
The website in question is the Microsoft login at this link.
The CSS/HTML of the text box in question:
<input type="email" name="loginfmt" id="i0116" maxlength="113" lang="en" class="form-control ltr_override" aria-describedby="usernameError loginHeader loginDescription" aria-required="true" data-bind="textInput: usernameTextbox.value,
hasFocusEx: usernameTextbox.focused,
placeholder: $placeholderText,
ariaLabel: tenantBranding.UserIdLabel || str['CT_PWD_STR_Username_AriaLabel'],
css: { 'has-error': usernameTextbox.error },
attr: inputAttributes" placeholder="Email, phone, or Skype" aria-label="Enter your email, phone, or Skype.">
The code I'm currently testing (which is basically three varied iterations of the same idea), after the given page loads:
element = driver.find_element_by_id("i0116")
element.click()
element.clear()
element.send_keys("wbhyatt3#gmail.com")
element.send_keys(Keys.RETURN)
time.sleep(1)
element = driver.find_element_by_name("loginfmt")
element.click()
element.clear()
element.send_keys("wbhyatt3#gmail.com")
element.send_keys(Keys.RETURN)
time.sleep(1)
element = driver.find_element_by_css_selector("input.email")
element.click()
element.clear()
element.send_keys("wbhyatt3#gmail.com")
element.send_keys(Keys.RETURN)
Unfortunately, trying to select the textbox via the input id, class, or name don't seem to be working. It should be worth noting that the page CSS I'm referencing for the text box includes an element "input" prior - I'm not sure if this will affect my current code. I'm fairly certain that either the send_keys aren't working, or perhaps the selection of the element, itself.
What makes the situation even more frustrating is that the page's default focus is on the textbox - so I don't even truly need to select the element, I just need to be able to enter text and submit/enter.
I've also tried targeting it as an iframe, but that hasn't seemed to help either.
Any ideas? Any and all help would be deeply appreciated. I am simply trying to find a way to enter text into the login box.
To enter an EmailID into the field with placeholder text as Email, phone, or Skype you can use the following code block :
from selenium import webdriver
from selenium.webdriver.chrome.options import Options
options = Options()
options.add_argument("start-maximized")
options.add_argument("disable-infobars")
options.add_argument("--disable-extensions")
driver = webdriver.Chrome(chrome_options=options, executable_path=r'C:\Utility\BrowserDrivers\chromedriver.exe')
driver.get('https://login.live.com/login.srf')
print("Page Title is : %s" %driver.title)
element = driver.find_element_by_xpath("//input[#class='form-control ltr_override' and #name='loginfmt']")
element.click()
element.clear()
element.send_keys("wbhyatt3#gmail.com")
Console Output :
Page Title is : Sign in to your Microsoft account
Snapshot :
I'm trying to scrape a site that needs login information, and after hours of trying to figure out why I keep getting "Login failed", I believe it is simply because the "Log in" or "Submit" button is not actually getting clicked. I realized this by saving a screenshot of the browser right when it "fails". My username and password are filled into the fields.
I've tried things like wait, elementScrollBehavior, nothing seems to work. I'd really appreciate some help with this! Code below.
def load(self):
global browser
DesiredCapabilities.PHANTOMJS["elementScrollBehavior"] = 1
#browser field
browser = webdriver.PhantomJS()
wait = WebDriverWait(browser, 10)
#browser = webdriver.Firefox()
#browser = webdriver.Chrome()
loginId = self.id
password = self.pw
browser.get('https://link.example.com')
browser.find_element_by_id('cf-login').send_keys(loginId)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_name('submit').click()
#wait.until(EC.presence_of_element_located((By.ID, "crefli_HC_SSS_STUDENT_CENTER")))
try:
if browser.find_element_by_id('crefli_HC_SSS_STUDENT_CENTER'):
#return login status
return True
else:
return False
except:
print('element not found on page')
print(browser.current_url)
#browser.save_screenshot('~/Desktop/screen2.png')
HTML of form:
<form name="loginform" action="/oam/server/auth_cred_submit" method="post">
<div class="nonfloat-box">
Username:
<input type="text" id="cf-login" name="username" class="username inputbox" autocomplete="OFF">
</div>
<div class="float-box">
Password:
<input id="password" name="password" type="password" class="password inputbox" autocomplete="OFF">
</div>
<input type="image" src="https://www.cuny.edu/site/citizencuny/cunyfirst-login/loginbutton.jpg" onclick="javascript: return signon_validate()" alt="Submit" name="submit">
</form>
I believe I need to SOMEHOW get that bit of javascript to run. But HOW?
UPDATE: Selenium has a submit() method that automatically submits the <form> in HTML. Even using this, it does not work. As you can see in the HTML, it IS a form. At this point I do not know what else to try.
Please try this, hope it helps
from selenium.webdriver.common.keys import Keys
driver.find_element_by_name('submit').send_keys(Keys.RETURN)
(or)
driver.find_element_by_name('submit').send_keys(Keys.ENTER)
I have a feeling 'submit' is not being found by find_element_by_name. Try find_element_by_xpath(//*[#name='submit'])
HTML is needed to make a more accurate determination, but I usually use enter key to submit forms, sometimes javascript messes up the ability to submits by click and a simple enter usually does the trick
from selenium.webdriver.common.keys import Keys
def load(self):
(...)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_id('password').send_keys(Keys.ENTER)
(...)
Otherwise, make sure you mean find_element_by_name and not find_element_by_tag_name.
Use xpath, Please try this
//input[#name='submit']
or
//input[contains(#name,'submit')AND contains(#alt,'Submit')]
You can use submit() to submit the form. It needs to be sent to the <form> tag
browser.find_element_by_id('cf-login').send_keys(loginId)
browser.find_element_by_id('password').send_keys(password)
browser.find_element_by_name('loginform').submit()
If this doesn't work you can use JavaScript click as a work around
submit = browser.find_element_by_name('submit')
browser.execute_script("arguments[0].click();", submit)
You can also try sending Enter
from selenium.webdriver.common.keys import Keys
browser.find_element_by_id('cf-login').send_keys(loginId)
password_field = browser.find_element_by_id('password')
password_field.send_keys(password)
password_field.send_keys(Keys.RETURN)
#OR
password_field.send_keys(Keys.ENTER)
When you use selenium do automatic testing or scraping ，I suggest you use the method
webdriver.find_element_by_xpath(xpathString)
because you can check the xpathwebbroser's console
Try this command on console:
$x('xpathString')
I am writing a python script to input text, that then has a dropdown appear to select from a list of items. These items are all hidden values and are not inputted to the webpage until selected by clicking on said item.
Specifically I am attempting to input data on Fitbits website to track food (a thing for work, but it is tedious to input food consumption each day). I would like to automate this :)
My script currently looks like this...
#! python3
# fitbitSubmitFood.py this script submits the food log on fitbit.com
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
# open webpage to log food
browser.get('https://www.fitbit.com/foods/log')
# login to the website
# email username input
emailSelect = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[1]/input')
emailSelect.send_keys('EMAIL')
# email password input
inputPassword = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[2]/input')
inputPassword.send_keys('PASSWORD')
# click log in
clickLogin = browser.find_element_by_xpath('//*[#id="loginForm"]/div[1]/button')
clickLogin.click() # clicky click!
# input What did you eat?
foodSelect = browser.find_element_by_xpath('//*[#id="foodselectinput"]')
foodSelect.send_keys('Pizza, Bread') # select by visible text
# input How Much?
howMuch = browser.find_element_by_xpath('//*[#id="quantityselectinput"]')
howMuch.send_keys('3')
# click Log Food
logfood = browser.find_element_by_xpath('//*[#id="foodAutoCompButton"]')
# logfood.click() # clicky click!
# Close web browser
The current script above throws the following error.
selenium.common.exceptions.InvalidElementStateException: Message: invalid element state
So I have also tried suggestions from this stackoverflow question.
Entering a value into a type="hidden" field using Selenium + Python
WebDriverWait(foodSelect, 10).until(EC.visibility_of_element_located((By.XPATH,'//*[#id="foodId"]'))) # wait for hidden text to populate
This threw the following error...
selenium.common.exceptions.TimeoutException: Message:
Here is a snippet from the website
input type="text" name="foodselectinput" id="foodselectinput" class="text columnFull yui-ac-input" maxlength="80" tabindex="1" autocomplete="off"
I can see the following on the website for the hidden value. This value is not passed to the webpage until after the food is selected from the drop down menu.
input name="foodId" id="foodId" type="hidden" value="13272"
Note: I have already tried using the click method for the Pizza, Bread
clickFood = browser.find_element_by_xpath('//*[#id="foodselectcontainer"]/div/div[2]/ul/li[1]/div/div[1]')
clickFood.click() # click it!
This did not work.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="foodselectcontainer"]/div/div[2]/ul/li[1]/div/div[1]"}
The script cannot continue until the food item is selected on the webpage. Hence, the question.
How would one pass the hidden value or select from a dropdown?
From your snippet
input type="text" name="foodselectinput" id="foodselectinput" class="text columnFull yui-ac-input" maxlength="80" tabindex="1" autocomplete="off"
yui-ac-input refers AutoComplete from YUI 2 Library (That's what I thought)
As the site mentioned by you asked credentials. I created a sample code for YUI Example
Sorry I dont know python.Below is a Java code. The syntax looks more or less same. So get the idea from below code and implement it
public void start() {
driver.get("http://yui.github.io/yui2/docs/yui_2.9.0_full/examples/autocomplete/ac_basic_array.html");
yuiAutoSuggestSelect("Cali");
}
public void yuiAutoSuggestSelect(String value) {
WebElement inputElement = driver.findElement(By.className("yui-ac-input"));
inputElement.sendKeys(value);
By autoSuggSel = By.xpath("//div[#class='yui-ac-container']//div[#class='yui-ac-bd']//li");
WebElement autoSuggestEl = driver.findElement(autoSuggSel);
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.visibilityOf(autoSuggestEl)).click();
}
I know this question might seem quite straight forward, but I have tried every suggestion and none has worked.
I want to build a Python script that checks my school website to see if new grades have been put up. However I cannot for the life of me figure out how to scrape it.
The website redirects to a different page to login. I have tried all the scripts and answers I could find but I am lost.
I use Python 3, the website is in a https://blah.schooldomate.state.edu.country/website/grades/summary.aspx
format
The username section contains the following:
<input class="txt" id="username" name="username" type="text" autocomplete="off" style="cursor: auto;">
The password is the name except it contains an onfocus HTML element.
One successfully authenticated, I am automatically redirected to the correct page.
I have tried:
using Python 2's cookielib and Mechanize
Using HTTPBasicAuth
Passing the information as a dict to a requests.get()
Trying out many different peoples code including answers I found on this site
You can try with requests:
http://docs.python-requests.org/en/master/
from the web site:
import requests
r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
Maybe you can use Selenium library.
I let you my code example:
from selenium import webdriver
def loging():
browser = webdriver.Firefox()
browser.get("www.your_url.com")
#Edit the XPATH of Loging INPUT username
xpath_username = "//input[#class='username']"
#Edit the XPATH of Loging INPUT password
xpath_password = "//input[#class='password']"
#THIS will write the YOUR_USERNAME/pass in the xpath (Custom function)
click_xpath(browser, xpath_username, "YOUR_USERNAME")
click_xpath(browser, xpath_username, "YOUR_PASSWORD")
#THEN SCRAPE WHAT YOU NEED
#Here is the custom function
#If NO input, will only click on the element (on a button for example)
def click_xpath(self, browser, xpath, input="", time_wait=10):
try:
browser.implicitly_wait(time_wait)
wait = WebDriverWait(browser, time_wait)
search = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
search.click()
sleep(1)
#Write in the element
if input:
search.send_keys(str(input) + Keys.RETURN)
return search
except Exception as e:
#print("ERROR-click_xpath: "+xpath)
return False