I'm really stuck here.
I use Python + Selenium to automate a website form filling.
So I gave some data into the webpage, then "click" on a button, after that a new value appears in an element and I would like to get that value, but I stuck.
How should I get that value into a variable?
I tried to use find_element_by_xpath what works for "click", works for "send.keys", but to get any value from here, nothing.
Please help me!
Picture enclosed about the webpage inspection.
1
'''python
from selenium import webdriver
browser = webdriver.Chrome('chromedriver.exe')
browser.get('https://...')
browser.find_element_by_xpath('//input[parameter-
name="moduleId"]').send_keys('1234')
'''
So by that point everything is ok. I can fill the "moduleId" element with the 1234 value.
but from here I cannot read it back.
so If I try like this:
'''python
moduleId = browser.find_element_by_xpath('//input[#parameter-name="moduleId"]')
'''
the output is nothing.
Here is the HTML part of the website what is interesting.
html
<input type="text" name="parameterValue" class="form-control"
placeholder="Value" spellcheck="false" autocomplete="off" data-bind="value:
value, valueUpdate: 'keyup', autocomplete: { options: options, filtered:
true }, attr: { 'parameter-name': name, type: inputType }" parameter-name="moduleId">
Use this to get the value of the input element:
input.get_attribute('value')
You could also do this all in one shot, example
browser.find_element_by_xpath('//input[#parameter-name="moduleId"]').get_attribute('value')
Thx4 #scilence
I made this modification:
moduleId = browser.find_element_by_xpath('//input[#parameter-name="moduleId"]')
moduleId_value = moduleId.get_attribute('value')
print("data: ",moduleId_value)
and the output is what I need!
Related
This question already has an answer here:
Selenium "selenium.common.exceptions.NoSuchElementException" when using Chrome
(1 answer)
Closed 2 years ago.
I'm trying to scrape the promotion information of each product from a website by clicking on the product and go to its detailed page. When the spider clicks on the product, the web will ask it to log in, and I tried the following code:
def __init__(self):
self.driver = webdriver.Chrome(executable_path = '/usr/bin/chromedriver')
...
def start_scraping(self, response):
self.driver.get(response.url)
self.driver.find_element_by_id('fm-login-id').send_keys('iamgooglepenn')
self.driver.find_element_by_id('fm-login-password').send_keys('HelloWorld1_')
self.driver.find_element_by_class_name('fm-button fm-submit password-login').click()
...
However, there is NoSuchElementException when I run it.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":"[id="fm-login-id"]"}
'spider_exceptions/NoSuchElementException': 14,
The HTML of the login page is as follows:
<div class='input-plain-wrap input-wrap-loginid'>
<input id='fm-login-id' class='fm-text' name='fm-login-id'...>
event
</div>
So, I'm pretty sure the id should be 'fm-login-id'. The reason I could think of that might cause this issue is that this login page is a popup.
Basically, it pops up in the middle of the main page. Looking at the HTML of the site, I can see that the login type seems to be a new HTML window
<!DOCTYPE html>
<html>event
....
<\html>
I'm not sure if this is the issue, and if so, how to fix it? Also, is there other reasons that might've caused the issue?
The popup will have an ID. You might have to add f'#{popup_id}' to the end of response.url. Like this URL: https://stackoverflow.com/questions/62906380/nosuchelementexception-when-using-selenium-python/62906409#62906409. It contains #62906409 because 62906409 is the ID of an element in the page.
The login page inside a frame, you need switch it first:
#switch it first
self.driver.switch_to.frame(driver.find_element_by_id('J_loginIframe'))
self.driver.find_element_by_id('fm-login-id').send_keys('iamgooglepenn')
self.driver.find_element_by_id('fm-login-password').send_keys('HelloWorld1_')
And for login button you can't use .find_element_by_class_name, this method just for single class name. This element having multiple class name, so use .find_element_by_css_selector like bellow:
#submit button
self.driver.find_element_by_css_selector('.fm-button.fm-submit.password-login').click()
The login content seems to be nested in an iFrame element (if you trace it all the way to the top, you should find an iFrame with id="sufei-dialog-content"), which means you need to switch to that iFrame for that nested html before selecting your desired element, otherwise it will not work.
First you will need to use driver.switch_to.frame("sufei-dialog-content"), and then select your element with driver.find_element_by_name() or whatever you had.
A similar issue can be found here: Selenium and iframe in html
Just a simple mistake:
<div class='input-plain-wrap input-wrap-loginid'>
<input id='fm-login-id class='fm-text' name='fm-login-id'...>
event
</div>
is actually supposed to be:
<div class='input-plain-wrap input-wrap-loginid'>
<input id='fm-login-id' class='fm-text' name='fm-login-id'...>
event
</div>
You forgot a single-quote.
Have you tried driver.find_element_by_name('fm-login-id')?
You should try finding the elements by their XPaths. You just have to inspect the element, right-click on it and copy its XPath. The XPath of the first <input ... is //*[#id="fm-login-id"].
I am trying to collect email addresses from a form on a website that has readonly inside of it.
<input name="email" id="email" type="text" class="form-control" value="example#gmail.com" readonly="">
I want to be able to get the email address (example#gmail.com) but everything I try returns "unable to locate element".
Everything is configured properly as the rest of the script is working fine, which I have left out.
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
from bs4 import BeautifulSoup
import re
import pandas as pd
import os
x = 0
all_volunteers = driver.find_elements_by_xpath('//*[#title="View volunteer record"]')
for volunteer in all_volunteers:
volunteer.click()
driver.implicitly_wait(3)
# email_add = driver.find_element_by_id('emaillabel')
#email_add = driver.switch_to_frame(driver.find_element_by_name('email'))
#print(email_add.get_attribute('email'))
#email_add = driver.find_element_by_css_selector('input id')
#email_add = driver.find_element_by_xpath('//input [#name="email"]')
#email_add = driver.find_element_by_tag_name('Email Address')
email_add = driver.find_element_by_xpath('//*[#id="email"]')
print(email_add.get_attribute('value'))
# back button
driver.execute_script("window.history.go(-1)")
#increase counter by 1
x += 1
Everything commented out (followed by #) is what I have tried.
Is any one able to tell me what I am doing wrong or missing?
I have a debugging solution to locate the element.
In the browser, open the web page containing the email input
Open developer tools (F12)
Open console tab in the developer tools
Type $x('//input[#id="email"]') and see if the element is located. This is the native xpath locator
You can also try document.getElementById('email') in the console
If the element is not found still, try the iFrame selector marked in the screenshot to identify iframes and switch to it.
If more than one element is returned, it means that you might have to modify the selector to find unique element.
I am writing a python script to input text, that then has a dropdown appear to select from a list of items. These items are all hidden values and are not inputted to the webpage until selected by clicking on said item.
Specifically I am attempting to input data on Fitbits website to track food (a thing for work, but it is tedious to input food consumption each day). I would like to automate this :)
My script currently looks like this...
#! python3
# fitbitSubmitFood.py this script submits the food log on fitbit.com
from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
browser = webdriver.Chrome()
# open webpage to log food
browser.get('https://www.fitbit.com/foods/log')
# login to the website
# email username input
emailSelect = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[1]/input')
emailSelect.send_keys('EMAIL')
# email password input
inputPassword = browser.find_element_by_xpath('//*[#id="loginForm"]/fieldset/dl/dd[2]/input')
inputPassword.send_keys('PASSWORD')
# click log in
clickLogin = browser.find_element_by_xpath('//*[#id="loginForm"]/div[1]/button')
clickLogin.click() # clicky click!
# input What did you eat?
foodSelect = browser.find_element_by_xpath('//*[#id="foodselectinput"]')
foodSelect.send_keys('Pizza, Bread') # select by visible text
# input How Much?
howMuch = browser.find_element_by_xpath('//*[#id="quantityselectinput"]')
howMuch.send_keys('3')
# click Log Food
logfood = browser.find_element_by_xpath('//*[#id="foodAutoCompButton"]')
# logfood.click() # clicky click!
# Close web browser
The current script above throws the following error.
selenium.common.exceptions.InvalidElementStateException: Message: invalid element state
So I have also tried suggestions from this stackoverflow question.
Entering a value into a type="hidden" field using Selenium + Python
WebDriverWait(foodSelect, 10).until(EC.visibility_of_element_located((By.XPATH,'//*[#id="foodId"]'))) # wait for hidden text to populate
This threw the following error...
selenium.common.exceptions.TimeoutException: Message:
Here is a snippet from the website
input type="text" name="foodselectinput" id="foodselectinput" class="text columnFull yui-ac-input" maxlength="80" tabindex="1" autocomplete="off"
I can see the following on the website for the hidden value. This value is not passed to the webpage until after the food is selected from the drop down menu.
input name="foodId" id="foodId" type="hidden" value="13272"
Note: I have already tried using the click method for the Pizza, Bread
clickFood = browser.find_element_by_xpath('//*[#id="foodselectcontainer"]/div/div[2]/ul/li[1]/div/div[1]')
clickFood.click() # click it!
This did not work.
selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"xpath","selector":"//*[#id="foodselectcontainer"]/div/div[2]/ul/li[1]/div/div[1]"}
The script cannot continue until the food item is selected on the webpage. Hence, the question.
How would one pass the hidden value or select from a dropdown?
From your snippet
input type="text" name="foodselectinput" id="foodselectinput" class="text columnFull yui-ac-input" maxlength="80" tabindex="1" autocomplete="off"
yui-ac-input refers AutoComplete from YUI 2 Library (That's what I thought)
As the site mentioned by you asked credentials. I created a sample code for YUI Example
Sorry I dont know python.Below is a Java code. The syntax looks more or less same. So get the idea from below code and implement it
public void start() {
driver.get("http://yui.github.io/yui2/docs/yui_2.9.0_full/examples/autocomplete/ac_basic_array.html");
yuiAutoSuggestSelect("Cali");
}
public void yuiAutoSuggestSelect(String value) {
WebElement inputElement = driver.findElement(By.className("yui-ac-input"));
inputElement.sendKeys(value);
By autoSuggSel = By.xpath("//div[#class='yui-ac-container']//div[#class='yui-ac-bd']//li");
WebElement autoSuggestEl = driver.findElement(autoSuggSel);
WebDriverWait wait = new WebDriverWait(driver, 10);
wait.until(ExpectedConditions.visibilityOf(autoSuggestEl)).click();
}
I know this question might seem quite straight forward, but I have tried every suggestion and none has worked.
I want to build a Python script that checks my school website to see if new grades have been put up. However I cannot for the life of me figure out how to scrape it.
The website redirects to a different page to login. I have tried all the scripts and answers I could find but I am lost.
I use Python 3, the website is in a https://blah.schooldomate.state.edu.country/website/grades/summary.aspx
format
The username section contains the following:
<input class="txt" id="username" name="username" type="text" autocomplete="off" style="cursor: auto;">
The password is the name except it contains an onfocus HTML element.
One successfully authenticated, I am automatically redirected to the correct page.
I have tried:
using Python 2's cookielib and Mechanize
Using HTTPBasicAuth
Passing the information as a dict to a requests.get()
Trying out many different peoples code including answers I found on this site
You can try with requests:
http://docs.python-requests.org/en/master/
from the web site:
import requests
r = requests.get('https://api.github.com/user', auth=('user', 'pass'))
Maybe you can use Selenium library.
I let you my code example:
from selenium import webdriver
def loging():
browser = webdriver.Firefox()
browser.get("www.your_url.com")
#Edit the XPATH of Loging INPUT username
xpath_username = "//input[#class='username']"
#Edit the XPATH of Loging INPUT password
xpath_password = "//input[#class='password']"
#THIS will write the YOUR_USERNAME/pass in the xpath (Custom function)
click_xpath(browser, xpath_username, "YOUR_USERNAME")
click_xpath(browser, xpath_username, "YOUR_PASSWORD")
#THEN SCRAPE WHAT YOU NEED
#Here is the custom function
#If NO input, will only click on the element (on a button for example)
def click_xpath(self, browser, xpath, input="", time_wait=10):
try:
browser.implicitly_wait(time_wait)
wait = WebDriverWait(browser, time_wait)
search = wait.until(EC.element_to_be_clickable((By.XPATH, xpath)))
search.click()
sleep(1)
#Write in the element
if input:
search.send_keys(str(input) + Keys.RETURN)
return search
except Exception as e:
#print("ERROR-click_xpath: "+xpath)
return False
I'm trying to connect to a school url and automate the process with selenium. Originally I tried using splinter, but ran into similar problems. I can't seem to be able to interact with the username and password fields. I realized a little ways in that it is an iframe that I need to interact with. Currently I have:
import time
from selenium import webdriver
from selenium.webdriver.common.keys import Keys
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/")
driver.switch_to.frame('Content') #I tried contentFrame and content as well
loginid = driver.find_elements_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_elements_by_id('password')
passwd.send_keys("***")
sub = driver.find_elements_by_id('login')
sub.click()
time.sleep(5)
driver.close()
Here is the HTML that I am trying to interact with:
The Website: https://my.oregonstate.edu/webapps/portal/frameset.jsp
The iframe:
<iframe id="contentFrame" style="height: 593px;" name="content" title="Content" src="/webapps/portal/execute/tabs/tabAction?tab_tab_group_id=_1_1" frameborder="0"></iframe>
The forms:
Username:
<input name="user_id" id="user_id" size="25" maxlength="50" type="text">
Password:
<input size="25" name="password" id="password" autocomplete="off" type="password">
It seems that selenium can locate the elements just find, but I am unable to input any information into these fields, I got the error 'List object has no attribute'. When I realized it was the iframe I tried to navigate into that but it says 'Unable to locate frame: Content'. Is there another iframe that I am missing? Or something obvious? This is my first time here so sorry if I messed something up with the code linking.
Thanks for the help.
driver.switch_to.frame() takes frame's id or name, where your frame have id = contentFrame and name = content. (The reason they didn't work is probably because of a different issue, read through please)
First, please try use either one of them, not Content (which has upper case C).
Once you have fixed the issue above, there will be another error in your code.
loginid = driver.find_elements_by_id('user_id')
loginid.send_keys("***")
driver.find_elements_by_id finds all matching elements, which is a list. So you can't use send_keys. Please use driver.find_element_by_id('user_id').
Here is the code I tested working.
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/")
driver.switch_to.frame('content') # all lower case to match your actual frame name
loginid = driver.find_element_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_element_by_id('password')
passwd.send_keys("***")
Regarding issue in your following comments
from selenium import webdriver
driver = webdriver.Firefox()
driver.get("https://my.oregonstate.edu/webapps/login/?action=relogin")
loginid = driver.find_element_by_id('user_id')
loginid.send_keys("***")
passwd = driver.find_element_by_id('password')
passwd.send_keys("***")
driver.find_element_by_css_selector('.submit.button-1').click()