I have been working on a web scraping script that gets past the login page but I can't find a way to get this radio button selected as the script attempts to login.
Here's the button:
<input type="radio" name="UserType" id="UserType" value="PARENTSWEB-PARENT" tabindex="4">
When it is clicked it looks like this:
<input type="radio" name="UserType" id="UserType" value="PARENTSWEB-PARENT" tabindex="4" checked="checked">
I am using a dictionary to submit the username and password but not sure how to add the button functionality into it.
Dict:
payload={
'username':USERNAME,
'password':PASSWORD,
'DistrictCode':DistCode,
'PARENTSWEB-STUDENT':'checked'
}
I am using the requests library and lxml to submit and look at the data. But if there is a better library or another one I can also use I'm open to anything.
Related
I want to login to the site below using requests module in python.
https://accounts.dmm.com/service/login/password
But I cannot find the "login_id" and "password" fields in the requests' response.
I CAN find them using "Inspect" menu in Chrome.
<input type="text" name="login_id" id="login_id" placeholder="メールアドレス" value="">
and
<input type="password" name="password" id="password" placeholder="パスワード" value="">
I tried to find them in the response from requests, but couldn't.
Here is my code:
import requests
url = 'https://accounts.dmm.com/service/login/password'
session = requests.session()
response = session.get(url)
with open('test_saved_login.html','w',encoding="utf-8")as file:
file.write(response.text) # Neither "login_id" nor "password" field found in the file.
How should I do?
Selenium is an easy solution, but I do not want to use it.
The login form is created with javascript. Try viewing the page in a browser with javascript disabled there will be no form. The people who control that site are trying to prevent people from doing exactly what you're trying to do. In addition to the fact the form elements don't appear (which really doesn't matter with requests,) they are also using a special token that you won't be able to guess which I expect is also in obfuscated javascript. So it is likely impracticable to script a login with requests and unless you have special permission from this company it is highly inadvisable that you continue with doing what you're trying to do.
I'm really new to python programming. I'm working on automation of a web-browser. I started with selenium, but found it to be really slow for what I need.
I'm working on a code that can Login to a webpage and fill out few text-boxes and click on few buttons. I finally achieved the 1st part. My program can finally sign in using the robobrowser.
import re
from robobrowser import RoboBrowser
browser = RoboBrowser()
login_url = 'https://webbroker.td.com/waw/idp/login.htm?execution=e1s1'
browser.open(login_url)
form = browser.get_form(id="login")
form["login:AccessCard"].value = "****"
form["login:Webpassword"].value = "****"
browser.submit_form(form)
As soon as I log, this webpage is asking me to answer an authentication question.
<div class="td-layout-row td-margin-top-medium">
<div class="td-layout-column td-layout-grid15"><label class="questionText" for="MFAChallengeForm:answer" id="MFAChallengeForm:question">
What is your favourite TV show?</label></div>
</div>
<div class="td-layout-row">
<div class="td-layout-column td-layout-grid7"><input autocomplete="off" id="MFAChallengeForm:answer" maxlength="25" name="MFAChallengeForm:answer" onkeydown="trapEnter(event,'MFAChallengeForm',id,'next')" size="25" type="password" value=""/></div>
</div>
How do I carry on form here? I need to enter and submit my authentication answer in order to proceed. In selenium it would be something like this.
AQ = driver.find_element_by_id("MFAChallengeForm:answer")
AQ.send_keys("******")
*Click Submit*
However, how would I do it in robobrowser/lxml/beautifulsoup? I need to submit my answer(while sill being logged in). Thank you in advance.
I want to go to a site and click on a button or link for logging in. But login does not use form.
I think login procedure use javascript.
Input for username:
<input tabindex="1"
class="dxeEditArea_Office2003Blue dxeEditAreaSys" onkeydown="aspxEKeyDown('ctl00_wucLogin1_txtUID', event)"
name="ctl00$wucLogin1$txtUID"
onkeyup="aspxEKeyUp('ctl00_wucLogin1_txtUID', event)"
type="text"
id="ctl00_wucLogin1_txtUID_I"
onblur="aspxELostFocus('ctl00_wucLogin1_txtUID')"
onfocus="aspxEGotFocus('ctl00_wucLogin1_txtUID')"
onkeypress="aspxEKeyPress('ctl00_wucLogin1_txtUID', event)"
style="height:15px;">
The link for login is :
<a id="ctl00_wucLogin1_BtnLogin"
class="Search_button"
href="javascript:__doPostBack('ctl00$wucLogin1$BtnLogin','')"
style="...">Login
</a>
How can I click on this link and how can I fill than input for username by twill?
is there any other alternative for twill?
Thanks,
Try sending the User-Agent header with the request. In any case, twill doesn't handle JavaScript well, so you're better off trying something else.
For alternatives, there are:
Mechanize
Requests
BeautifulSoup (for parsing HTML)
Selenium (python bindings)
I want to click a button with python, the info for the form is automatically filled by the webpage. the HTML code for sending a request to the button is:
INPUT type="submit" value="Place a Bid">
How would I go about doing this?
Is it possible to click the button with just urllib or urllib2? Or will I need to use something like mechanize or twill?
Use the form target and send any input as post data like this:
<form target="http://mysite.com/blah.php" method="GET">
......
......
......
<input type="text" name="in1" value="abc">
<INPUT type="submit" value="Place a Bid">
</form>
Python:
# parse the page HTML with the form to get the form target and any input names and values... (except for a submit and reset button)
# You can use XML.dom.minidom or htmlparser
# form_target gets parsed into "http://mysite.com/blah.php"
# input1_name gets parsed into "in1"
# input1_value gets parsed into "abc"
form_url = form_target + "?" + input1_name + "=" + input1_value
# form_url value is "http://mysite.com/blah.php?in1=abc"
# Then open the new URL which is the same as clicking the submit button
s = urllib2.urlopen(form_url)
You can parse the HTML with HTMLParser
And don't forget to urlencode any post data with:
urllib.urlencode(query)
You may want to take a look at IronWatin - https://github.com/rtyler/IronWatin to fill the form and "click" the button using code.
Using urllib.urlopen, you could send the values of the form as the data parameter to the page specified in the form tag. But this won't automate your browser for you, so you'd have to get the form values some other way first.
I have to test an internal web page.
The web page contains text fields, buttons, and radio buttons.
Based on a specific radio button selected another sub-form is displayed in the web page.
I'm using urllib2, and some of its modules, to successfully connect to the web server and perform some actions.
However, not able to select the radio button, via a POST from the python script, I'm not able to proceed with the test automation.
Reading some of the online posts about selecting radio buttons I read that some people are using "mechanize". I'm not familiar with this. Is there another specific module in urllib2 that would allow me to send a POST request to select a specific radio button.
Roland
They are referring to this python module: http://wwwsearch.sourceforge.net/mechanize/. You can emulate the selected radio button in the form by posting the form properly with urllib2. Discussed in this post: urllib2: submitting a form and then redirecting.
Imagine you have a form with a radio input like this:
<input type="radio" value="1" name="something" />
The post body would be: something=1.
If we have for example this HTML code:
<input type="radio" name="register" value="0" checked="checked"> "Yes it's my password"
then you must put in your payload:
payload["register"]="Yes it's my password"