We Keep Coding

How can I login to the site using requests module in python?

I want to login to the site below using requests module in python.
https://accounts.dmm.com/service/login/password
But I cannot find the "login_id" and "password" fields in the requests' response.
I CAN find them using "Inspect" menu in Chrome.
<input type="text" name="login_id" id="login_id" placeholder="メールアドレス" value="">
and
<input type="password" name="password" id="password" placeholder="パスワード" value="">
I tried to find them in the response from requests, but couldn't.
Here is my code:
import requests
url = 'https://accounts.dmm.com/service/login/password'
session = requests.session()
response = session.get(url)
with open('test_saved_login.html','w',encoding="utf-8")as file:
file.write(response.text) # Neither "login_id" nor "password" field found in the file.
How should I do?
Selenium is an easy solution, but I do not want to use it.

The login form is created with javascript. Try viewing the page in a browser with javascript disabled there will be no form. The people who control that site are trying to prevent people from doing exactly what you're trying to do. In addition to the fact the form elements don't appear (which really doesn't matter with requests,) they are also using a special token that you won't be able to guess which I expect is also in obfuscated javascript. So it is likely impracticable to script a login with requests and unless you have special permission from this company it is highly inadvisable that you continue with doing what you're trying to do.

Python, robobrowser, answer authentication-challenge after login

I'm really new to python programming. I'm working on automation of a web-browser. I started with selenium, but found it to be really slow for what I need.
I'm working on a code that can Login to a webpage and fill out few text-boxes and click on few buttons. I finally achieved the 1st part. My program can finally sign in using the robobrowser.
import re
from robobrowser import RoboBrowser
browser = RoboBrowser()
login_url = 'https://webbroker.td.com/waw/idp/login.htm?execution=e1s1'
browser.open(login_url)
form = browser.get_form(id="login")
form["login:AccessCard"].value = "****"
form["login:Webpassword"].value = "****"
browser.submit_form(form)
As soon as I log, this webpage is asking me to answer an authentication question.
<div class="td-layout-row td-margin-top-medium">
<div class="td-layout-column td-layout-grid15"><label class="questionText" for="MFAChallengeForm:answer" id="MFAChallengeForm:question">
What is your favourite TV show?</label></div>
</div>
<div class="td-layout-row">
<div class="td-layout-column td-layout-grid7"><input autocomplete="off" id="MFAChallengeForm:answer" maxlength="25" name="MFAChallengeForm:answer" onkeydown="trapEnter(event,'MFAChallengeForm',id,'next')" size="25" type="password" value=""/></div>
</div>
How do I carry on form here? I need to enter and submit my authentication answer in order to proceed. In selenium it would be something like this.
AQ = driver.find_element_by_id("MFAChallengeForm:answer")
AQ.send_keys("******")
*Click Submit*
However, how would I do it in robobrowser/lxml/beautifulsoup? I need to submit my answer(while sill being logged in). Thank you in advance.

Radio Button Click with Requests

I have been working on a web scraping script that gets past the login page but I can't find a way to get this radio button selected as the script attempts to login.
Here's the button:
<input type="radio" name="UserType" id="UserType" value="PARENTSWEB-PARENT" tabindex="4">
When it is clicked it looks like this:
<input type="radio" name="UserType" id="UserType" value="PARENTSWEB-PARENT" tabindex="4" checked="checked">
I am using a dictionary to submit the username and password but not sure how to add the button functionality into it.
Dict:
payload={
'username':USERNAME,
'password':PASSWORD,
'DistrictCode':DistCode,
'PARENTSWEB-STUDENT':'checked'
}
I am using the requests library and lxml to submit and look at the data. But if there is a better library or another one I can also use I'm open to anything.

login from scrapy when website has captcha

I need extract data from website with scrapy but before anything I should login.
login page has a captcha, so What should I do ?
website : tinyz.us
username field:
<input class="en" name="login_user" type="text">
username field:
<input class="en" name="login_password" type="password">

You can try first through Optical Character Recognition (OCR) and then with a
CAPTCHA solving API. See the chapter 7 of this book: https://www.packtpub.com/big-data-and-business-intelligence/web-scraping-python
There are also on-line services to solve captcha. For example:
https://anti-captcha.com/

Sending a web request with a python script to select a specific radio button on a web page

I have to test an internal web page.
The web page contains text fields, buttons, and radio buttons.
Based on a specific radio button selected another sub-form is displayed in the web page.
I'm using urllib2, and some of its modules, to successfully connect to the web server and perform some actions.
However, not able to select the radio button, via a POST from the python script, I'm not able to proceed with the test automation.
Reading some of the online posts about selecting radio buttons I read that some people are using "mechanize". I'm not familiar with this. Is there another specific module in urllib2 that would allow me to send a POST request to select a specific radio button.
Roland

They are referring to this python module: http://wwwsearch.sourceforge.net/mechanize/. You can emulate the selected radio button in the form by posting the form properly with urllib2. Discussed in this post: urllib2: submitting a form and then redirecting.
Imagine you have a form with a radio input like this:
<input type="radio" value="1" name="something" />
The post body would be: something=1.

If we have for example this HTML code:
<input type="radio" name="register" value="0" checked="checked"> "Yes it's my password"
then you must put in your payload:
payload["register"]="Yes it's my password"