We Keep Coding

Submitting data to a web portal

I'm currently trying to access a website with python and I'm having some trouble using the requests and mechanize modules. Basically the way I do this task manually is that I load the website portal and log on then click a button and fill out a form and submit this to receive a download. I've gotten to the log on stage and am having trouble submitting my username and log in am currently using this method
import requests
payload = {"username":"user","password":"pass"}
r = requests.post("portal login address",data=payload)
response = r.content
print(response)
but this gives me the exact same output as a get request where I don't include the payload. I am also wondering how I can simulate these button clicks and form submissions, I know mechanize can be used but I'm unclear as to how

You can use the mechanize module, in this way:
import re
import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)
br.open("<page>")
# you can access the form by name or some other means
# ive used a loop here just as an example
for form in br.forms():
form["username"] = "saurabhav"
form["password"] = "8558881858"
form.submit()
Have a look at Mechanize

How to use Mechanize to fill HTML forms in Python

I'm new to mechanize, and i don't quite understand how does it work, I tried a lot of tutorials, but most of them were outdated and didn't work.
First question is, What effect does Mechanize make? does it fill forms in specific browser so end-users can see it, Or does it make everything in Mechanize browser that cannot be seen by end-user?
I'm trying to make Mechanize fill out the form, Form changes input name after reloading page, How can i change its value by number?
import mechanize
br = mechanize.Browser()
br.set_handle_robots(False)
br.addheaders = [("User-agent","Mozilla/5.0")]
gitbot = br.open("https://arkhamnetwork.org/community/register")
br.select_form(nr=0)
br["user[username]"] = "username"
br["user[email]"] = "email"
br["user[password]"] = "password"
sign_up = br.submit()
Error i am getting after executing code:
NameError: name 'username' is not defined
I want to fill out all the forms on the page, without using input name, How can i do it?

I have found solution:
Forms are actually containing controls, Thats why i needed to select form.
Code that fills out forms on this specific website:
import mechanize
br = mechanize.Browser()
response = br.open("https://arkhamnetwork.org/community/register")
br.addheaders = [("User-agent","Mozilla/5.0")]
gitbot = br.open("https://arkhamnetwork.org/community/register")
br.select_form(nr=1)
br.set_all_readonly(False)
br.form.set_value("test", nr=0)
br.method = "POST"
response = br.submit()
print response.geturl('http://arkhamnetwork.org/community/register/register')

Using mechanize with Python properly to fill out HTML forms

I'm trying to fill out "name" form from specific website, I am new to mechanize and i'm not really sure how to use it, I have tried a lot of solutions... But this is what i've gone by far:
import mechanize
import cookielib
import re
NAME = "whatever"
def login():
Browser = mechanize.Browser()
cj = cookielib.LWPCookieJar()
Browser.set_cookiejar(cj)
Browser.set_handle_robots(False)
Browser.set_handle_gzip(True)
Browser.set_handle_redirect(True)
Browser.set_handle_referer(True)
Browser.set_handle_refresh(mechanize._http.HTTPRefreshProcessor(), max_time=1)
Browser.open('http://arkhamnetwork.org/community/login/login')
print [form for form in Browser.forms()][0] # Tried to see all forms
Browser.select_form(nr=0)
Browser.form["cdf254f828f75d57e0320558987a137d"] = NAME
Browser.submit()
return browser
login()
and i'm constantly getting this error:
mechanize._form.ControlNotFoundError: no control matching name 'cdf254f828f75d57e0320558987a137d'
This is what i get from: print [form for form in Browser.forms()][0]:
<CheckboxControl(visible=[*1])>
<HiddenControl(_xfToken=) (readonly)>>
However in page source, Name of the first form was: cdf254f828f75d57e0320558987a137d
Question:
How can i fix this? Is there any other proper way to fill out forms in Mechanize?

python mechanize new page

I am using python and mechanize to login into a site. I can get it to login, but once I am in I need to have mechanize select a new form and then submit again. I will need to do this 3 or for times to get to the page I need. Once I am logged in how od I slect the form on the 2nd apge?
import mechanize
import urlparse
br = mechanize.Browser()
br.open("https://test.com")
print(br.title())
br.select_form(name="Login")
br['login_name'] = "test"
br['pwd'] = "test"
br.submit()
new_br = mechanize.Browser()
new_br.open("test2.com")
new_br.select_form(name="frm_page2") # where the error happens
I get the following error.
FormNotFoundError: no form matching name 'frm_page2'
Thanks for the help.

You can't use name=' ' when finding a form because Mechanize already uses 'name' itself.
If you want to find something by name, you need to do:
br.select_form(attrs={'name': 'Login'})

Python's Mechanize not recognizing a form that the HTML and BeautifulSoup does

I'm submitting a form, which then has a confirmation page. At the confirmation page in a browser there is an that is an image the user clicks to confirm the order.
Mechanize is not recognizing the form at all when it is present in the HTML mech has:
content = mech.submit().read()
soup = BeautifulSoup(content)
print soup.findAll('form')
displays the correct form, while mech claims there are no forms present. I have tried doing:
mech.click(inputName)
and mech claims the input does not exist. Meanwhile the input shows up just fine with:
print soup.findAll('input')
Any ideas? I have also done this:
mech = mechanize.Browser(factory=mechanize.RobustFactory())
With no luck.

Try parsing all html responses with BeautifulSoup, then mechanize should recognise the form.
You can see how to do it in this answer Is it possible to hook up a more robust HTML parser to Python mechanize?