I want to fill a HTML form with urllib2 and urllib.
import urllib
import urllib2
url = 'site.com/registration.php'
values = {'password' : 'password',
'username': 'username'
}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
But on the end of the form is a button(input type='submit'). If you don't click the button you can't send the data what you wrote in the input(type text)
How can I click the button with urllib and urllib2?
IF you look at your forms action attribute you could figure out what uri your form is submitting to. You can then make a post request to that uri.
This can be done like :
import urllib
import urllib2
url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
https://docs.python.org/2/howto/urllib2.html
You could also use the requests library which makes it a lot easier. You can read about it here
Another thing you might need to factor in is the CSRF(Cross Site Request Forgery) token which is embedded in forms. You will have to some how acquire it and pass it in with your post.
This is really more of something you would do with Selenium or similar. selenium.webdriver.common.action_chains.ActionChains.click, perhaps.
Related
Im performing a simple post request with urllib2 on a HTTPS url, i have one parameter and a JSESSIONID from a logged-in user. However when i Post i get 'your browser does not support iframes' error, status HTTP:200
import cookielib
import urllib
import urllib2
url = 'https://.../template/run.do?id=4'
http_header = {
"JSESSIONID": "A4604B1CFA8D2B5A8296AAB3B5EADC0C;",
}
params = {
'id' : 4
}
# setup cookie handler
cookie_jar = cookielib.LWPCookieJar()
cookie = urllib2.HTTPCookieProcessor(cookie_jar)
opener = urllib2.build_opener(cookie)
req = urllib2.Request(url, urllib.urlencode(params), http_header)
res = urllib2.urlopen(req)
print res.read()
I keep trigerring this method using CURL with no problem , but somehow can't via urllib, i DID try using all Request Headers that are used by browser but to no avail.
I fear this might be a stupid misconception, but I'm already wondering for hours!
I have an upload.php page that sends some data to a Python CGI script through a form, I then process the data in the background and I want to redirect to another php page, response_page.php, which displays info based on the processed data. I cannot send the data back to PHP and make the redirect at the same time, though.
My code:
#!/usr/bin/env python
import cgi
import cgitb
cgitb.enable()
try:
form = cgi.FieldStorage()
fn = form.getvalue('picture_name')
cat_id = form.getvalue('selected')
except KeyError:
print "Content-type: text/html"
print
print "<html><head>"
print "</head><body>error</body></html>"
else:
...
# here I processed the form data and stored it in data_to_be_displayed
# data to be processed and displayed in the response page
data_to_be_displayed = [1,2,3]
import httplib, json, urllib
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
conn = httplib.HTTPConnection('192.168.56.101:80')
#converting list to a json stream
data_to_be_displayed = json.dumps(data_to_be_displayed, ensure_ascii = 'False')
postData = urllib.urlencode({'matches':data_to_be_displayed})
conn.request("POST", "/response_page.php", postData, headers)
response = conn.getresponse()
if response.status == 200:
print "Location: /response_page.php"
print # to end the CGI response headers.
conn.close()
I found this: How to make python urllib2 follow redirect and keep post method , but I don't understand how I should use the urllib2.HTTPRedirectHandlerClass in my code.
Why don't you post to response_page.php using liburl2?
import urllib
import urllib2
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
data_to_be_displayed = json.dumps(data_to_be_displayed, ensure_ascii = 'False')
postData = urllib.urlencode({'matches':data_to_be_displayed})
req = urllib2.Request(url, data, headers)
response = urllib2.urlopen(req)
the_page = response.read()
For reference I've used the idea from pythons' documentation: https://docs.python.org/2/howto/urllib2.html#headers
you might also consider using Twisted apt for higher level code:
https://twistedmatrix.com/
EDIT:
After understanding better what are your asking for, I've found this post referring that redirect 307 is EXACTLY what you want (if now I understand right):
https://softwareengineering.stackexchange.com/questions/99894/why-doesnt-http-have-post-redirect
I want to use python urllib2 to simulate a login action, I use Fiddler to catch the packets and got that the login action is just an ajax request and the username and password is sent as json data, but I have no idea how to use urllib2 to send json data, help...
For Python 3.x
Note the following
In Python 3.x the urllib and urllib2 modules have been combined. The module is named urllib. So, remember that urllib in Python 2.x and urllib in Python 3.x are DIFFERENT modules.
The POST data for urllib.request.Request in Python 3 does NOT accept a string (str) -- you have to pass a bytes object (or an iterable of bytes)
Example
pass json data with POST in Python 3.x
import urllib.request
import json
json_dict = { 'name': 'some name', 'value': 'some value' }
# convert json_dict to JSON
json_data = json.dumps(json_dict)
# convert str to bytes (ensure encoding is OK)
post_data = json_data.encode('utf-8')
# we should also say the JSON content type header
headers = {}
headers['Content-Type'] = 'application/json'
# now do the request for a url
req = urllib.request.Request(url, post_data, headers)
# send the request
res = urllib.request.urlopen(req)
# res is a file-like object
# ...
Finally note that you can ONLY send a POST request if you have SOME data to send.
If you want to do an HTTP POST without sending any data, you should send an empty dict as data.
data_dict = {}
post_data = json.dumps(data_dict).encode()
req = urllib.request.Request(url, post_data)
res = urllib.request.urlopen(req)
import urllib2
import json
# Whatever structure you need to send goes here:
jdata = json.dumps({"username":"...", "password":"..."})
urllib2.urlopen("http://www.example.com/", jdata)
This assumes you're using HTTP POST to send a simple json object with username and password.
You can specify data upon request:
import urllib
import urllib2
url = 'http://example.com/login'
values = YOUR_CREDENTIALS_JSON
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
You can use the 'requests' python library to achieve this:
http://docs.python-requests.org/en/latest/index.html
You will find this example:
http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests (More complicated POST requests)
>>> import requests
>>> payload = {'key1': 'value1', 'key2': 'value2'}
>>> r = requests.post("http://httpbin.org/post", data=payload)
It seems python do not set good headers when you are trying to send JSON instead of urlencoded data.
With this code, urllib2 make a GET request:
#!/usr/bin/python
import urllib2
req = urllib2.Request('http://www.google.fr')
req.add_header('User-Agent', '')
response = urllib2.urlopen(req)
With this one (which is almost the same), a POST request:
#!/usr/bin/python
import urllib2
headers = { 'User-Agent' : '' }
req = urllib2.Request('http://www.google.fr', '', headers)
response = urllib2.urlopen(req)
My question is: how can i make a GET request with the second code style ?
The documentation (http://docs.python.org/release/2.6.5/library/urllib2.html) says that
headers should be a dictionary, and
will be treated as if add_header() was
called with each key and value as
arguments
Yeah, except that in order to use the headers parameter, you have to pass data, and when data is passed, the request become a POST.
Any help will be very appreciated.
Use:
req = urllib2.Request('http://www.google.fr', None, headers)
or:
req = urllib2.Request('http://www.google.fr', headers=headers)
I have this-
en.wikipedia.org/w/api.php?action=login&lgname=user&lgpassword=password
But it doesn't work because it is a get request. What would the the post request version of this?
Cheers!
The variables for a POST request are in the HTTP headers, not in the URL.
Check urllib.
edit:
Try this (i got it from here):
import urllib
import urllib2
url = 'en.wikipedia.org/w/api.php'
values = {'action' : 'login',
'lgname' : 'user',
'password' : 'password' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
params = urllib.urlencode({'action' : 'login', 'lgname' : 'user', 'lgpassword' : 'password'})
response = urllib.urlopen("http://en.wikipedia.org/w/api.php", params)
info about urllib can be found here.
Since your sample is in PHP, use $_REQUEST, this holds the contents of both $_GET and $_POST.