I need to use urllib/urllib2 libraries to login to a first website to retrieve session cookie that will allow me to log in to the proper, final website. Using requests library is pretty straight forward (I did it to make sure I can actually access the website):
import requests
payload = {"userName": "username", "password": "password", "apiKey": "myApiKey"}
url = "https://sso.somewebsite.com/api/authenticateme"
session = requests.session()
r = session.post(url, payload)
# Now that I have a cookie I can actually access my final website
r2 = session.get("https://websiteineed.somewebsite.com")
I tried to replicate this behavior using urllib/urllib2 libraries but keep getting HTTP Error 403: Forbidden:
cj = cookielib.CookieJar()
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cj))
urllib2.install_opener(opener)
values = {"userId": username , "password": password, "apiKey": apiKey}
url = 'https://sso.somewebsite.com/api/authenticateme'
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
resp = urllib2.urlopen(req)
req2 = urllib2.Request('https://download.somewebsite.com')
resp2 = urllib2.urlopen(req2)
I tried solutions I found here and here and here but none of them worked for me... I would appreciate any suggestions!
The reason why the 'final page' was rejecting the cookies is because Python was adding 'User-agent', 'Python-urllib/2.7'to the header. After removing this element I was able to login to a website:
opener.addheaders.pop(0)
I tried to get successful login post request, but don't understand why do i always get status code 405. I checked many tutorials and all of them worked good for some examples.
import requests
import urllib.parse
def logintest(usr, pswd):
link = 'link'
headers = { 'Content-Type' : 'application/x-www-form-urlencoded',
'Server' : 'Apache-Coyote/1.1',
'Transfer-Encoding' : 'chunked'
}
payload = urllib.parse.urlencode({
'username': usr,
'password': pswd
})
responseget = requests.get(link)
print (responseget.status_code, responseget.reason)
if responseget.status_code == 200:
responsepost = requests.post(link, params = payload, headers = headers)
print (responsepost.status_code, responsepost.reason)
logintest('username', 'password')
Because the elibrary/home endpoint does not support POST requests. You probably want to send the request to http://81.180.75.144:8080/elibrary/auth/login, like the system does.
I want to fill a HTML form with urllib2 and urllib.
import urllib
import urllib2
url = 'site.com/registration.php'
values = {'password' : 'password',
'username': 'username'
}
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
But on the end of the form is a button(input type='submit'). If you don't click the button you can't send the data what you wrote in the input(type text)
How can I click the button with urllib and urllib2?
IF you look at your forms action attribute you could figure out what uri your form is submitting to. You can then make a post request to that uri.
This can be done like :
import urllib
import urllib2
url = 'http://www.someserver.com/cgi-bin/register.cgi'
values = {'name' : 'Michael Foord',
'location' : 'Northampton',
'language' : 'Python' }
data = urllib.urlencode(values)
req = urllib2.Request(url, data)
response = urllib2.urlopen(req)
the_page = response.read()
https://docs.python.org/2/howto/urllib2.html
You could also use the requests library which makes it a lot easier. You can read about it here
Another thing you might need to factor in is the CSRF(Cross Site Request Forgery) token which is embedded in forms. You will have to some how acquire it and pass it in with your post.
This is really more of something you would do with Selenium or similar. selenium.webdriver.common.action_chains.ActionChains.click, perhaps.
With this code, urllib2 make a GET request:
#!/usr/bin/python
import urllib2
req = urllib2.Request('http://www.google.fr')
req.add_header('User-Agent', '')
response = urllib2.urlopen(req)
With this one (which is almost the same), a POST request:
#!/usr/bin/python
import urllib2
headers = { 'User-Agent' : '' }
req = urllib2.Request('http://www.google.fr', '', headers)
response = urllib2.urlopen(req)
My question is: how can i make a GET request with the second code style ?
The documentation (http://docs.python.org/release/2.6.5/library/urllib2.html) says that
headers should be a dictionary, and
will be treated as if add_header() was
called with each key and value as
arguments
Yeah, except that in order to use the headers parameter, you have to pass data, and when data is passed, the request become a POST.
Any help will be very appreciated.
Use:
req = urllib2.Request('http://www.google.fr', None, headers)
or:
req = urllib2.Request('http://www.google.fr', headers=headers)
Using Chris Atlee's python poster library is there any way to include cookie handling?
I have python http login code, which works with cookies:
cookiejar = cookielib.CookieJar()
urlOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
request = urllib2.Request(login_url, params)
result = urlOpener.open(request)
But when I need to upload a file, I don't know how to use both poster lib code and cookie handling code. Poster lib seems to need to call urllib2.urlopen() and not some custom url opener, like in the code above.
For instance, I don't know how to use cookies with the python file post code below:
register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)
I sent an email to Chris AtLee asking whether we could get a basic authentication example. He was very cool about answering my questions and even ran some example code I sent him.
To include cookie handling, you do something like this:
opener = poster.streaminghttp.register_openers()
opener.add_handler(urllib2.HTTPCookieProcessor(cookielib.CookieJar())) # Add cookie handler
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = poster.encode.multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)
To add basic authentication to the request, you simply do this (I added the base64 encode line for completeness):
opener = poster.streaminghttp.register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = poster.encode.multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
auth = base64.encodestring('%s:%s' % ('username', 'password'))[:-1] # This is just standard un/pw encoding
request.add_header('Authorization', 'Basic %s' % auth ) # Add Auth header to request
result = urllib2.urlopen(request)
Hope this helps. And another big thanks to Chris AtLee.
You don't have to modify the original source code, just install all required openers manually (without calling register_openers()):
import urllib2
import cookielib
import poster
handlers = [poster.streaminghttp.StreamingHTTPHandler(),
poster.streaminghttp.StreamingHTTPRedirectHandler(),
urllib2.HTTPCookieProcessor(cookielib.CookieJar())]
urllib2.install_opener(urllib2.build_opener(*handlers))
datagen, headers = poster.encode.multipart_encode({"image1": open("DSC0001.jpg", "rb")})
request = urllib2.Request("http://localhost:5000/upload_image", datagen, headers)
print urllib2.urlopen(request).read()
Have you tried this:
cookiejar = cookielib.CookieJar()
urlOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
register_openers()
params = {'file': open("test.txt", "rb"), 'name': 'upload test'}
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urlOpener.open(request)
I've figured out how to do this. I'm not sure if this is the best way to go about things, but it works, so I'll share it. In order to use the poster lib with cookies one must add urllib2.HTTPCookieProcessor to the opener built in poster.streaminghttp.register_openers().
Essentially, modify poster.streaminghttp.register_openers() to look like the code below, and if you want to have cookie handling, pass in a cookiejar object.
def register_openers(cookiejar=None):
"""Register the streaming http handlers in the global urllib2 default
opener object.
Returns the created OpenerDirector object."""
handlers = [StreamingHTTPHandler, StreamingHTTPRedirectHandler]
if hasattr(httplib, "HTTPS"):
handlers.append(StreamingHTTPSHandler)
if cookiejar:
opener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar), *handlers)
else:
opener = urllib2.build_opener(*handlers)
urllib2.install_opener(opener)
return opener
Sample Usage:
# Logging in
import urllib, urllib2, cookielib
from poster.encode import multipart_encode
from poster.streaminghttp import register_openers
cookiejar = cookielib.CookieJar()
loginOpener = urllib2.build_opener(urllib2.HTTPCookieProcessor(cookiejar))
params = urllib.urlencode({'username':'admin', 'password':'default'})
login_url = "http://127.0.0.1:8000/account/login/"
request = urllib2.Request(login_url, params)
login = loginOpener.open(request)
# Upload File
# use the login cookie for file upload
register_openers(cookiejar=cookiejar)
params = {'entity_file': open("test.txt", "rb"),'name': 'test', 'action':'upload'}
upload_url = "http://127.0.0.1:8000/upload/"
datagen, headers = multipart_encode(params)
request = urllib2.Request(upload_url, datagen, headers)
result = urllib2.urlopen(request)