Using session from requests module in python, it seems that the session sends authorization only with first request, I can't understand why this happened.
import requests
session = requests.Session()
session.auth = (u'user', 'test')
session.verify = False
response = session.get(url='https://my_url/rest/api/1.0/users')
If I look for this response request headers I see:
{'Authorization': 'Basic auth_data', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.12.3'}
but if I send next request using the same or not url:
response = session.get(url='https://my_url/rest/api/1.0/users')
I can see that there is no auth header in request anymore:
print response.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.12.3'}
And I'm getting 401 response because of it.
Why is it so? Shouldn't session send auth with every request made using it?
How can I send auth data with every request using session?
What I see when I run that exact code in your comment is that the Authorization header is missing in the first print, yet it is present in the second. This seems to be the opposite of the problem that you report.
This is explained by the fact that the first request is redirected by a 301 response, and the auth header is not propagated in the follow up request to the redirected location. You can see that the auth header was sent in the initial request by looking in response.history[0].request.headers.
The second request is not redirected because the session has kept the connection to the host open (due the the Connection: keep-alive header), so the auth headers appear when you print response.request.headers.
I doubt that you are actually using https://test.com, but probably a similar thing is happening with the server that you are using.
For testing I recommend using the very handy public test HTTP server https://httpbin.org/headers. This will return the headers received by the server in the response body. You can test redirected requests with one of the redirect URLs.
I didn't find any reliable answer on how to pass, auth info while making request's session in python. So below is my finding:
with requests.sessions.Session() as session:
session.auth = ("username", "password")
# Make any requests here without provide auth info again
session.get("http://www.example.com/users")
Related
I'm trying to authenticate on a French water provider website to get my water consumption data. The website does not provide any api and I'm trying to make a python script that authenticates on the website and crawls the data. My work is based on a working Domoticz python script and a shell script.
The workflow is the following:
Get a token from the website
Authenticate with login, password, and token get at step 1
Get 1 or more cookies from step 2
Get data using the cookie(s) from 3
I'm stuck at step 2 where I can't get the cookies with my python script. I tried with postman, curl, and wget and it is working. I even used the python code generated by postman and I still get no cookies.
Heres is a screenshot of my postman post request
which gives two cookies in the response.
And here is my python code:
import requests
url = "https://www.toutsurmoneau.fr/mon-compte-en-ligne/je-me-connecte"
querystring = {"_username":"mymail#gmail.com","_password":"mypass","_csrf_token":"knfOIFZNhiCVxHS0U84GW5CrfMt36eLvqPPYGDSsOww","signin[username]":"mymail#gmail.com","signin[password]":"mypass","tsme_user_login[_username]":"mymail#gmail.com","tsme_user_login[_password]":"mypass"}
payload = ""
headers = {
'Accept': "application/json, text/javascript, */*; q=0.01",
'Content-Type': "application/x-www-form-urlencoded",
'Accept-Language': "fr,fr-FR;q=0.8,en;q=0.6",
'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Mobile Safari/537.36",
'Connection': "keep-alive",
'cache-control': "no-cache",
'Postman-Token': "c7e5f7ca-abea-4161-999a-3c28ec979628"
}
response = requests.request("POST", url, data=payload, headers=headers, params=querystring)
print(response.cookies.get_dict())
The output is {}.
I cannot figure out what I'm doing wrong.
If you have any help to provide, I'll be happy to get it.
Thanks for reading.
Edit:
Some of my assumptions were wrong. The shell script was indeed working but not Postman. I was confused because of response 200 I receive.
So I answer my own question.
First, when getting the token at step 1, I receive a cookie. I'm supposed to use this cookie when logging in which I did not do before.
Then, when using this cookie and the token to log in step 2, I was not able to see any cookie in the response I receive while I was well connected (I find in the content a "disconnect" string which is here only if well logged in). That's a normal behavior since cookies are not sent in the response of a post request.
I had to create a requests.session to post my log in form, and the session stores the cookie.
Now, I'm able to use this information to grab the data from the server.
Hope that will help others.
Due to requests adding unwanted headers, I decided to prepare the request manually and use Session Send().
Sadly, The following code produces the wrong request
import requests
ARCHIVE_URL = "http://10.0.0.10/post/tmp/archive.zip"
headers = {
'Content-Type': 'application/x-www-form-urlencoded',
'Cache-Control': 'no-cache',
'Connection': 'Keep-Alive',
'Host': '10.0.0.10'
}
DataToSend = 'data'
req = requests.Request('POST', ARCHIVE_URL, data=DataToSend, headers=headers)
prepped = req.prepare()
s = requests.Session()
response = s.send(prepped)
If I look at the request using fiddler I get this:
GET http://10.0.0.10/tmp/archive.zip HTTP/1.1
Accept-Encoding: identity
Connection: Keep-Alive
Host: 10.0.0.10
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
What am I missing?
since prepared request is not connected to the session when using req.prepare() instead of s.prepare_request(req) when s is the session , you must specify request headers since there are no default one that come from the session object.
use s.prepare_request(req) instead of req.prepare() or specify headers dictionary
I've got a script that is meant to at first login to Twitter. I get a 200 response if I check it, but I don't redirect to a logged in Twitter account after succeeding, instead it stays on the same page.
url = 'https://twitter.com/login/error?redirect_after_login=%2F'
r = requests.session()
# Need the headers below as so Twitter doesn't reject the request.
headers = {
'Host': "twitter.com",
'User-Agent': "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0",
'Accept': "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
'Accept-Language': "en-US,en;q=0.5",
'Accept-Encoding': "gzip, deflate, br",
'Referer': "https://twitter.com/login/error?redirect_after_login=%2F",
'Upgrade-Insecure-Requests': "1",
'Connection': "keep-alive"
}
login_data = {"session[username_or_email]":"Username", "session[password]":"Password"}
response = r.post(url, data=login_data, headers=headers, allow_redirects=True)
How do I go about redirecting to my account upon successful POST request to the logged in state. Am I not using the correct headers or something like that? I've not done a huge amount of web stuff before, so I'm sorry if it's something really obvious.
Note: (I cannot use Twitter API to do this) & The referrer is the error page because that's where I'm logging in from - unless of course I'm wrong in doing that.
Perhaps the GET parameter redirect_after_login will use kind of javascript or html meta refresh redirection instead of HTTP redirection, so if it's the case, the requests python module will not handle it correctly.
So once you retrieve your authentication token from your first request, you could make again the second request to https://twiter.com/ without to forget your specify your security token from your HTTP request fields. You can find more information about REST API of twitter here: https://dev.twitter.com/overview/api
But the joy of python is to have libraries for everything, so I suggest you to take a look here:
https://github.com/bear/python-twitter
It's a library to communicate with the REST API of twitter.
I'm writing code trying to get Cookies from webserver using Request module, but what confused me is that:
I'm seeing Cookies returned when I tested using PostMan - REST Client
I didn't sent any cookies in my sent request, but what surprised me is that could find the cookie I want in Sent requests.
I want to get cookie data that I could use in code to request another application.
Following is my code:
import requests
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
username = 'user123' # real username/password not showing
password = '1234567'
login_data = {'id':username, 'pass_word':password, 'action': 'login'}
r = requests.post("www.example.com/login/", data=login_data)
print r.cookies
print r.request.header['Cookie']
Output:
<<class 'requests.cookies.RequestsCookieJar'>[]> # why nothing??
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.6 Darwin/14.1.0', 'Connection': 'keep-alive', 'Cookie': 'blahblahblahblah', 'Content-Type': 'application/x-www-form-urlencoded'}
For confidential reason, I cannot show my real cookies data here, but I do see it's in my send request, how come? I didn't tell it to send it in my request, and this is the data that i expect to get from response cookie, but it showed as none.
The r.cookies jar contains all cookies that are new, that the server has sent in the current response, so any cookies that have been sent will not appear there unless the server resent them. The sent cookies will appear in r.request.headers (the jar is at r.request._cookies). If you want to preserve the cookies across multiple requests, use a Session:
session = requests.Session()
session.get(...) # Your first request
session.get(...) # Following requests pass the cookies on
While using the session, you can retrieve any cookies by looking in session.cookies. I don't know why you saw cookies sent when you didn't, I'd have to see more code.
I'm writing an Ajax post with python's Request's library to a django backend
Code:
import requests
import json
import sys
URL = 'http://localhost:8000/'
client = requests.session()
client.get(URL)
csrftoken = client.cookies['csrftoken']
data = { 'file': "print \"It works!\"", 'fileName' : "JSONtest", 'fileExt':".py",'eDays':'99','eHours':'1', 'eMinutes':'1' }
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken}
r = requests.post(URL+"au", data=json.dumps(data), headers=headers)
Django gives me a 403 error stating that the CSRF token isn't set even though the request.META from csrf_failure() shows it is set. Is there something I'm missing or a stupid mistake I'm not catching?
I asked my friend and he figured out the problem, basically you have to send the cookies that django gives you every time you do a request.
corrected:
cookies = dict(client.cookies)
r = requests.post(URL+"au", data=json.dumps(data), headers=headers,cookies=cookies)
You need to pass the referer to the headers, from the django docs:
In addition, for HTTPS requests, strict referer checking is done by
CsrfViewMiddleware. This is necessary to address a Man-In-The-Middle
attack that is possible under HTTPS when using a session independent
nonce, due to the fact that HTTP ‘Set-Cookie’ headers are
(unfortunately) accepted by clients that are talking to a site under
HTTPS. (Referer checking is not done for HTTP requests because the
presence of the Referer header is not reliable enough under HTTP.)
so change this:
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken, "Referer": URL}