We Keep Coding

Cannot get cookies with python requests while postman, curl, and wget work

I'm trying to authenticate on a French water provider website to get my water consumption data. The website does not provide any api and I'm trying to make a python script that authenticates on the website and crawls the data. My work is based on a working Domoticz python script and a shell script.
The workflow is the following:
Get a token from the website
Authenticate with login, password, and token get at step 1
Get 1 or more cookies from step 2
Get data using the cookie(s) from 3
I'm stuck at step 2 where I can't get the cookies with my python script. I tried with postman, curl, and wget and it is working. I even used the python code generated by postman and I still get no cookies.
Heres is a screenshot of my postman post request
which gives two cookies in the response.
And here is my python code:
import requests
url = "https://www.toutsurmoneau.fr/mon-compte-en-ligne/je-me-connecte"
querystring = {"_username":"mymail#gmail.com","_password":"mypass","_csrf_token":"knfOIFZNhiCVxHS0U84GW5CrfMt36eLvqPPYGDSsOww","signin[username]":"mymail#gmail.com","signin[password]":"mypass","tsme_user_login[_username]":"mymail#gmail.com","tsme_user_login[_password]":"mypass"}
payload = ""
headers = {
'Accept': "application/json, text/javascript, */*; q=0.01",
'Content-Type': "application/x-www-form-urlencoded",
'Accept-Language': "fr,fr-FR;q=0.8,en;q=0.6",
'User-Agent': "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_6_8) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Mobile Safari/537.36",
'Connection': "keep-alive",
'cache-control': "no-cache",
'Postman-Token': "c7e5f7ca-abea-4161-999a-3c28ec979628"
}
response = requests.request("POST", url, data=payload, headers=headers, params=querystring)
print(response.cookies.get_dict())
The output is {}.
I cannot figure out what I'm doing wrong.
If you have any help to provide, I'll be happy to get it.
Thanks for reading.
Edit:
Some of my assumptions were wrong. The shell script was indeed working but not Postman. I was confused because of response 200 I receive.

So I answer my own question.
First, when getting the token at step 1, I receive a cookie. I'm supposed to use this cookie when logging in which I did not do before.
Then, when using this cookie and the token to log in step 2, I was not able to see any cookie in the response I receive while I was well connected (I find in the content a "disconnect" string which is here only if well logged in). That's a normal behavior since cookies are not sent in the response of a post request.
I had to create a requests.session to post my log in form, and the session stores the cookie.
Now, I'm able to use this information to grab the data from the server.
Hope that will help others.

session auth in python

Using session from requests module in python, it seems that the session sends authorization only with first request, I can't understand why this happened.
import requests
session = requests.Session()
session.auth = (u'user', 'test')
session.verify = False
response = session.get(url='https://my_url/rest/api/1.0/users')
If I look for this response request headers I see:
{'Authorization': 'Basic auth_data', 'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.12.3'}
but if I send next request using the same or not url:
response = session.get(url='https://my_url/rest/api/1.0/users')
I can see that there is no auth header in request anymore:
print response.request.headers
{'Connection': 'keep-alive', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.12.3'}
And I'm getting 401 response because of it.
Why is it so? Shouldn't session send auth with every request made using it?
How can I send auth data with every request using session?

What I see when I run that exact code in your comment is that the Authorization header is missing in the first print, yet it is present in the second. This seems to be the opposite of the problem that you report.
This is explained by the fact that the first request is redirected by a 301 response, and the auth header is not propagated in the follow up request to the redirected location. You can see that the auth header was sent in the initial request by looking in response.history[0].request.headers.
The second request is not redirected because the session has kept the connection to the host open (due the the Connection: keep-alive header), so the auth headers appear when you print response.request.headers.
I doubt that you are actually using https://test.com, but probably a similar thing is happening with the server that you are using.
For testing I recommend using the very handy public test HTTP server https://httpbin.org/headers. This will return the headers received by the server in the response body. You can test redirected requests with one of the redirect URLs.

I didn't find any reliable answer on how to pass, auth info while making request's session in python. So below is my finding:
with requests.sessions.Session() as session:
session.auth = ("username", "password")
# Make any requests here without provide auth info again
session.get("http://www.example.com/users")

How can I use cookies in Python Requests?

I am trying to log in to a page and access another link in the page.
I get a "405 Not Allowed" error from this attempt:
payload={'username'=<username>,'password'=<password>}
with session() as s:
r = c.post(<URL>, data=payload)
print(r)
print(r.content)
I checked the post method details using Chrome developer tools and found a URL that appeard to be an API endpoint. I posted to that URL with the payload and it seemed to work; I got a response similar to what I could see in the developer.
Unfortunately, when trying to 'get' another URL after logging in, I am still getting the content from the login page.
Why is the login not sticking? Should I use cookies? How?

You can use a session object. It stores the cookies so you can make requests, and it handles the cookies for you
s = requests.Session()
# all cookies received will be stored in the session object
s.post('http://www...',data=payload)
s.get('http://www...')
Docs: https://requests.readthedocs.io/en/master/user/advanced/#session-objects
You can also save the cookie data to an external file, and then reload them to keep session persistent without having to login every time you run the script:
How to save requests (python) cookies to a file?

From the documentation:
get cookie from response
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies
{'example_cookie_name': 'example_cookie_value'}
give cookie back to server on subsequent request
url = 'http://httpbin.org/cookies'
cookies = {'cookies_are': 'working'}
r = requests.get(url, cookies=cookies)`

Summary (#Freek Wiekmeijer, #gtalarico) other's answer:
Logic of Login
Many resource(pages, api) need authentication, then can access, otherwise 405 Not Allowed
Common authentication=grant access method are:
cookie
auth header
Basic xxx
Authorization xxx
How use cookie in requests to auth
first get/generate cookie
send cookie for following request
manual set cookie in headers
auto process cookie by requests's
session to auto manage cookies
response.cookies to manually set cookies
use requests's session auto manage cookies
curSession = requests.Session()
# all cookies received will be stored in the session object
payload={'username': "yourName",'password': "yourPassword"}
curSession.post(firstUrl, data=payload)
# internally return your expected cookies, can use for following auth
# internally use previously generated cookies, can access the resources
curSession.get(secondUrl)
curSession.get(thirdUrl)
manually control requests's response.cookies
payload={'username': "yourName",'password': "yourPassword"}
resp1 = requests.post(firstUrl, data=payload)
# manually pass previously returned cookies into following request
resp2 = requests.get(secondUrl, cookies= resp1.cookies)
resp3 = requests.get(thirdUrl, cookies= resp2.cookies)

As others noted, Here is an example of how to add cookies as string variable to the headers parameter -
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
'cookie': '_fbp=fb.1.1654447470850.2143140577; _ga=GA1.2.1...'
}
response = requests.get(url, headers=headers)

Django rejects Requests' CSRF Token

I'm writing an Ajax post with python's Request's library to a django backend
Code:
import requests
import json
import sys
URL = 'http://localhost:8000/'
client = requests.session()
client.get(URL)
csrftoken = client.cookies['csrftoken']
data = { 'file': "print \"It works!\"", 'fileName' : "JSONtest", 'fileExt':".py",'eDays':'99','eHours':'1', 'eMinutes':'1' }
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken}
r = requests.post(URL+"au", data=json.dumps(data), headers=headers)
Django gives me a 403 error stating that the CSRF token isn't set even though the request.META from csrf_failure() shows it is set. Is there something I'm missing or a stupid mistake I'm not catching?

I asked my friend and he figured out the problem, basically you have to send the cookies that django gives you every time you do a request.
corrected:
cookies = dict(client.cookies)
r = requests.post(URL+"au", data=json.dumps(data), headers=headers,cookies=cookies)

You need to pass the referer to the headers, from the django docs:
In addition, for HTTPS requests, strict referer checking is done by
CsrfViewMiddleware. This is necessary to address a Man-In-The-Middle
attack that is possible under HTTPS when using a session independent
nonce, due to the fact that HTTP ‘Set-Cookie’ headers are
(unfortunately) accepted by clients that are talking to a site under
HTTPS. (Referer checking is not done for HTTP requests because the
presence of the Referer header is not reliable enough under HTTP.)
so change this:
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken, "Referer": URL}

Python create cookies and then load a page with the cookies

I would like to access a web page from a python program.
I have to set up cookies to load the page.
I used the httplib2 library, but I didn't find how add my own cookie
resp_headers, content = h.request("http://www.theURL.com", "GET")
How can I create cookies with the right name and value, add it to the function and then load the page?
Thanks

From http://code.google.com/p/httplib2/wiki/Examples hope will help )
Cookies
When automating something, you often need to "login" to maintain some sort of session/state with the server. Sometimes this is achieved with form-based authentication and cookies. You post a form to the server, and it responds with a cookie in the incoming HTTP header. You need to pass this cookie back to the server in subsequent requests to maintain state or to keep a session alive.
Here is an example of how to deal with cookies when doing your HTTP Post.
First, lets import the modules we will use:
import urllib
import httplib2
Now, lets define the data we will need. In this case, we are doing a form post with 2 fields representing a username and a password.
url = 'http://www.example.com/login'
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
Now we can send the HTTP request:
http = httplib2.Http()
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
At this point, our "response" variable contains a dictionary of HTTP header fields that were returned by the server. If a cookie was returned, you would see a "set-cookie" field containing the cookie value. We want to take this value and put it into the outgoing HTTP header for our subsequent requests:
headers['Cookie'] = response['set-cookie']
Now we can send a request using this header and it will contain the cookie, so the server can recognize us.
So... here is the whole thing in a script. We login to a site and then make another request using the cookie we received:
#!/usr/bin/env python
import urllib
import httplib2
http = httplib2.Http()
url = 'http://www.example.com/login'
body = {'USERNAME': 'foo', 'PASSWORD': 'bar'}
headers = {'Content-type': 'application/x-www-form-urlencoded'}
response, content = http.request(url, 'POST', headers=headers, body=urllib.urlencode(body))
headers = {'Cookie': response['set-cookie']}
url = 'http://www.example.com/home'
response, content = http.request(url, 'GET', headers=headers)

http = httplib2.Http()
# get cookie_value here
headers = {'Cookie':cookie_value}
response, content = http.request("http://www.theURL.com", 'GET', headers=headers)
You may want to add another header parameters to specify another HTTP request parameters.

You can also use just urllib2 library
import urllib2
opener = urllib2.build_opener()
opener.addheaders.append(('Cookie', 'cookie1=value1;cookie2=value2'))
f = opener.open("http://www.example.com/")
the_page_html = f.read()