I have experiment with sessions in requests. One thing confuses me: when I reuse a session, on the second request the cookies are empty.
This short example boils it down, and the result is same with all host I try.
import requests
import time
# ==== First Request ====
session = requests.Session()
response = session.get(url="http://www.example.com")
print(response.cookies)
# <RequestsCookieJar[<Cookie UID=759854d4058cf52df60bbbe2a19d1402f5aee (...)
time.sleep(2)
# ==== Second Request ====
response = session.get(url="http://www.example.com")
print(response.cookies)
# <RequestsCookieJar[]> (EMPTY!)
But according to documentation:
The Session object allows you to persist certain parameters across
requests. It also persists cookies across all requests made from the
Session instance (...)
What am I missing?
Edit: the answer explained that I was doing wrong. And dir(session) made me realize that the cookies were stored in session.cookies
This is because you check the response's http header instead of the request.
Your first request creates the session on the server for the first time and the server responds to your request with the Set-Cookie HTTP header. This is what you see in the printout of the first response.
In your second request, the session is already created, therefore the server doesn't need to include the cookie in its response.
Try to examine your requests instead of the responses.
Related
I want to start a requests.Session() and add a cookie before starting the first request. I expected to have a cookie argument or something similar to do this
def session_start()
self.session = requests.Session(cookies=['session-id', 'xxx'])
def req1():
self.session.get('example.org')
def req2():
self.session.get('example2.org')
but this wont work, I only can provide cookies in the .get() method. Do I need to do a "dummy request" in session_start() or is there a way to prepare the cookie before starting the actual request?
From the documentation:
Note, however, that method-level parameters will not be persisted across requests, even if using a session. This example will only send the cookies with the first request, but not the second:
s = requests.Session()
r = s.get('https://httpbin.org/cookies', cookies={'from-my': 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'
r = s.get('https://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'
session.cookies is a RequestsCookieJar and you can add cookies to that after constructing the session:
self.session = requests.Session()
self.session.cookies.set('session-id', 'xxx')
So session objects will persist any cookies that the url requests themselves set, but if you provide a cookie as an argument it will not persist on the next request.
You can add cookies in manually that persist too though, from the documentation:
"If you want to manually add cookies to your session, use the Cookie utility functions to manipulate Session.cookies."
https://requests.readthedocs.io/en/master/user/advanced/#session-objects
Where it links to on how to manipulate cookies: https://requests.readthedocs.io/en/master/api/#api-cookies
I'm relatively new to Python so excuse any errors or misconceptions I may have. I've done hours and hours of research and have hit a stopping point.
I'm using the Requests library to pull data from a website that requires a login. I was initially successful logging in through through a session.post,(payload)/session.get. I had a [200] response. Once I tried to view the JSON data that was beyond the login, I hit a [403] response. Long story short, I can make it work by logging in through a browser and inspecting the web elements to find the current session cookie and then defining the headers in requests to pass along that exact cookie with session.get
My questions is...is it possible to set/generate/find this cookie through python after logging in? After logging in and out a few times, I can see that some of the components of the cookie remain the same but others do not. The website I'm using is garmin connect.
Any and all help is appreciated.
If your issue is about login purposes, then you can use a session object. It stores the corresponding cookies so you can make requests, and it generally handles the cookies for you. Here is an example:
s = requests.Session()
# all cookies received will be stored in the session object
s.post('http://www...',data=payload)
s.get('http://www...')
Furthermore, with the requests library, you can get a cookie from a response, like this:
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies
But you can also give cookie back to the server on subsequent requests, like this:
url = 'http://httpbin.org/cookies'
cookies = dict(cookies_are='working')
r = requests.get(url, cookies=cookies)
I hope this helps!
Reference: How to use cookies in Python Requests
I'm trying to get data from
https://www.biman-airlines.com/bookings/flight_selection.aspx
For example, when I choose flight from Dhaka(DAC) to Sylhet(ZYL), it goes to
https://www.biman-airlines.com/bookings/flight_selection.aspx?TT=RT&SS=&RT=&FL=on&DC=DAC&AC=ZYL&AM=2018-01&AD=09&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&RM=2018-01&RD=10&PA=1&PT=&PC=&PI=&CC=&NS=&CD=&FS=B4B9631
and shows the flight information
but when I'm trying to perform such get request using python, it shows no info
here is my code:
import requests
print(requests.get('https://www.biman-airlines.com/bookings/flight_selection.aspx?TT=RT&SS=&RT=&FL=on&DC=DAC&AC=ZYL&AM=2018-01&AD=09&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&RM=2018-01&RD=10&PA=1&PT=&PC=&PI=&CC=&NS=&CD=&FS=').text)
What am I doing wrong?
thanks in advance for any help
but when I'm trying to perform such get request using python, it shows no info. What am I doing wrong?
The request result shows no info because there is no cookie data in the python HTTP request.
If you check the HTTP request in browser debug window, you can see there is cookie along with the request -- the cookie identifies who the client is and tells server "Hi, server, I'm a valid user":
With reasonable guess, in this biman-airlines.com case, the server would check the cookie and return result only if the cookie is valid.
Thus, you need to add your Cookie header in the python code:
# The cookie below is just for example, you would get your own cookie once visiting the website.
headers = {
'Cookie': 'chocolateChip=nbixfy44dvziejjdxd2wmzs3; BNI_bg_zapways=0000000000000000000000009301a8c000005000; ASPSESSIONIDSQDCSSDT=PFJPADACFOGBDMONPBHPMFJN'
}
print(requests.get('https://www.biman-airlines.com/bookings/flight_selection.aspx?TT=RT&SS=&RT=&FL=on&DC=DAC&AC=ZYL&AM=2018-01&AD=09&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&DC=&AC=&AM=&AD=&RM=2018-01&RD=10&PA=1&PT=&PC=&PI=&CC=&NS=&CD=&FS=B4B9631', headers=headers).text)
I'm trying to send a POST Request with form data to some form. I'm using requests package.
#!/bin/python3.6
import requests
FFF_search = 'https://www.fff.fr/la-vie-des-clubs/resultats'
data = {'address':75000}
session = requests.session()
r = session.post(FFF_search, data=data, headers={'User-Agent':'test'})
print(r.text)
This gives me the result page where no result is found.
Maybe my problem is more about the data I'm sending.
Here's the POST request as it must be done on the website (Chrome dev tools).
You should consider several issues:
Your's headers dict is not valid. There is no 'User-Agnet' that called test. You should use only valid headers headers. Try to send all the headers you see in the network tab.
Your's data us clearly wrong. You should send the described Form Data ad a dictionary and pass it with data=data, as you do (but you do it with a wrong data). Maybe the data you are passing now is the params?
Try to send the exact request like your's browser sending (with the same params and headers), and see the results you are receiving, with paying attention to the issues above.
I have problem with simple authorization and upload API script.
When authorized, client receives several cookies, including PHPSESSID cookie (in browser).
I use requests.post method with form data for authorization:
r = requests.post(url, headers = self.headers, data = formData)
self.cookies = requests.utils.dict_from_cookieja(r.cookies)
Headers are used for custom User-Agent only.
Authorization is 100% fine (there is a logout link on the page).
Later, i try to upload data using the authorized session cookies:
r = requests.post(url, files = files, data = formData, headers = self.headers, cookies = self.cookies)
But site rejects the request. If we compare the requests from script and google chrome (using Wireshark), there is no differences in request body.
Only difference is that 2 cookies sent by requests class, while google chrome sends 7.
Update: Double checked, first request receives 7 cookies. post method just ignore half...
My mistake in code was that i was assigning cookies from each next API request to the session cookies dictionary. On each request since logged in, cookies was 'reset' by upcoming response cookies, that's was the problem. As auth cookies are assigned only at login request, they were lost at the next request.
After each authorized request i use update(), not assigning.
self.cookies.update( requests.utils.dict_from_cookiejar(r.cookies) )
Solves my issue, upload works fine!