So I am trying to log in to a system using Python Requests module. When I do the post request:
r = requests.post(url, proxies=proxies, headers=headers, cookies=cookies, data=payload, verify=False)
print(r.status_code)
print(r.cookies)
and I try to print out the cookies, it shows me none. However, in the Burp Suite, I can see that it gets the response with the cookies I need, but it automatically asks for a landing page of the website and for some other stuff (basically it performs 2 more requests using the cookies I got in the first response). So when I ask for the cookies, it shows me cookies of the last response (obviously there are not any).
How can I get cookies from the first request?
A solution could be to set the flag allow_redirects=False and check the response for cookies and then follow the redirect manually. Given that it is a redirect that initiates the next request.
Related
I have a software in python that use post request to reach an API from an external website.
Sometimes, instead of getting the result of my request, I got an URL with a captcha to solve.
Here the code of the requests
headers= {}
headers['Accept']= 'application/json,application/hal+json'
headers['Content-Type']= 'application/json; charset=UTF-8'
headers['Connection']= 'Keep-Alive'
url = 'https://api.website.com/v1/price'
data = [{'category':int(category_id)}]
r = requests.session()
r = requests.post(url, headers=headers, data=json.dumps(data))
So far it's OK, then I get all the date in the following variable and print then in the software using Tkinter:
json_result = json.loads(r.text)
Unfortunately, sometimes the API returns an URL with a link to solve a captcha
{"url":"https://geo.captcha-delivery.com/captcha/?initialCid=AHrlqAAAMAwIhGhOSjsfUAsLEFTg==&cid=Y.M~8XWYwAHts6n_H32O8fm~WTeMw.1cDlRGOH16q4PhWtuo8Xm~KgWeW6d1jQptRljywWkJHFMu9IgEZYRheo3OPww6XjgqXQcs1X1m&referer=https%3A%2F%2Fapi.website.com&hash=05B30BD905598BD2EE8F5A199D973&t=fe&s=575"}
If I open this url in a browser, I can solve the captcha but then I broke the chain so the software is unable to continues.
How could I open in python the url, solve the captcha and when it is solved, continue the request to get the element ?
EDIT : the solution must store the cookie in the request session
Thank you
I'm trying to run a search on a website for the word 'Adrian'. I already understand that first I have to send a request to the website, in the response I will have an XSRF-token that I need to use for the second request. As I understand, if I'm using session.get(), it keeps the cookies automatically for the second request, too.
I run the first request, get a 200 OK response, I print out the cookies, the token is there. I run the second request, I get back a 400 error but if I print out the header of the second request, the token is there. I don't know where it is going wrong.
Why do I get 400 for the second one?
import requests
session = requests.Session()
response = session.get('https://www.racebets.com/en/horse-racing/formguide')
print(response)
cookies = session.cookies.get_dict()
print(cookies)
XSRFtoken = cookies['XSRF-TOKEN']
print(XSRFtoken)
response = session.get('https://www.racebets.com/ajax/formguide/search?s=Adrian')
print(response)
print(response.request.headers)
I also tried to skip session and use requests.get() in the second request and add the token to the header by myself but the result is the same:
import requests
session = requests.Session()
response = session.get('https://www.racebets.com/en/horse-racing/formguide')
print(response)
cookies = session.cookies.get_dict()
print(cookies)
XSRFtoken = cookies['XSRF-TOKEN']
print(XSRFtoken)
headers = {'XSRF-TOKEN': XSRFtoken}
response = session.get('https://www.racebets.com/ajax/formguide/search?s=Adrian', headers=headers)
print(response)
print(response.request.headers)
As Paul said:
The API you're trying to make an HTTP GET request to cares about two request headers: cookie and x-xsrf-token. Log your browser's network traffic to see what they're composed of.
The header needs to be named x-xsrf-token. Try this:
token = session.cookies.get('XSRF-TOKEN')
headers = {'X-XSRF-TOKEN': token}
response = session.get('https://www.racebets.com/ajax/formguide/search?s=Adrian', headers=headers)
I continue to get a very generic, unhelpful error message from python requests when making a POST request that works fine in Postman.
No matter what I try, I continue to receive one of two error messages. And note that the calling python script does not have a line 155, nor does the payload contain the letter "u":
{"error":{"detail":"SyntaxError: Unexpected token: u (sys_script_include.d2426c9ec0a8016501958bf2ac79c775.script; line 155)","message":"Unexpected token: u"},"status":"failure"}
{"error":{"message":"Unexpected token: u","detail":"SyntaxError: Unexpected token: u (sys_script_include.d2426c9ec0a8016501958bf2ac79c775.script; line 155)"},"status":"failure"}
In Postman, the parameters are correctly interpreted and then appended to the url such as:
https://areallylongurl?params={"catalogItem": "Req Name"}
In Python Requests I have tried various combinations of things without luck.
payload = {"params": '{"catalogItem": "Req Name"}'}
response = requests.post(url, headers=headers, json=payload, verify=False)
response = requests.post(url, headers=headers, json=json.dumps(payload), verify=False)
response = requests.post(url, headers=headers, data=payload, verify=False)
response = requests.post(url, headers=headers, data=json.dumps(payload), verify=False)
By using this very helpful SO answer, I was able to further analyze how the Requests library interpreted my provided payload but I am still not sure exactly how to interpret this generic error message, or what the cause may be.
Does anyone have thoughts as to what the underlying issue could be? And note that I can GET from this API without issue from Requests, it's only the POST that's problematic.
Since in postman the parameters are "appended to the url" like https://areallylongurl?params={"catalogItem": "Req Name"}, it means that the request is likely a GET request with JSON passed as a value to the params parameter, rather than a payload to a POST request, in which case you should do this instead:
response = requests.get(url, headers=headers, params={"params": json.dumps(payload)}, verify=False)
I'm making a request to an api using python3 urllib. This is my code,
headers = {}
headers['Content-Type']='application/x-www-form-urlencoded; charset=UTF-8'
#headers['X-Csrf-Token']= {'mode: cors'}
req = urllib.request.Request(url=URL, headers=headers, method='POST')
res = urllib.request.urlopen(req)
print(res.read())
I get the following response,
urllib.error.HTTPError: HTTP Error 415: Unsupported Media Type
The API endpoint is working fine and I've tested it with Postman. This is the equivalent javascript code which gets the job done,
return fetch(url, { credentials : 'include', method: 'post'})
So I'm assuming I've to find a way to add credentials to the header. Can I do a credentials include parallel in Python or do I have to fetch the specific cookie and set it in the request. Any help appreciated.
The error 415 indicates an issue with 'Content-type'.
'application/x-www-form-urlencoded' does not have any parameters.
https://www.w3.org/TR/html5/iana.html#application/x-www-form-urlencoded
Regarding fetch()'s "credentials", you would need the ability to maintain a session (sending cookies as required).
It is much easier with the requests library's Session
http://docs.python-requests.org/en/master/user/advanced/#session-objects
I am trying to log in to a page and access another link in the page.
I get a "405 Not Allowed" error from this attempt:
payload={'username'=<username>,'password'=<password>}
with session() as s:
r = c.post(<URL>, data=payload)
print(r)
print(r.content)
I checked the post method details using Chrome developer tools and found a URL that appeard to be an API endpoint. I posted to that URL with the payload and it seemed to work; I got a response similar to what I could see in the developer.
Unfortunately, when trying to 'get' another URL after logging in, I am still getting the content from the login page.
Why is the login not sticking? Should I use cookies? How?
You can use a session object. It stores the cookies so you can make requests, and it handles the cookies for you
s = requests.Session()
# all cookies received will be stored in the session object
s.post('http://www...',data=payload)
s.get('http://www...')
Docs: https://requests.readthedocs.io/en/master/user/advanced/#session-objects
You can also save the cookie data to an external file, and then reload them to keep session persistent without having to login every time you run the script:
How to save requests (python) cookies to a file?
From the documentation:
get cookie from response
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies
{'example_cookie_name': 'example_cookie_value'}
give cookie back to server on subsequent request
url = 'http://httpbin.org/cookies'
cookies = {'cookies_are': 'working'}
r = requests.get(url, cookies=cookies)`
Summary (#Freek Wiekmeijer, #gtalarico) other's answer:
Logic of Login
Many resource(pages, api) need authentication, then can access, otherwise 405 Not Allowed
Common authentication=grant access method are:
cookie
auth header
Basic xxx
Authorization xxx
How use cookie in requests to auth
first get/generate cookie
send cookie for following request
manual set cookie in headers
auto process cookie by requests's
session to auto manage cookies
response.cookies to manually set cookies
use requests's session auto manage cookies
curSession = requests.Session()
# all cookies received will be stored in the session object
payload={'username': "yourName",'password': "yourPassword"}
curSession.post(firstUrl, data=payload)
# internally return your expected cookies, can use for following auth
# internally use previously generated cookies, can access the resources
curSession.get(secondUrl)
curSession.get(thirdUrl)
manually control requests's response.cookies
payload={'username': "yourName",'password': "yourPassword"}
resp1 = requests.post(firstUrl, data=payload)
# manually pass previously returned cookies into following request
resp2 = requests.get(secondUrl, cookies= resp1.cookies)
resp3 = requests.get(thirdUrl, cookies= resp2.cookies)
As others noted, Here is an example of how to add cookies as string variable to the headers parameter -
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
'cookie': '_fbp=fb.1.1654447470850.2143140577; _ga=GA1.2.1...'
}
response = requests.get(url, headers=headers)