I am trying to log in to a page and access another link in the page.
I get a "405 Not Allowed" error from this attempt:
payload={'username'=<username>,'password'=<password>}
with session() as s:
r = c.post(<URL>, data=payload)
print(r)
print(r.content)
I checked the post method details using Chrome developer tools and found a URL that appeard to be an API endpoint. I posted to that URL with the payload and it seemed to work; I got a response similar to what I could see in the developer.
Unfortunately, when trying to 'get' another URL after logging in, I am still getting the content from the login page.
Why is the login not sticking? Should I use cookies? How?
You can use a session object. It stores the cookies so you can make requests, and it handles the cookies for you
s = requests.Session()
# all cookies received will be stored in the session object
s.post('http://www...',data=payload)
s.get('http://www...')
Docs: https://requests.readthedocs.io/en/master/user/advanced/#session-objects
You can also save the cookie data to an external file, and then reload them to keep session persistent without having to login every time you run the script:
How to save requests (python) cookies to a file?
From the documentation:
get cookie from response
url = 'http://example.com/some/cookie/setting/url'
r = requests.get(url)
r.cookies
{'example_cookie_name': 'example_cookie_value'}
give cookie back to server on subsequent request
url = 'http://httpbin.org/cookies'
cookies = {'cookies_are': 'working'}
r = requests.get(url, cookies=cookies)`
Summary (#Freek Wiekmeijer, #gtalarico) other's answer:
Logic of Login
Many resource(pages, api) need authentication, then can access, otherwise 405 Not Allowed
Common authentication=grant access method are:
cookie
auth header
Basic xxx
Authorization xxx
How use cookie in requests to auth
first get/generate cookie
send cookie for following request
manual set cookie in headers
auto process cookie by requests's
session to auto manage cookies
response.cookies to manually set cookies
use requests's session auto manage cookies
curSession = requests.Session()
# all cookies received will be stored in the session object
payload={'username': "yourName",'password': "yourPassword"}
curSession.post(firstUrl, data=payload)
# internally return your expected cookies, can use for following auth
# internally use previously generated cookies, can access the resources
curSession.get(secondUrl)
curSession.get(thirdUrl)
manually control requests's response.cookies
payload={'username': "yourName",'password': "yourPassword"}
resp1 = requests.post(firstUrl, data=payload)
# manually pass previously returned cookies into following request
resp2 = requests.get(secondUrl, cookies= resp1.cookies)
resp3 = requests.get(thirdUrl, cookies= resp2.cookies)
As others noted, Here is an example of how to add cookies as string variable to the headers parameter -
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) ...',
'cookie': '_fbp=fb.1.1654447470850.2143140577; _ga=GA1.2.1...'
}
response = requests.get(url, headers=headers)
Related
I'm trying to login to a website using python requests, however the webpage has a mandatory data protection consent form pop-up on the first page. I think this is why I cannot yet login, because posting your login credentials to the login URL requires these content cookies (which are probably dynamic).
After checking out the login post headers request (via inspection tools) it says it requires the cookies from a CMP, specifically a variable called euconsent-v2 (https://help.consentmanager.net/books/cmp/page/cookies-set-by-the-cmp), so my question is how to get these cookies (and/or other necessary cookies) from the website after accepting a consent pop-up, so I can login.
Here is my code so far:
import requests
# Website
base_url = 'https://www.wg-gesucht.de'
# Login URL
login_url = 'https://www.wg-gesucht.de/ajax/sessions.php?action=login'
# Post headers (just a sample of all variables)
headers = {...,
'Cookie': 'euconsent-v2=********'}
# Post params
payload = {'display_language': "de",
'login_email_username': "******",
'login_form_auto_login': "1",
'login_password': "******"}
# Setup session and login
sess = requests.session()
resp_login = sess.post(login_url, data=payload, headers=headers)
UPDATE: I have searched through all recorded requests from starting up the website to login and the only mention of euconsent-v2 is in the response of this:
cookie_url = 'https://cdn.consentmanager.mgr.consensu.org/delivery/cmp_en.min.js'
referer = 'https://www.wg-gesucht.de'
headers = {'Referer': referer,
'User-Agent': 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.66 Safari/537.36'}
sess = requests.session()
resp_init = sess.get(cookie_url, headers=headers)
But I still cannot get the required cookies
The best way would be creating a session, then requesting all the sites that set the cookie that you need. Then with all the cookies in the session you make, you request the login page.
https://help.consentmanager.net/books/cmp/page/cookies-set-by-the-cmp
On the right hand side there is the location.
The image shown below, is just an example of what I mean. Its a random site/url that on the response header, it sets two cookies. A session will save all the cookies and then when you have all the mandatory ones, you make a request to the login page with post data.
So I am trying to log in to a system using Python Requests module. When I do the post request:
r = requests.post(url, proxies=proxies, headers=headers, cookies=cookies, data=payload, verify=False)
print(r.status_code)
print(r.cookies)
and I try to print out the cookies, it shows me none. However, in the Burp Suite, I can see that it gets the response with the cookies I need, but it automatically asks for a landing page of the website and for some other stuff (basically it performs 2 more requests using the cookies I got in the first response). So when I ask for the cookies, it shows me cookies of the last response (obviously there are not any).
How can I get cookies from the first request?
A solution could be to set the flag allow_redirects=False and check the response for cookies and then follow the redirect manually. Given that it is a redirect that initiates the next request.
I am writing a python program which will send a post request with a password, if the password is correct, the server will return a special cookie "BDCLND".
I did this in Postman first. You can see the url, headers, the password I used and the response cookies in the snapshots below.
The response cookie didn't have the "BDCLND" cookie because the password 'ssss' was wrong. However, the server did send a 'BAIDUID' cookie back, now, if I sent another post request with the 'BAIDUID' cookie and the correct password 'v0vb', the "BDCLND" cookie would show up in the response. Like this:
Then I wrote the python program like this:
import requests
import string
import re
import sys
def main():
url = "https://pan.baidu.com/share/verify?surl=pK753kf&t=1508812575130&bdstoken=null&channel=chunlei&clienttype=0&web=1&app_id=250528&logid=MTUwODgxMjU3NTEzMTAuMzM2MTI4Njk5ODczMDUxNw=="
headers = {
"Content-Type":"application/x-www-form-urlencoded; charset=UTF-8",
"Referer":"https://pan.baidu.com/share/init?surl=pK753kf"
}
password={'pwd': 'v0vb'}
response = requests.post(url=url, data=password, headers=headers)
cookieJar = response.cookies
for cookie in cookieJar:
print(cookie.name)
response = requests.post(url=url, data=password, headers=headers, cookies=cookieJar)
cookieJar = response.cookies
for cookie in cookieJar:
print(cookie.name)
main()
When I run this, the first forloop did print up "BAIDUID", so that part is good, However, the second forloop printed nothing, it turned out the second cookiejar was just empty. I am not sure what I did wrong here. Please help.
Your second response has no cookies because you set the request cookies manually in the cookies parameter, so the server won't send a 'Set-Cookie' header.
Passing cookies across requests with the cookies parameter is not a good idea, use a Session object instead.
import requests
def main():
ses = requests.Session()
ses.headers['User-Agent'] = 'Mozilla/5'
url = "https://pan.baidu.com/share/verify?surl=pK753kf&t=1508812575130&bdstoken=null&channel=chunlei&clienttype=0&web=1&app_id=250528&logid=MTUwODgxMjU3NTEzMTAuMzM2MTI4Njk5ODczMDUxNw=="
ref = "https://pan.baidu.com/share/init?surl=pK753kf"
headers = {"Referer":ref}
password={'pwd': 'v0vb'}
response = ses.get(ref)
cookieJar = ses.cookies
for cookie in cookieJar:
print(cookie.name)
response = ses.post(url, data=password, headers=headers)
cookieJar = ses.cookies
for cookie in cookieJar:
print(cookie.name)
main()
I'm writing code trying to get Cookies from webserver using Request module, but what confused me is that:
I'm seeing Cookies returned when I tested using PostMan - REST Client
I didn't sent any cookies in my sent request, but what surprised me is that could find the cookie I want in Sent requests.
I want to get cookie data that I could use in code to request another application.
Following is my code:
import requests
import urllib3.contrib.pyopenssl
urllib3.contrib.pyopenssl.inject_into_urllib3()
username = 'user123' # real username/password not showing
password = '1234567'
login_data = {'id':username, 'pass_word':password, 'action': 'login'}
r = requests.post("www.example.com/login/", data=login_data)
print r.cookies
print r.request.header['Cookie']
Output:
<<class 'requests.cookies.RequestsCookieJar'>[]> # why nothing??
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'User-Agent': 'python-requests/2.6.0 CPython/2.7.6 Darwin/14.1.0', 'Connection': 'keep-alive', 'Cookie': 'blahblahblahblah', 'Content-Type': 'application/x-www-form-urlencoded'}
For confidential reason, I cannot show my real cookies data here, but I do see it's in my send request, how come? I didn't tell it to send it in my request, and this is the data that i expect to get from response cookie, but it showed as none.
The r.cookies jar contains all cookies that are new, that the server has sent in the current response, so any cookies that have been sent will not appear there unless the server resent them. The sent cookies will appear in r.request.headers (the jar is at r.request._cookies). If you want to preserve the cookies across multiple requests, use a Session:
session = requests.Session()
session.get(...) # Your first request
session.get(...) # Following requests pass the cookies on
While using the session, you can retrieve any cookies by looking in session.cookies. I don't know why you saw cookies sent when you didn't, I'd have to see more code.
I'm writing an Ajax post with python's Request's library to a django backend
Code:
import requests
import json
import sys
URL = 'http://localhost:8000/'
client = requests.session()
client.get(URL)
csrftoken = client.cookies['csrftoken']
data = { 'file': "print \"It works!\"", 'fileName' : "JSONtest", 'fileExt':".py",'eDays':'99','eHours':'1', 'eMinutes':'1' }
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken}
r = requests.post(URL+"au", data=json.dumps(data), headers=headers)
Django gives me a 403 error stating that the CSRF token isn't set even though the request.META from csrf_failure() shows it is set. Is there something I'm missing or a stupid mistake I'm not catching?
I asked my friend and he figured out the problem, basically you have to send the cookies that django gives you every time you do a request.
corrected:
cookies = dict(client.cookies)
r = requests.post(URL+"au", data=json.dumps(data), headers=headers,cookies=cookies)
You need to pass the referer to the headers, from the django docs:
In addition, for HTTPS requests, strict referer checking is done by
CsrfViewMiddleware. This is necessary to address a Man-In-The-Middle
attack that is possible under HTTPS when using a session independent
nonce, due to the fact that HTTP ‘Set-Cookie’ headers are
(unfortunately) accepted by clients that are talking to a site under
HTTPS. (Referer checking is not done for HTTP requests because the
presence of the Referer header is not reliable enough under HTTP.)
so change this:
headers = {'Content-type': 'application/json', "X-CSRFToken":csrftoken, "Referer": URL}