I am doing a simple get request to get a website cookies:
import requests
with requests.Session() as s:
session = requests.Session()
response = session.get("http://www.dailymail.co.uk/home/index.html")
ncooks = session.cookies.get_dict()
print(ncooks)
But, when the ncooks gets returned it is empty {}
Why is this? How can I solve this problem to get the cookies for the website?
Your code works, the website you're using just doesn't set a cookie. Try with http://nytimes.com, the dict won't be empty.
Unrelated, but you're creating an unnecessary extra Session object; it should look like this:
import requests
with requests.Session() as s:
response = s.get("http://nytimes.com")
cooks = s.cookies.get_dict()
print(cooks)
Related
I want to start a requests.Session() and add a cookie before starting the first request. I expected to have a cookie argument or something similar to do this
def session_start()
self.session = requests.Session(cookies=['session-id', 'xxx'])
def req1():
self.session.get('example.org')
def req2():
self.session.get('example2.org')
but this wont work, I only can provide cookies in the .get() method. Do I need to do a "dummy request" in session_start() or is there a way to prepare the cookie before starting the actual request?
From the documentation:
Note, however, that method-level parameters will not be persisted across requests, even if using a session. This example will only send the cookies with the first request, but not the second:
s = requests.Session()
r = s.get('https://httpbin.org/cookies', cookies={'from-my': 'browser'})
print(r.text)
# '{"cookies": {"from-my": "browser"}}'
r = s.get('https://httpbin.org/cookies')
print(r.text)
# '{"cookies": {}}'
session.cookies is a RequestsCookieJar and you can add cookies to that after constructing the session:
self.session = requests.Session()
self.session.cookies.set('session-id', 'xxx')
So session objects will persist any cookies that the url requests themselves set, but if you provide a cookie as an argument it will not persist on the next request.
You can add cookies in manually that persist too though, from the documentation:
"If you want to manually add cookies to your session, use the Cookie utility functions to manipulate Session.cookies."
https://requests.readthedocs.io/en/master/user/advanced/#session-objects
Where it links to on how to manipulate cookies: https://requests.readthedocs.io/en/master/api/#api-cookies
This is what I have so far. I'm very new to this so point out if I'm doing anything wrong.
import requests
url = "https://9anime.to/user/watchlist"
payload = {
"username":"blahblahblah",
"password":"secretstringy"
# anything else?
}
with requests.Session() as s:
res = s.post(url, data=payload)
print(res.status_code)
You will need to inspect the form and see where the form is posting to in the action tag. In this case it is posting to user/ajax/login. Instead of requesting the watchlist URL you should post those details to the loginurl. Once you are logged in you can request your watchlist.
from lxml.html import fromstring
import requests
url = "https://9anime.to/user/watchlist"
loginurl = "https://9anime.to/user/ajax/login"
payload = {
"username":"someemail#gmail.com",
"password":"somepass"
}
with requests.Session() as s:
res = s.post(loginurl, data=payload)
print(res.content)
# b'{"success":true,"message":"Login successful"}'
res = s.get(url)
tree = fromstring(res.content)
elem = tree.cssselect("div.watchlist div.widget-body")[0]
print(elem.text_content())
# Your watch list is empty.
You would need to have knowledge (documentation of some form) on what that URL is expecting and how you are expected to interact with it. There is no way to know just given the information you have provided.
If you have some system that is able to interact with that URL already (e.g. your able to log in with your browser), then you could try to reverse-engineer what it is your browser is doing...
I'm trying to implement the following working code:
import requests
session = requests.Session()
jar = requests.cookies.RequestsCookieJar()
jar.set('key', 'value')
session.cookies = jar
r = session.get('https://example.com/user/info')
print(r.text)
print(r.status_code)
I am trying to get user info using above code. I logged in my account from browser and have this Cookie but I dont know which key/value is supposed to be used to get user info. The code is working for other website with right Cookie. Any ideas about choicing right key/value?
When you inspect and find Cookies you will see like:
Cookie: sess-is='123456'; other1-key=other1-val; other2-key=other2-val; otherN-
key=otherN-val;
most browser keep cookie in this format. In your case:
import requests
session = requests.Session()
jar = requests.cookies.RequestsCookieJar()
jar.set('Cookie', 'sess-is='123456'; other1-key=other1-val; other2-key=other2-val;
otherN-key=otherN-val;')
session.cookies = jar
r = session.get('https://example.com/user/info')
print(r.text)
print(r.status_code)
Have a great web scraping :)
My propose is to login at my application through python requests. I was able to get a token, that is expected, but passing it by GET isn't enough. So, i want to store the request in a cookie, pass the token, and maybe the browser can login.
So, let's resume what i did (this is pseudo code)
session = requests.Session()
session.get('<url>salt')
r = session.get('<url>login', params={username, password})
r.headers['token']
I discovered this by looking the requests while login. The token is passed to the application after. So, how can i store the "r" as a cookie?
you can simply access your session cookie using:
client = requests.session()
cook = client.cookies
extracted_token_value = client.cookies['token']
#this will print your cookie and token
print cook.text
print extracted_token_value
#updating your header now:
client.headers.update({'New Header': 'extracted_token_value')
BR
I'm trying to get the sessionId, so i can do other requests.
So i looked in the Firefox Network monitor (Ctrl+Shift+Q) and saw this:
So i wondered how i could do the request in python 3 and tried things like this:
import requests
payload = {'uid' : 'username',
'pwd' : 'password'}
r = requests.get(r'http://192.168.2.114(cgi-bin/wwwugw.cgi', data=payload)
print r.text
But I always get "Response [400]".
If the request is correct, I should get something like this:
Thanks
Alex
Just use a session, which will handle redirects and cookies for you:
import requests
payload = {'uid' : 'username',
'pwd' : 'password'}
with requests.Session() as session:
r = session.post(r'http://192.168.2.114(cgi-bin/wwwugw.cgi', data=payload)
print(r.json)
This way you don't explicitly need to get the sessionId, but if you still want to, you can access the returned JSON as a dictionary.
if you want to get the Session ID you can use Session() from the requests library:
URL = "Some URL here"
client = requests.Session()
client.get(URL)
client.cookies['sessionid']
Although it's not very clear from your question, but I've noticed few issues with what you are trying to accomplish.
If you are using session authentication, then you are suppose the send session_id as a Cookie header, which you aren't doing.
400 response code means bad request, not authentication required. Why are you sending data in get request to begin with? There's a difference between data and query params.