I'm using urllib2.urlopen():
req = urllib2.Request('http://www.google.com')
resp = urllib2.urlopen(req)
print resp.info()
print resp.info()['set-cookie']
Date: Sat, 14 May 2011 01:24:12 GMT
Expires: -1
Cache-Control: private, max-age=0
Content-Type: text/html; charset=ISO-8859-1
Set-Cookie: PREF=ID=5ec78624283cc050:FF=0:TM=1305336252:LM=1305336252:S=eRXgUUuzhQbRmZxk; expires=Mon, 13-May-2013 01:24:12 GMT; path=/; domain=.google.com
Set-Cookie: NID=46=GxyZVeWbT9dn0sLa9waPGSusm1hFqGf46SPqewahg0bzbYIQX0oHff0bzJ33E2yO89npEsYkqSoX0HLSqHbCxj5tCK2E931PfEJbqDMB6lTDk4ngVAiiyObWmbHgRUC9; expires=Sun, 13-Nov-2011 01:24:12 GMT; path=/; domain=.google.com; HttpOnly
Server: gws
X-XSS-Protection: 1; mod
PREF=ID=5ec78624283cc050:FF=0:TM=1305336252:LM=1305336252:S=eRXgUUuzhQbRmZxk; expires=Mon, 13-May-2013 01:24:12 GMT; path=/; domain=.google.com, NID=46=GxyZVeWbT9dn0sLa9waPGSusm1hFqGf46SPqewahg0bzbYIQX0oHff0bzJ33E2yO89npEsYkqSoX0HLSqHbCxj5tCK2E931PfEJbqDMB6lTDk4ngVAiiyObWmbHgRUC9; expires=Sun, 13-Nov-2011 01:24:12 GMT; path=/; domain=.google.com; HttpOnly
As you can see in the headers received in the response, there are TWO statements of 'set-cookie', HOWEVER in the resp.info() object I receive it has grouped both cookie statements together and separates them by a ',' (comma)
This is troublesome to separate the cookies by this delimiter since there are commas inside the cookie information i'm try to separate with this comma delimiter
Is there an easy way to call upon each cookie string individually with this mimetools.message object? (resp.info())
else-> I'll just have to parse the headers manually without this not so helpful mimetools.message/dictionary object
Try using getheaders() to get a list of the cookies:
>>> msg = resp.info()
>>> msg.getheaders('Set-Cookie')
['PREF=ID=5975a5ee255f0949:FF=0:TM=1305336283:LM=1305336283:S=1vkES6eF4Yxd-_oM; expires=Mon, 13-May-2013 01:24:43 GMT; path=/; domain=.google.com.au', 'NID=46=lQVFZg6yKUsoWT529Hqp5gA8B_CKYd2epPIbANmw_J0UzeMt2BhuMF-gtmGsRhenUTeajKz2zILXd9xWpHWT8ZGvDcmNdkzaGX-L_-sKyY1w4e2l3DKd80JzSkt2Vp-H; expires=Sun, 13-Nov-2011 01:24:43 GMT; path=/; domain=.google.com.au; HttpOnly']
In this case, you get a list of two strings.
Then you can iterate over that list and grab whichever cookie you like. str.startswith() is your friend:
>>> cookies = msg.getheaders('Set-Cookie')
>>> for cookie in cookies:
... if cookie.startswith('PREF='):
... print 'Got PREF: ', cookie
... else:
... print 'Got another: ', cookie
...
Got PREF: PREF=ID=5975a5ee255f0949:FF=0:TM=1305336283:LM=1305336283:S=1vkES6eF4Yxd-_oM; expires=Mon, 13-May-2013 01:24:43 GMT; path=/; domain=.google.com.au
Got another: NID=46=lQVFZg6yKUsoWT529Hqp5gA8B_CKYd2epPIbANmw_J0UzeMt2BhuMF-gtmGsRhenUTeajKz2zILXd9xWpHWT8ZGvDcmNdkzaGX-L_-sKyY1w4e2l3DKd80JzSkt2Vp-H; expires=Sun, 13-Nov-2011 01:24:43 GMT; path=/; domain=.google.com.au; HttpOnly
How a newbie can find the documentation in Python
% python
Python 2.7.1 (r271:86832, Jan 29 2011, 13:30:16)
[GCC 4.2.1 (Apple Inc. build 5664)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import urllib2
>>> req = urllib2.Request('http://www.google.com')
>>> resp = urllib2.urlopen(req)
>>> help(resp.info())
Related
In AWS docs it says, you can set cookie to their domain like this:
Set-Cookie: CloudFront-Policy=eyJTdGF0ZW1lbnQiOlt7IlJlc291cmNlIjoiaHR0cDovL2QxMTExMTFhYmNkZWY4LmNsb3VkZnJvbnQubmV0L2dhbWVfZG93bmxvYWQuemlwIiwiQ29uZGl0aW9uIjp7IklwQWRkcmVzcyI6eyJBV1M6U291cmNlSXAiOiIxOTIuMC4yLjAvMjQifSwiRGF0ZUxlc3NUaGFuIjp7IkFXUzpFcG9jaFRpbWUiOjE0MjY1MDAwMDB9fX1dfQ__; Domain=d111111abcdef8.cloudfront.net; Path=/; Secure; HttpOnly
Set-Cookie: CloudFront-Signature=dtKhpJ3aUYxqDIwepczPiDb9NXQ_; Domain=d111111abcdef8.cloudfront.net; Path=/; Secure; HttpOnly
Set-Cookie: CloudFront-Key-Pair-Id=K2JCJMDEHXQW5F; Domain=d111111abcdef8.cloudfront.net; Path=/; Secure; HttpOnly
but my django code:
response.set_cookie(
'CloudFront-Policy',
'eyJTdGF0ZW1lbnQ...',
domain=settings.AWS_CLOUDFRONT_DOMAIN,
secure=True,
httponly=True
)
not setting the cookie to their domain. I know it is security issue, but why AWS mentions this in documentation? any help appreciated.
okay, at the end, the solution was to use cname with altername domain. :/ turned out to be the right way anyway.
I need to automate connecting to logs analytics on Azure. Before I can do that, I need to get an access token.
With different docs and https://www.youtube.com/watch?v=ujzrq8Fg9Gc, I am trying to set that up.
TRY1
My first try was using SoapUI to send a POST request to:
https://login.microsoftonline.com/MY TENANT ID/oauth2/token
?grant_type=client_credentials
&client_id=MY CLIENT ID
&redirect_uri=MY URL
&resource=https%3A%2F%2Fwestus2.api.loganalytics.io
&client_secret=MY CLIENT SECRET
With header:
Content-Type: application/x-www-form-urlencoded
I always get this response:
HTTP/1.1 400 Bad Request
Cache-Control: no-cache, no-store
Pragma: no-cache
Content-Type: application/json; charset=utf-8
Expires: -1
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
x-ms-request-id: SOMETHING
P3P: CP="DSP CUR OTPi IND OTRi ONL FIN"
Set-Cookie: fpc=SOMETHING; expires=Mon, 05-Aug-2019 13:14:50 GMT; path=/; secure; HttpOnly
Set-Cookie: x-ms-gateway-slice=prod; path=/; secure; HttpOnly
Set-Cookie: stsservicecookie=ests; path=/; secure; HttpOnly
Date: Sat, 06 Jul 2019 13:14:49 GMT
Content-Length: 437
{
"error":"invalid_request",
"error_description":"AADSTS900144: The request body must contain
the following parameter: 'grant_type'.\r\n
Trace ID: SOMETHING\r\n
Correlation ID: SOMETHING\r\n
Timestamp: 2019-07-06 13:14:50Z",
"error_codes":[900144],
"timestamp":"2019-07-06 13:14:50Z",
"trace_id":"SOMETHING",
"correlation_id":"SOMETHING"
}
TRY 2
I wrote it in Python using import requests, like so:
import os,sys
import requests
Azure_client_id = 'MY CLIENT ID'
Azure_redirect_uri = 'MY URL'
Azure_client_secret = 'CLIENT SECRET'
Azure_tenant_id = 'TENANT ID'
Azure_resource = 'https://westus2.api.loganalytics.io'
###############################################################################
token_url = 'https://login.microsoftonline.com/' + Azure_tenant_id + '/oauth2/token'
token_headers = {
'Content-type': 'application/x-www-form-urlencoded',
}
token_params = {
'grant_type': 'client_credentials',
'client_id': Azure_client_id,
'redirect_uri': Azure_redirect_uri,
'resource': Azure_resource,
'client_secret': Azure_client_secret,
}
token_response = requests.post(token_url, headers=token_headers, params=token_params)
# This is to see what was sent
print(token_response.url + "\n\n")
# get the response and print it
token_result = ''
for chunk in token_response.iter_content(chunk_size=128):
token_result = token_result + str(chunk)
print(token_result.replace("\\n","\n"))
The URL that was sent is this (formatted for readability):
https://login.microsoftonline.com/MY TENANT ID/oauth2/token
?grant_type=client_credentials
&client_id=MY CLIENT ID
&redirect_uri=https%3A%2F%2FMY URL
&resource=https%3A%2F%2Fwestus2.api.loganalytics.io
&client_secret=MY SECRET URL ENCODED
The response I get is this (reformatted for readability):
b'{"error":"invalid_request",
"error_description":"AADSTS900144: The request body must contain
the following parameter: \'grant_type'b"'.\\r\
Trace ID: SOMETHING\\r\
Correlation ID: SOMETHING\\r\
Timestamp: 2019-"b'07-06 13:49:59Z",
"error_codes":[900144],
"timestamp":"2019-07-06 13:49:59Z",
"trace_id":"SOMETHING",
"co'b'rrelation_id":"SOMETHING"}''
At least I get the same error (!). Since my requests clearly include a "grant_type" parameter, my guess is that there is either something wrong with the encoding (which is done by SoapUI and Python's requests), something wrong with my URL, or I am not using the proper IDs.
Is there a way in Azure to validate that my client secret is valid? Once created, it cannot be read anymore. And someone else created this key so I cannot assume what he gave me is ok.
Any comment, hints, pointing out blatant mistakes on my part are appreciated.
Change
token_response = requests.post(token_url, headers=token_headers, params=token_params)
to
token_response = requests.post(token_url, data=token_params)
You don't need to specify the Content-type header, it's inferred from your payload (dictionary, so x-www-form-urlencoded), also data is what you want (payload), not params (URL parameters).
Your request should look like this on the wire -
POST /TENANT_ID/oauth2/token HTTP/1.1
Host: localhost:9000
User-Agent: python-requests/2.22.0
Accept-Encoding: gzip, deflate
Accept: */*
Connection: keep-alive
Content-Length: 151
Content-Type: application/x-www-form-urlencoded
grant_type=client_credentials
&client_id=MY+CLIENT+ID
&redirect_uri=MY+URL
&resource=https%3A%2F%2Fwestus2.api.loganalytics.io
&client_secret=CLIENT+SECRET
Everything is in the body where it should be for x-www-form-urlencoded.
More on x-www-form-urlencoded here -
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/POST#Example
I am trying to write a client to some web app. At one point it sends the following Set-cookie:
JSESSIONID=1BDC39CBF91C299C3330963D1EEFE399; Path=; HttpOnly; Secure, JSESSIONID=E6FFF3B159AFB9575D47662FC70DC161; Path=/; Secure; HttpOnly, XSESSIONID=b163abe6-bd6c-4381-9f68-01eeaee15a6c; Path=/; Secure; HttpOnly
It is unclear to me what is the meaning of having the same cookie name twice and what I am supposed to send back to the server?
I checked it with the following python code:
import sys
if sys.version_info.major == 2:
from Cookie import SimpleCookie
else:
from http.cookies import SimpleCookie
set_cookie = "JSESSIONID=1BDC39CBF91C299C3330963D1EEFE399; Path=; HttpOnly; Secure, JSESSIONID=E6FFF3B159AFB9575D47662FC70DC161; Path=/; Secure; HttpOnly, XSESSIONID=b163abe6-bd6c-4381-9f68-01eeaee15a6c; Path=/; Secure; HttpOnly"
cookies = SimpleCookie()
cookies.load(set_cookie)
for name in cookies.keys():
print("{} = {}".format(name, cookies[name].value))
for field in ['secure', 'httponly', 'path']:
print(" {}: {}".format(field, cookies[name][field]))
The python 2 code shows only one of the duplicate keys. The python 3 version does not recognize this at all.
$ /usr/bin/python cookie.py
XSESSIONID = b163abe6-bd6c-4381-9f68-01eeaee15a6c
secure: True
httponly: True
path: /
JSESSIONID = E6FFF3B159AFB9575D47662FC70DC161
secure: True
httponly: True
path: /
$ python3 cookie.py
So I would like to understand what is the meaning of having the same key twice and what should be sent back to the server?
It would be also nice to understand why does the Python 3 library disregard the whole string. What do I need to do to fix it?
I just want a better idea of what's going on here, I can of course "work around" the problem by using urllib2.
import urllib
import urllib2
url = "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html"
# urllib2 works fine (foo.headers / foo.read() also behave)
foo = urllib2.urlopen(url)
# urllib throws errors though, what specifically is causing this?
bar = urllib.urlopen(url)
http://pae.st/AxDW/ shows this code in action with the exception/stacktrace. foo.headers and foo.read() work fine
stu#sente.cc ~ $: curl -I "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html"
HTTP/1.1 302 Object Moved
Cache-Control: private
Transfer-Encoding: chunked
Content-Type: text/html; charset=utf-8
Location: /S-FSTWJcduy5w/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html
Server: Microsoft-IIS/7.5
Set-Cookie: SESSIONID=FSTWJcduy5w; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/
Set-Cookie: SYSTEMID=0; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/
Set-Cookie: SESSIONDATE=02/23/2012 17:07:00; domain=.crutchfield.com; expires=Fri, 22-Feb-2013 22:06:43 GMT; path=/
X-AspNet-Version: 4.0.30319
HostName: cws105
Date: Thu, 23 Feb 2012 22:06:43 GMT
Thanks.
This server is both non-deterministic and sensitive to HTTP version. urllib2 is HTTP/1.1, urllib is HTTP/1.0. You can reproduce this by running curl --http1.0 -I "http://www.crutchfield.com/S-pqvJFyfA8KG/p_15410415/Dynamat-10415-Xtreme-Speaker-Kit.html"
a few times in a row. You should see the output curl: (52) Empty reply from server occasionally; that's the error urllib is reporting. (If you re-issue the request a bunch of times with urllib, it should succeed sometimes.)
I solved the Problem. I simply using now the urrlib instead of urllib2 and anything works fine thank you all :)
When i use python SimpleCookie object to pick up cookie from http headers, some exception occurs:
cookiestr = "a_em=[BU]co|12345678-901234567[DG]; Expires=Sat, 31 Dec 2016 17:09:50 GMT; Domain=.somesite.com; Path=/"
C = Cookie.SimpleCookie()
C.load(cookiestr)
print C
the output is:
Set-Cookie: a_em=; Domain=.somesite.com; expires=Sat,; Path=/
the cookie value and the cookie expires time is error!
how should i solve this?
RFC format for expires should be:
Expires=Sat, 31-Dec-2016 17:09:50 GMT
The full string should be (note quotes)
cookiestr = 'a_em="[BU]co|12345678-901234567[DG]"; Expires=Sat, 31-Dec-2016 17:09:50 GMT; Domain=.somesite.com; Path=/'