Is there a way to get the headers from a request created with Urllib2 or to confirm the HTTP headers sent with urllib2.urlopen?
An easy way to see request (and response headers) is to enable debug output:
opener = urllib2.build_opener(urllib2.HTTPHandler(debuglevel=1))
You then can see the precise headers sent/recieved:
>>> opener.open('http://python.org')
send: 'GET / HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: python.org\r\nConnection: close\r\nUser-Agent: Python-urllib/2.7\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Date: Tue, 14 Jun 2011 08:23:35 GMT
header: Server: Apache/2.2.16 (Debian)
header: Last-Modified: Mon, 13 Jun 2011 19:41:35 GMT
header: ETag: "105800d-486d-4a59d1b6699c0"
header: Accept-Ranges: bytes
header: Content-Length: 18541
header: Connection: close
header: Content-Type: text/html
header: X-Pad: avoid browser bug
<addinfourl at 140175550177224 whose fp = <socket._fileobject object at 0x7f7d29c3d5d0>>
You can also set with the urllib2.Request objects headers before making the request (and override the default headers, although won't be present in the headers dict beforehand):
>>> req = urllib2.Request(url='http://python.org')
>>> req.add_header('User-Agent', 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0)')
>>> req.headers
{'User-agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10.6; rv:5.0)'}
Related
I'm trying to download a file from a website using Python's request module.
However the site will allow me to download the file only if the download link is clicked directly from the download page.
So using requests, I tried hitting the download page's URL first using requests.get() then proceeding to download the file. But unfortunately this doesn't seem to work. A text asking me to open the download page first simply gets written into file.torrent"
import requests
def download(username, password):
with requests.Session() as session:
session.post('https://website.net/forum/login.php', data={'login_username': username, 'login_password': password})
# Download page URL
requests.get('https://website.net/forum/viewtopic.php?t=2508126')
# The download URL itself
response = requests.get('https://website.net/forum/dl.php?t=2508126')
with open('file.torrent', 'wb') as f:
f.write(response.content)
download(username='XXXXX', password='YYYYY')
Response when downloading directly from the download page (works) :
General :
Request URL: https://website.net/forum/dl.php?t=2508126
Request Method: GET
Status Code: 200 OK
Remote Address: 185.37.128.136:443
Referrer Policy: no-referrer-when-downgrade
Response Headers :
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Content-Disposition: attachment; filename="[website.net].t2508126.torrent"
Content-Length: 33641
Content-Type: application/x-bittorrent; name="[website.net].t2508126.torrent"
Date: Thu, 14 Feb 2019 07:57:08 GMT
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Thu, 14 Feb 2019 07:57:09 GMT
Pragma: no-cache
Server: nginx
Set-Cookie: bb_dl=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/forum/; domain=.website.net
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: bb_t=a%3A3%3A%7Bi%3A2507902%3Bi%3A1550052944%3Bi%3A2508011%3Bi%3A1550120230%3Bi%3A2508126%3Bi%3A1550125516%3B%7D; bb_data=1-27969311-wXVPJGcedLE1I2mM9H0u-3106784170-1550128652-1550131012-3061288864-1; bb_dl=2508126
Host: website.net
Referer: https://website.net/forum/viewtopic.php?t=2508126
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3701.0 Safari/537.36
Query String Parameters :
t: 2508126
Response when opening the download link on it's own (doesn't work) :
General :
Request URL: https://website.net/forum/dl.php?t=2508126
Request Method: GET
Status Code: 200 OK
Remote Address: 185.37.128.136:443
Referrer Policy: no-referrer-when-downgrade
Response Headers :
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Content-Type: text/html; charset=windows-1251
Date: Thu, 14 Feb 2019 08:03:29 GMT
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Thu, 14 Feb 2019 08:03:29 GMT
Pragma: no-cache
Server: nginx
Transfer-Encoding: chunked
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: bb_t=a%3A3%3A%7Bi%3A2507902%3Bi%3A1550052944%3Bi%3A2508011%3Bi%3A1550120230%3Bi%3A2508126%3Bi%3A1550125516%3B%7D; bb_data=1-27969311-wXVPJGcedLE1I2mM9H0u-3106784170-1550128652-1550131390-3061288864-1
Host: website.net
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3701.0 Safari/537.36
Query String Parameters :
t: 2508126
This works for me:
data={'login_username': username, 'login_password': password, 'login': ''}
and using session.get() instead of requests.get()
I am using ntlm to access an internal server that uses windows authentication. The url that I am trying to access keeps redirecting. Here is my code:
import urllib2
from ntlm import HTTPNtlmAuthHandler
import cookielib
user = r'Domain\username'
password = "password"
url = r"http://cmsll.jvservices.com/Livelink/"
passman = urllib2.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, user, password)
# create the NTLM authentication handler
auth_NTLM = HTTPNtlmAuthHandler.HTTPNtlmAuthHandler(passman, debuglevel=1)
cookieJar = cookielib.CookieJar()
# create and install the opener
opener = urllib2.build_opener(auth_NTLM, urllib2.HTTPCookieProcessor(cookieJar))
urllib2.install_opener(opener)
url = r"http://cmsll.jvservices.com/Livelink/livelink.exe?func=ll&objId=87167&objAction=runReport&inputLabel1_ID=118163&inputLabel1_Name=%22Lastname%2C+Firstname+%28domain\username%29%22&inputLabel2=D%2F2013%2F5%2F21%3A0%3A0%3A0&inputLabel2_dirtyFlag=1&inputLabel2_month=5&inputLabel2_day=21&inputLabel2_year=2013&inputLabel2_hour=13&inputLabel2_minute=53&inputLabel2_second=0&inputLabel2_ampm=0&inputLabel3=D%2F2014%2F5%2F21%3A0%3A0%3A0&inputLabel3_dirtyFlag=0&inputLabel3_month=5&inputLabel3_day=21&inputLabel3_year=2014&inputLabel3_hour=0&inputLabel3_minute=0&inputLabel3_second=0&inputLabel3_ampm=0>"
# retrieve the result
req = urllib2.Request(url)
req.add_header('User-Agent', 'Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 FirePHP/0.7.4')
response = urllib2.urlopen(req)
print(response.read())
Here is the output:
send: 'GET /Livelink/livelink.exe?func=ll&objId=87167&objAction=runReport&inputLabel1_ID=118163&inputLabel1_Name=%22Lastname%2C+Firstname+%28domain\\username%29%22&inputLabel2=D%2F2013%2F5%2F21%3A0%3A0%3A0&inputLabel2_dirtyFlag=1&inputLabel2_month=5&inputLabel2_day=21&inputLabel2_year=2013&inputLabel2_hour=13&inputLabel2_minute=53&inputLabel2_second=0&inputLabel2_ampm=0&inputLabel3=D%2F2014%2F5%2F21%3A0%3A0%3A0&inputLabel3_dirtyFlag=0&inputLabel3_month=5&inputLabel3_day=21&inputLabel3_year=2014&inputLabel3_hour=0&inputLabel3_minute=0&inputLabel3_second=0&inputLabel3_ampm=0 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: cmsll.jvservices.com\r\nConnection: Keep-Alive\r\nAuthorization: <stuff here>\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 FirePHP/0.7.4\r\n\r\n'
reply: 'HTTP/1.1 401 Unauthorized\r\n'
header: Content-Length: 1539
header: Content-Type: text/html
header: Server: Microsoft-IIS/6.0
header: WWW-Authenticate: <stuff here>
header: X-Powered-By: ASP.NET
header: Date: Thu, 22 May 2014 14:16:38 GMT
send: 'GET /Livelink/livelink.exe?func=ll&objId=87167&objAction=runReport&inputLabel1_ID=118163&inputLabel1_Name=%22Lastname%2C+Firstname+%28domain\\username%29%22&inputLabel2=D%2F2013%2F5%2F21%3A0%3A0%3A0&inputLabel2_dirtyFlag=1&inputLabel2_month=5&inputLabel2_day=21&inputLabel2_year=2013&inputLabel2_hour=13&inputLabel2_minute=53&inputLabel2_second=0&inputLabel2_ampm=0&inputLabel3=D%2F2014%2F5%2F21%3A0%3A0%3A0&inputLabel3_dirtyFlag=0&inputLabel3_month=5&inputLabel3_day=21&inputLabel3_year=2014&inputLabel3_hour=0&inputLabel3_minute=0&inputLabel3_second=0&inputLabel3_ampm=0 HTTP/1.1\r\nAccept-Encoding: identity\r\nHost: cmsll.jvservices.com\r\nConnection: Close\r\nAuthorization: <stuff here>\r\nUser-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:29.0) Gecko/20100101 Firefox/29.0 FirePHP/0.7.4\r\n\r\n'
reply: 'HTTP/1.1 302 Redirect\r\n'
header: Content-Length: 895
header: Content-Type: text/html
header: Expires: -1
header: Location: http://cmsll.jvservices.com/Livelink/livelink.exe?func=ll.GetTZ&NextURL=%2FLivelink%2Flivelink%2Eexe%3Ffunc%3Dll%26objId%3D87167%26objAction%3DrunReport%26inputLabel1_ID%3D118163%26inputLabel1_Name%3D%2522Lastname%252C%2BFirstname%2B%2528domain%5Cu370471%2529%2522%26inputLabel2%3DD%252F2013%252F5%252F21%253A0%253A0%253A0%26inputLabel2_dirtyFlag%3D1%26inputLabel2_month%3D5%26inputLabel2_day%3D21%26inputLabel2_year%3D2013%26inputLabel2_hour%3D13%26inputLabel2_minute%3D53%26inputLabel2_second%3D0%26inputLabel2_ampm%3D0%26inputLabel3%3DD%252F2014%252F5%252F21%253A0%253A0%253A0%26inputLabel3_dirtyFlag%3D0%26inputLabel3_month%3D5%26inputLabel3_day%3D21%26inputLabel3_year%3D2014%26inputLabel3_hour%3D0%26inputLabel3_minute%3D0%26inputLabel3_second%3D0%26inputLabel3_ampm%3D0
header: Server: Microsoft-IIS/6.0
header: X-Powered-By: ASP.NET
header: Date: Thu, 22 May 2014 14:16:39 GMT
header: Connection: close
<head><title>Document Moved</title></head>
<body><h1>Object Moved</h1>This document may be found here</body>
The response html isn't what I'm looking for. I have tried following this redirect manually, and it gives me another redirect. What am I doing to cause it to redirect like this?
This is probably due to you having the "offset time zone" setting in the Admin pages. "GetTZ" means "Get Time Zone". To stop it, uncheck the box in "Modify Date and Times" on each server/installation instance. It's not a system-wide setting.
I am trying to write a script that will download a bunch files from a website that has REST URLs.
Here is the GET request:
GET /test/download/id/5774/format/testTitle HTTP/1.1
Host: testServer.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:23.0) Gecko/20100101 Firefox/23.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Cookie: __utma=11863783.1459862770.1379789243.1379789243.1379789243.1; __utmb=11863783.28.9.1379790533699; __utmc=11863783; __utmz=11863783.1379789243.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none); PHPSESSID=fa844952890e9091d968c541caa6965f; loginremember=Qraoz3j%2BoWXxwqcJkgW9%2BfGFR0SDFLi1FLS7YVAfvbcd9GhX8zjw4u6plYFTACsRruZM4n%2FpX50%2BsjXW5v8vykKw2XNL0Vqo5syZKSDFSSX9mTFNd5KLpJV%2FFlYkCY4oi7Qyw%3D%3D; ma-refresh-storage=1; ma-pref=KLSFKJSJSD897897; skipPostLogin=0; pp-sid=hlh6hs1pnvuh571arl59t5pao0; __utmv=11863783.|1=MemberType=Yearly=1; nats_cookie=http%253A%252F%252Fwww.testServer.com%252F; nats=NDc1NzAzOjQ5MzoyNA%2C74%2C0%2C0%2C0; nats_sess=fe3f77e6e326eb8d18ef0111ab6f322e; __utma=163815075.1459708390.1379790355.1379790355.1379790355.1; __utmb=163815075.1.9.1379790485255; __utmc=163815075; __utmz=163815075.1379790355.1.1.utmcsr=ppp.contentdef.com|utmccn=(referral)|utmcmd=referral|utmcct=/postlogin; unlockedNetworks=%5B%22rk%22%2C%22bz%22%2C%22wkd%22%5D
Connection: close
If the request is good, it will return a 302 response such as this one:
HTTP/1.1 302 Found
Date: Sat, 21 Sep 2013 19:32:37 GMT
Server: Apache
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
location: http://downloads.test.stuff.com/5774/stuff/picture.jpg?wed=20130921152237&wer=20130922153237&hash=0f20f4a6d0c9f1720b0b6
Vary: User-Agent,Accept-Encoding
Content-Length: 0
Connection: close
Content-Type: text/html; charset=UTF-8
What I need the script to do is check to see if it was a 302 response. If it is not, it will "pass", if it is, it will need to parse out the location parameter shown here:
location: http://downloads.test.stuff.com/5774/stuff/picture.jpg?wed=20130921152237&wer=20130922153237&hash=0f20f4a6d0c9f1720b0b6
Once I have the location parameter, I will have to make another GET request to download that file. I will also have to maintain the cookie for my session in order to download the file.
Can someone point me in the right direction for what library is best to use for this? I am having trouble finding out how to parse the 302 response and adding a cookie value like the one shown in my GET request above. I am sure there must be some library that can do all of this.
Any help would be much appreciated.
import urllib.request as ur
import urllib.error as ue
'''
Note that http.client.HTTPResponse.read([amt]) reads and returns the response body, or up to
the next amt bytes. This is because there is no way for urlopen() to automatically determine
the encoding of the byte stream it receives from the http server.
'''
url = "http://www.example.org/images/{}.jpg"
dst = ""
arr = ["01","02","03","04","05","06","07","08","09"]
# arr = range(10,20)
try:
for x in arr:
print(str(x)+"). ".ljust(4),end="")
hrio = ur.urlopen(url.format(x)) # HTTPResponse iterable object (returns the response header and body, together, as bytes)
fh = open(dst+str(x)+".jpg","b+w")
fh.write(hrio.read())
fh.close()
print("\t[REQUEST COMPLETE]\t\t<Error ~ [None]>")
except ue.URLError as e:
print("\t[REQUEST INCOMPLETE]\t",end="")
print("<Error ~ [{}]>".format(e))
Using python and mechanize im trying to login to web-site.
The code i'm using is:
from mechanize import Browser
def calOnline(uname,passwd):
br = Browser()
br.set_debug_http(True)
br.set_debug_redirects(True)
br.set_debug_responses(True)
br.set_handle_redirect(True)
br.set_handle_robots(False)
br.set_handle_refresh(True)
#~ br.encoding()
br.addheaders = [('User-Agent','Mozilla/5.0 (X11; Linux x86_64; rv:13.0) Gecko/20100101 Firefox/13.0.1'),
('Accept','text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'),
('Accept-Language','en-us,en;q=0.5'),
('Accept-Encoding','gzip, deflate'),
('Connection','keep-alive')]
print('opening site')
br.open('https://m.cal-online.co.il/')
print('\n\nsubmiting first form')
br.select_form(nr=0)
br.submit()
print('\n\nselecting login form')
br.select_form(nr=0)
br['ctl00$cphMain$LGN$UserName'] = uname
br['ctl00$cphMain$LGN$Password'] = passwd
print('\n\nsubmitting form')
br.submit(name='ctl00$cphMain$LGN$LoginButton')
print('\n\nOpening details page')
res = br.open('https://m.cal-online.co.il/SCREENS/Transactions/TrSearch.aspx')
print res.read()
uname = 'someuname'
passwd = 'somepasswd'
a = calOnline(uname, passwd)
print a.read()
The website is mobile version of credit card website from which im trying to get my expenses.
Browser (Android as mobile or Firefox/Chrome/Opera as desktop) opens the site and logs in without any issue.
While trying to login programatically it just stuck, here is debug log with iOS user-agent:
opening site
send: 'GET / HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: m.cal-online.co.il\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nConnection: close\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: Set-Cookie: JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449; Path=/; Secure
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Date: Fri, 29 Jun 2012 21:44:52 GMT
header: Cache-Control: no-cache, no-store, must-revalidate, no-transform
header: Pragma: no-cache
header: Expires: -1
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 302
header: Connection: close
send: 'GET /SCREENS/AccountManagement/HomePage.aspx HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: m.cal-online.co.il\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nConnection: close\r\nCookie: JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Location: https://m.cal-online.co.il/SCREENS/AccountManagement/Opening.aspx?ReturnUrl=%2fSCREENS%2fAccountManagement%2fHomePage.aspx
header: Content-Length: 0
header: Date: Fri, 29 Jun 2012 21:44:53 GMT
header: Connection: close
send: 'GET /SCREENS/AccountManagement/Opening.aspx?ReturnUrl=%2fSCREENS%2fAccountManagement%2fHomePage.aspx HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: m.cal-online.co.il\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nConnection: close\r\nCookie: JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Date: Fri, 29 Jun 2012 21:44:55 GMT
header: Cache-Control: private
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 1256
header: Connection: close
send: 'GET /SCREENS/AccountManagement/Login.aspx HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: m.cal-online.co.il\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nConnection: close\r\nCookie: JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\n\r\n'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Location: https://m.cal-online.co.il/SCREENS/AccountManagement/Login.aspx?cc=c009a07&rnd=2103197098
header: Set-Cookie: test_cookie=ok; expires=Sat, 30-Jun-2012 21:45:12 GMT; path=/
header: Content-Length: 0
header: Date: Fri, 29 Jun 2012 21:45:07 GMT
header: Connection: close
send: 'GET /SCREENS/AccountManagement/Login.aspx?cc=c009a07&rnd=2103197098 HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nHost: m.cal-online.co.il\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nConnection: close\r\nCookie: test_cookie=ok; JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Set-Cookie: ASP.NET_SessionId=clocqc55tdzykh45zql10045; path=/; HttpOnly
header: Date: Fri, 29 Jun 2012 21:45:09 GMT
header: Cache-Control: no-cache, no-store, must-revalidate, no-transform
header: Pragma: no-cache
header: Expires: -1
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 3153
header: Connection: close
submiting first form
send: 'POST /SCREENS/AccountManagement/Login.aspx?rnd=2103197098&cc=c009a07 HTTP/1.1\r\nContent-Length: 189\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nHost: m.cal-online.co.il\r\nReferer: https://m.cal-online.co.il/\r\nCookie: test_cookie=ok; ASP.NET_SessionId=clocqc55tdzykh45zql10045; JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n__VIEWSTATE=&ctl00%24__MATRIX_VIEWSTATE=1&ctl00%24cphMain%24LGN%24UserName=&ctl00%24cphMain%24LGN%24Password=&ctl00%24cphMain%24LGN%24LoginButton.x=1&ctl00%24cphMain%24LGN%24LoginButton.y=1'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Date: Fri, 29 Jun 2012 21:45:10 GMT
header: Cache-Control: no-cache, no-store, must-revalidate, no-transform
header: Pragma: no-cache
header: Expires: -1
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 3210
header: Connection: close
selecting login form
submitting form
send: 'POST /SCREENS/AccountManagement/Login.aspx?rnd=2103197098&cc=c009a07 HTTP/1.1\r\nContent-Length: 206\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nHost: m.cal-online.co.il\r\nReferer: https://m.cal-online.co.il/SCREENS/AccountManagement/Login.aspx?rnd=2103197098&cc=c009a07\r\nCookie: test_cookie=ok; ASP.NET_SessionId=clocqc55tdzykh45zql10045; JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\nContent-Type: application/x-www-form-urlencoded\r\n\r\n__VIEWSTATE=&ctl00%24__MATRIX_VIEWSTATE=2&ctl00%24cphMain%24LGN%24UserName=<uname>&ctl00%24cphMain%24LGN%24Password=<passwd>&ctl00%24cphMain%24LGN%24LoginButton.x=1&ctl00%24cphMain%24LGN%24LoginButton.y=1'
reply: 'HTTP/1.1 302 Moved Temporarily\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Location: https://m.cal-online.co.il/SCREENS/AccountManagement/HomePage.aspx
header: Set-Cookie: .ASPXAUTH=478FDDCD007398FEB264895D0F6EDB51B391DD0F5FBA3C71FC6A9E747AF3A97E6382E7B939614DFC07B25A1D4A641ED121F15508483A676AC49BAA550BEADF382F93792E849F63E99B03FA45143391ACD5E18CA7124FAC43AC378D16703DB5B2A374E4D1B3278BF9B886F3B4A41BB12E3569162D; path=/; HttpOnly
header: Content-Length: 0
header: Date: Fri, 29 Jun 2012 21:45:14 GMT
header: Connection: close
send: 'GET /SCREENS/AccountManagement/HomePage.aspx HTTP/1.1\r\nAccept-Language: en-us,en;q=0.5\r\nAccept-Encoding: gzip, deflate\r\nConnection: close\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8\r\nUser-Agent: Mozilla/5.0 (iPhone; U; CPU iPhone OS 3_0 like Mac OS X; en-us) AppleWebKit/528.18 (KHTML, like Gecko) Version/4.0 Mobile/7A341 Safari/528.16\r\nHost: m.cal-online.co.il\r\nCookie: test_cookie=ok; ASP.NET_SessionId=clocqc55tdzykh45zql10045; .ASPXAUTH=478FDDCD007398FEB264895D0F6EDB51B391DD0F5FBA3C71FC6A9E747AF3A97E6382E7B939614DFC07B25A1D4A641ED121F15508483A676AC49BAA550BEADF382F93792E849F63E99B03FA45143391ACD5E18CA7124FAC43AC378D16703DB5B2A374E4D1B3278BF9B886F3B4A41BB12E3569162D; JSESSIONID=E46452F3D9D4B7303C6E93F04BE54449\r\nReferer: https://m.cal-online.co.il/SCREENS/AccountManagement/Login.aspx?rnd=2103197098&cc=c009a07\r\n\r\n'
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: Apache-Coyote/1.1
header: X-Powered-By: ASP.NET
header: X-MA-MIS-Device: root^html^mozilla/5^safari^appleiphone^appleiphone(os_3.0)
header: Server: Microsoft-IIS/6.0
header: X-AspNet-Version: 2.0.50727
header: Date: Fri, 29 Jun 2012 21:45:16 GMT
header: Cache-Control: no-cache, no-store, must-revalidate, no-transform
header: Pragma: no-cache
header: Expires: -1
header: Content-Type: text/html;charset=utf-8
header: Content-Length: 5235
header: Connection: close
Same stuff happens with my Firefox User-agent. It just getting stuck on this moment till timeout on website which in place redirect me to LogOut page due to inactivity.
Is there something I'm missing out?
Thanks in advance
i need to create .ASPXAUTH cookie on python. i programing to desktop client. and first request not need .ASPXAUTH cookie but second request is need.
My First Request Headers:
User-Agent: WebPolicy
Host: xxx.host
Cache-Control: no-cache
My First Response Headers:
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: private
header: Transfer-Encoding: chunked
header: Content-Type: text/xml; charset=utf-8
header: Server: Microsoft-IIS/7.5
header: Set-Cookie: tivi_=3tnihi55ezuk50zyrrpuwv45; path=/; HttpOnly
header: X-AspNet-Version: 2.0.50727
header: X-Powered-By: ASP.NET
header: Date: Thu, 14 Oct 2010 13:05:50 GMT
And i need second send headers :
Accept: */*
Content-Length: 259
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://tempuri.org/IMiddlewareServices/Login"
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 1.1.4322)
Host: mw.webtv.ttnet.com.tr
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: .ASPXAUTH=090F2718E32AF3F9B1C9E5A15BA54CFD8D4430C44A91029719953D4A6D38DD1D9164D86D772E2645C0C0545A71C12EA80AE5A8F725FD6037BD00DB291A863DD577735E16D8745E2833979F337935F29A37C509FB0350F1180DA0D2C1C44F97D0F081B13D33984C198ECD695C34B2E79A3E7CFBDD2D67D630C019714C3A70280E; tivi_=nqkngs45drnsh4z4y4b30g55
please help me! how to create ".ASPXAUTH" cookie ?
Python requests module does the job for me with .ASPXAUTH cookie-based authentication. In a simple example below .ASPXAUTH cookie is received after the first session.get call and used for the second call:
import requests
session = requests.Session()
session.get('https://host/user/login?username=###&password=###')
print session.cookies
session.get('https://host/apicall')
print session.cookies