I'm trying to log in into one web application with python but very attempt ends with 500 error and the html body shows the error: [HttpAntiForgeryException]. I tried to apply a few solutions from the other questions here but nothing helped. So now, I'm sucked at first request which response Is giving me 500.
import requests
from bs4 import BeautifulSoup
url = "http://localhost:52053/Account/Login"
username = "test#test.sk"
user_password = "pass"
session = requests.Session()
response = session.get(url)
soup = BeautifulSoup(response.content, features="html.parser")
#print(soup)
states = ["__RequestVerificationToken", "Email", "RememberMe"]
login_data = {"username": username, "password": user_password, "Login": "submit"}
headers = {"Host": "localhost:52053",
"Content-Type": "application/x-www-form-urlencoded",
"Connection": "close",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0)",
"Cookie": str(session.cookies.get_dict())}
for state in states: # search for existing aspnet states and get its values
result = soup.find('input', {'name': state})
if not (result is None): # when existent (some may not be needed!)
if state == "Email":
login_data.update({state: login_data["username"]})
else:
login_data.update({state: result['value']})
post_request = session.post(url, headers=headers, data=login_data)
Successful login attempt looks like this.
POST /Account/Login HTTP/1.1
Host: localhost:52053
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sk,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Content-Length: 193
Origin: http://localhost:52053
Connection: close
Referer: http://localhost:52053/Account/Login
Cookie: __RequestVerificationToken=j9yFGpTFSlH5_aQt0k-Gvz10I16TVXbDk31NKPm1HkcWsksUfKXkjL567yFplCS_VovTR7lVuEgNjwgp-EO3RjNj4gQOvNUXnPkjymZx_jA1
Upgrade-Insecure-Requests: 1
__RequestVerificationToken=LjHuOdKSCr1A7KRDNie4GUnCZ3qRwUCdHyLlPYT40DsEB-GNUvEKxe5nvZWf5gZ4ZflwI43xGWPyYu8GI15wroEg9WRRVtSzZ9-KY9Mu_JA1&Email=test%40test.sk&Password=pass&RememberMe=false
Following response is:
HTTP/1.1 302 Found
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Location: /
Server: Microsoft-IIS/10.0
X-AspNetMvc-Version: 5.2
X-AspNet-Version: 4.0.30319
Set-Cookie: .AspNet.ExternalCookie=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: .AspNet.ApplicationCookie=KCLm03FHj8v_6rIpTzBTm7EzEtzpKmIz1Z9_z29wycUSqUVyKbGEmptXUwG41MqNOMR7Vbeq2u576ijazupNLffLP-Ua0n60aLmnVSDsLsdTqYT7jjqyGPw1Ppp8AnIDs3sdefmksazX2UvKTxzxRBufFCoxtCJx51mWtBv7v0JzUeC1hnfu1AIJ7GH_8T59KD3iv0hRSHDqlWHlkWzyN1Xt0m5ixC14e4eC2YxEm3_acy96atB2Jv5u0HREPzssLmywuzj6sLa9cHCllTG2gMVWvHA3IDhCWu7Ojf8BO02Eml3pPM5QTJ-sq540fcj9QyELayUOwBZWffSgsJeq8mlt3FupQcJ-JTJxDzAsDc4Cmk-BcvYSfpAJq4SdR-Y4mTN_6vu-wwAOLZPSgh-5K7guWmZ3VfRitZHXd_rvTEmMiVrgHFTEQAkUYu4zTSupxRplTtKb1VSDs0Nc1uEos2z0_aw-nBbRBrTPpvmqGok
Auth flow continues with this request. I'm not trying to sent this request yet (I put it here just for better imagination):
GET / HTTP/1.1
Host: localhost:52053
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sk,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:52053/Account/Login
Connection: close
Cookie: __RequestVerificationToken=j9yFGpTFSlH5_aQt0k-Gvz10I16TVXbDk31NKPm1HkcWsksUfKXkjL567yFplCS_VovTR7lVuEgNjwgp-EO3RjNj4gQOvNUXnPkjymZx_jA1;
.AspNet.ApplicationCookie=gvv113IJhtdaOhdc0Rz2N--5Ob18W6gS64J3wtOJggRTqE70h-8HyBGQAmLvSM2qCV2e-dXR2Uto-BktD6NmNz6dJtxckIYasPOfqodDNZX33YJxNEDg7a64LPi1bNnmrnvQcOHAceQNqZDykXrhFm55dqoo1oZnJHfZQnltwqAdg7DGO31PZpzu-GAZh2_gzuxd_saJdS09ZZQrc9h7WiU2ONqeya87pSAN7ZyHQ_XvsU5cUwDGq7FWLpzlIeeZWkay6iWVmCSwNEofpdVsb880P3XZnFKEj2SW2PfazdNLfgy86YNjkoD6_3Vb1BLirRoSP0XIQMcs2F_CzgXkxD5GvDray8TPYqcQJ4L2fikReUJHadx9fFnslF2BFcnKYC8D-Xusrda_5r-CQoQ4SzAe2Cqn0h1NYHxS1wsxt35neC5RuQ3geadAEEghjrSSVhSl8jCfACtQtcBeNL2x_m6I9L3XJCjMpzJjtP6up3E
Upgrade-Insecure-Requests: 1
Next response is just kind of 200 - you are in.
So my problem is that the response from the first request is failing. Is someone able to see some mistake or did I forgot something?
Failed response from the first request call:
HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNcUENBZG1pbmlzdHJhdG9yXERlc2t0b3BccGVuIHRlc3RpbmdcU2VjdXJpdHlXb3Jrc2hvcC1EVldBLW1hc3RlclxkdndhLXRyYWluaW5nXGR2d2EtdHJhaW5pbmdcQWNjb3VudFxMb2dpbg==?=
If I try to pint request headers and login_data, result is:
print(post_request.request.headers)
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0)', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'close', 'Host': 'localhost:52053', 'Content-Type': 'application/x-www-form-urlencoded', 'Cookie': "{'__RequestVerificationToken': 'yg-7mFRyZiONwsZ2dIVkIIW5tB7gSL_sazgphg-VuW2OpNNRRkxmLH-9SZJXiN9whUC_BYTo8RgsiDrVjcYtLEf9anW56rVwZ2RQPzxHA481'}", 'Content-Length': '249'}
print(login_data)
{'username': 'test#test.sk', 'password': 'pass', 'Login': 'submit', '__RequestVerificationToken': '14OuwaRqldlGKi93C91zf6QD_ouOorHBDe63s4KgfP3gbt85V0QMy2X5OMwWAo1TUrD8zJ-zoZbXLPpgDI_wrxVZv3ceYNos_e5_elFhVt01', 'Email': 'test#test.sk', 'RememberMe': 'true', 'Password': 'pass'}
I just find the solution out.
Requests could handle all headers by itself (and my headers was, for some reason, causing errors), it was redirecting my request to 200 response so I didn't instantly saw, that it is actually working and catching 302 response.
I found out by printing:
print(post_request.history)
Which gave me <[302]>
Now, when I know, there is a redirection, I just have to allow_redirects=False and now I'm able to catch my set-cookie header
Full code, witch is getting an expected responses is:
import requests
from bs4 import BeautifulSoup
url = "http://localhost:52053/Account/Login"
username = "test#test.sk"
user_password = "pass"
session = requests.Session()
response = session.get(url)
soup = BeautifulSoup(response.content, features="html.parser")
#print(soup)
states = ["__RequestVerificationToken", "Email", "RememberMe"]
login_data = {"username": username, "password": user_password, "Login": "submit"}
for state in states: # search for existing aspnet states and get its values
result = soup.find('input', {'name': state})
if not (result is None): # when existent (some may not be needed!)
if state == "Email":
login_data.update({state: login_data["username"]})
else:
login_data.update({state: result['value']})
post_request = session.post(url, data=login_data, allow_redirects=False)
print(login_data)
#the code below is testing, if the HttpAntiForgeryException is in code
if "HttpAntiForgeryException" not in post_request.text:
print(post_request.headers)
else:
print("antiforgery")
Related
I am using Laravel 8 and Sanctum and attempting to make an API call from Python into the Laravel app.
I don't seem to be sending the request variables correctly from python as they are not being picked up as $request->variable (see "auction_item_id":null in Laravel log below) however they do show up in the request (see below Laravel log).
Is this a problem with the python request header?
Python:
import requests
import re
import json
############## LOGIN ####################
url = 'http://127.0.0.1:8082/api/apilogin'
data = {
'email': 'user#email.com',
'password': 'password'}
r = requests.post(url, data=data)
user = json.loads(r.text)
############## MAKE REST API CALL using login token from prior call ####################
url2 = 'http://127.0.0.1:8082/api/beginlivebiddingforitemAPI'
token = "Bearer " + user["token"]
headers = {"Authorization": token, 'Accept': 'application/json'}
data = {
'auction_id' : 103,
'auction_item_id' : 1224
}
rr = requests.get(url2, data=data, headers=headers)
AuctionBidsController:
public function beginlivebiddingforitem (Request $request){
Log::info("AuctionBidsController#beginlivebiddingforitem", ['auction_item_id' => $request->auction_item_id]);
Api.php:
Route::post('/apilogin', 'UsersController#APIlogin');
Route::middleware(['auth:sanctum'])->group(function () {
Route::get('/beginlivebiddingforitemAPI', 'AuctionBidsController#beginlivebiddingforitem');
});
Laravel log:
[2020-11-06 16:46:44] local.INFO: AuctionBidsController#beginlivebiddingforitem {"auction_item_id":null,"request":{"Illuminate\\Http\\Request":"GET /api/beginlivebiddingforitemAPI HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Bearer 100|3EjawNPRinapHXxZvDirzyKM73AzeuJC8OTP6xOZ
Connection: keep-alive
Content-Length: 35
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:8082
User-Agent: python-requests/2.23.0
auction_id=103&auction_item_id=1224"}}
^^^^^^^^^^^^^^^
THEY ARE HERE
** But look at "auction_item_id":null **
Here is what a normal request looks like inside laravel:
[2020-11-06 18:34:56] local.INFO: AuctionBidsController#beginlivebiddingforitem {"auction_item_id":"1223","request":{"Illuminate\\Http\\Request":"GET /beginlivebiddingforitem?auction_id=103&auction_item_id=1223 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: laravel_cookie_consent=1; _ga=GA1.1.1339345323.1594720189; PGADMIN_INT_KEY=129e52c8-dddd-41a0-bc43-53e91dec5402; pga4_session=1c3a240a-46a1-4a9e-8e25-d489bb8ddddddddNltIXmby1M+34KepOYCsGAsY=; PGADMIN_LANGUAGE=en; _gid=GA1.1.1579327561.1604514920; _gat_gtag_UA_170463940_1=1; XSRF-TOKEN=eyJpdiI6ImIxT00yWW54REE2emVQSEpZZzZRRmc9PSIsInZhbHVlIjoiWmlCZGhia1pxeVJ0TmU4NGRBV2xMUURsS21RT0cyWGZUYddddddddddddddddddddddU9oUHg3cGQrdWZhd2gybnRCZ0xUajcwdEhuREh5OE5FcWkzV2tWb3lKOTRwUlljWld4TE9KL3BiMFp0M1dzeklmY2VhaEUiLCJtYWMiOiI0OWZhYmE3ZWM0ZjFhNTU5OWM1ZWE1MmY0MmIyMWQ3YTU1YzE3MmI0NWM1YjY0NTE0MzdhYTVmOWNmODIzMWVlIn0%3D; icollect_session=eyJpdiI6Ik5xSlAzR3NaZUR4QUhxV0RBRFBvd2c9PSIsInZhbHVlIjoianJLbFdVUDJCL2FKZEo5NTFUMElVUFJMRm9BdddddddddddddddHdYL3Arc3Avc1JTcUE5RmNPZ1J2MUVxblFUbmREcHJJeUZJaHZNVzJHTVNhR2p6Snp6M2JkQXJObE9BQU91Si9RbVIzRXMiLCJtYWMiOiI5MzcwMjc5ZWQ0MDE5ZDk2NTVmYmJjNGQ5NWYzMGY2YTVkMDI5YWQ0YTlkNGU1YTUyZTIyYzhkODU4NjNiNDA2In0%3D
Dnt: 1
Host: 127.0.0.1:8082
Referer: http://127.0.0.1:8082/bidliveauctioneer/103
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36 Edg/86.0.622.43
X-Csrf-Token: wrdZoCDCVeqddddzVxAxByw183bFMuddddP4QBKy
X-Requested-With: XMLHttpRequest
X-Socket-Id: 9357.547174
Cookie: laravel_cookie_consent=1; _ga=; PGADMIN_INT_KEY=; pga4_session=; PGADMIN_LANGUAGE=; _gid=; _gat_gtag_UA_170463940_1=; XSRF-TOKEN=wrdZoCDCVeqddddzVxAxByw183bFMuddddP4QBKy; icollect_session=Qx2QddddNvlIg3ogQ596BdddducrFk4Bmi0nLA89
**note that variables occur right after the GET
GET /beginlivebiddingforitem?auction_id=103&auction_item_id=1223 **
I was using data = data and it's params = data on requests.get(url2, params=data, headers=headers).
Source: https://requests.readthedocs.io/en/master/user/quickstart
I used the following URL to search for Piano with distance of 20 miles.
When I ran it with BurpSuite, the request appeared as:
POST /MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be HTTP/1.1
Host: www.mtna.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://www.mtna.org/MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be
X-Requested-With: XMLHttpRequest
X-MicrosoftAjax: Delta=true
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Content-Length: 4107
Connection: close
Cookie: ASP.NET_SessionId=cxqktlukl4zprfps4wfyex44; Asi.Web.Browser.CookiesEnabled=true; AnonymousCartId=00000000-0000-0000-0000-000000000000; _ga=GA1.2.1639735034.1545046479; _gid=GA1.2.6620465.1545046479
ctl01%24ScriptManager1=ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ListerPanel%7Cctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24SubmitButton&__WPPS=s&__CTRLKEY=&__SHIFTKEY=&ctl01_ScriptManager1_TSM=%3B%3BAjaxControlToolkit%2C%20Version%3D4.1.50508%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D28f01b0e84b6d53e%3Aen-US%3A0c8c847b-b611-49a7-8e75-2196aa6e72fa%3Aea597d4b%3Ab25378d2%3BTelerik.Web.UI%2C%20Version%3D2014.1.403.45%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D121fae78165ba3d4%3Aen-US%3A68d9452f-f268-45b2-8db7-8c3bbf305b8d%3A16e4e7cd%3Af7645509%3A24ee1bba%3Ae330518b%3A2003d0b8%3Ac128760b%3A1e771326%3A88144a7a%3Ac8618e41%3A1a73651d%3A16d8629e%3A874f8ea2%3Af46195d3%3A19620875%3Acda80b3%3Aed16cbdc%3A92fe8ea0%3Afa31b949%3A490a9d4e%3Abd8f85e4%3BAjaxControlToolkit%2C%20Version%3D4.1.50508.0%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D28f01b0e84b6d53e%3Aen-US%3A0c8c847b-b611-49a7-8e75-2196aa6e72fa%3A782b16ab&PageInstanceKey=0428325d-7c92-4972-a40f-a36c36bad719&TemplateUserMessagesID=ctl01_TemplateUserMessages_ctl00_Messages&PageIsDirty=false&IsControlPostBackctl01%24HeaderLogo%24HeaderLogoSpan=1&IsControlPostBackctl01%24SocialNetworking%24SocialNetworking=1&IsControlPostBackctl01%24SearchField=1&__EVENTTARGET=ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24SubmitButton&__EVENTARGUMENT=&NavMenuClientID=ctl01_Primary_NavMenu&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml%24ciNewContentHtml=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewShowConsumerAddressCommon%24ciNewShowConsumerAddressCommon=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml2%24ciNewContentHtml2=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciScriptforadtracking%24ciScriptforadtracking=1&IsControlPostBackctl01%24TemplateBody%24ContentPage1=1&IsControlPostBackctl01%24TemplateBody%24ContentPageFooter1=1&IsControlPostBackctl01%24FooterCommunications%24FooterCommunications=1&IsControlPostBackctl01%24FooterCommunications%24NewContentHtml=1&IsControlPostBackctl01%24FooterCopyright%24FooterCopyright=1&__VIEWSTATE=%2FwEPaA8FDzhkNjYzZjU1MjZmNTFlYhgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WCAUYY3RsMDEkTG9naW5TdGF0dXMxJGN0bDAxBRhjdGwwMSRMb2dpblN0YXR1czEkY3RsMDMFFWN0bDAxJFByaW1hcnkkTmF2TWVudQVXY3RsMDEkVGVtcGxhdGVCb2R5JFdlYlBhcnRNYW5hZ2VyMSRnd3BjaVRlYWNoZXJRdWVyaWVzJGNpVGVhY2hlclF1ZXJpZXMkUXVlcnlMaXN0RGlhbG9nBQtjdGwwMSRjdGwxNgUTY3RsMDEkR2VuZXJpY1dpbmRvdwUTY3RsMDEkT2JqZWN0QnJvd3NlcgUZY3RsMDEkT2JqZWN0QnJvd3NlckRpYWxvZ5YeffipvvNAXInmJ3MH5Z%2FoW9I1&__VIEWSTATEGENERATOR=C56D694D&ctl01%24lastClickedElementId=&ctl01%24SearchField%24SearchTerms=Keyword%20Search&ctl01_Primary_NavMenu_ClientState=&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenCacheQueryId=&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenQueryCached=False&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl01=277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL7&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl04=277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL9&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input0%24DropDown1=Piano&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input1%24DropDown1=20&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24HiddenKeyField1=&ctl01_TemplateBody_WebPartManager1_gwpciTeacherQueries_ciTeacherQueries_QueryListDialog_ClientState=&ctl01_GenericWindow_ClientState=&ctl01_ObjectBrowser_ClientState=&ctl01_ObjectBrowserDialog_ClientState=&ctl01_ctl16_ClientState=&__ASYNCPOST=true
In the response I get 63 results.
I converted the POST request to Python:
import requests
session = requests.Session()
paramsGet = {"cak":"31022421-c202-431a-b5ef-c6d10c90e6be"}
paramsPost = {"ctl01_ScriptManager1_TSM":";;AjaxControlToolkit, Version=4.1.50508, Culture=neutral, PublicKeyToken=28f01b0e84b6d53e:en-US:0c8c847b-b611-49a7-8e75-2196aa6e72fa:ea597d4b:b25378d2;Telerik.Web.UI, Version=2014.1.403.45, Culture=neutral, PublicKeyToken=121fae78165ba3d4:en-US:68d9452f-f268-45b2-8db7-8c3bbf305b8d:16e4e7cd:f7645509:24ee1bba:e330518b:2003d0b8:c128760b:1e771326:88144a7a:c8618e41:1a73651d:16d8629e:874f8ea2:f46195d3:19620875:cda80b3:ed16cbdc:92fe8ea0:fa31b949:490a9d4e:bd8f85e4;AjaxControlToolkit, Version=4.1.50508.0, Culture=neutral, PublicKeyToken=28f01b0e84b6d53e:en-US:0c8c847b-b611-49a7-8e75-2196aa6e72fa:782b16ab","__VIEWSTATEGENERATOR":"C56D694D","__EVENTARGUMENT":"","IsControlPostBackctl01%24HeaderLogo%24HeaderLogoSpan":"1","__VIEWSTATE":"/wEPaA8FDzhkNjYzZWY4NmM2Y2Y4MBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WCAUYY3RsMDEkTG9naW5TdGF0dXMxJGN0bDAxBRhjdGwwMSRMb2dpblN0YXR1czEkY3RsMDMFFWN0bDAxJFByaW1hcnkkTmF2TWVudQVXY3RsMDEkVGVtcGxhdGVCb2R5JFdlYlBhcnRNYW5hZ2VyMSRnd3BjaVRlYWNoZXJRdWVyaWVzJGNpVGVhY2hlclF1ZXJpZXMkUXVlcnlMaXN0RGlhbG9nBQtjdGwwMSRjdGwxNgUTY3RsMDEkR2VuZXJpY1dpbmRvdwUTY3RsMDEkT2JqZWN0QnJvd3NlcgUZY3RsMDEkT2JqZWN0QnJvd3NlckRpYWxvZ5t6EvKfQQS4e3kXIpgfVSNxCTNO","__WPPS":"s","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries":"1","__SHIFTKEY":"","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciScriptforadtracking%24ciScriptforadtracking":"1","NavMenuClientID":"ctl01_Primary_NavMenu","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenCacheQueryId":"","ctl01%24SearchField%24SearchTerms":"Keyword Search","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24HiddenKeyField1":"","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenQueryCached":"False","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml2%24ciNewContentHtml2":"1","ctl01_GenericWindow_ClientState":"","ctl01_Primary_NavMenu_ClientState":"","PageInstanceKey":"1f89a5d5-ce61-4147-9eb5-32cafa6174a6","ctl01_ObjectBrowserDialog_ClientState":"","__CTRLKEY":"","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl04":"277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL9","ctl01%24ScriptManager1":"ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ListerPanel|ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ResultsGrid\x24Sheet0\x24SubmitButton","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input1%24DropDown1":"20","IsControlPostBackctl01%24SocialNetworking%24SocialNetworking":"1","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input0%24DropDown1":"Piano","PageIsDirty":"false","__ASYNCPOST":"true","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewShowConsumerAddressCommon%24ciNewShowConsumerAddressCommon":"1","IsControlPostBackctl01%24TemplateBody%24ContentPageFooter1":"1","__EVENTTARGET":"ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ResultsGrid\x24Sheet0\x24SubmitButton","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl01":"277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL7","ctl01_ctl16_ClientState":"","IsControlPostBackctl01%24SearchField":"1","ctl01_TemplateBody_WebPartManager1_gwpciTeacherQueries_ciTeacherQueries_QueryListDialog_ClientState":"","IsControlPostBackctl01%24FooterCommunications%24FooterCommunications":"1","ctl01%24lastClickedElementId":"","TemplateUserMessagesID":"ctl01_TemplateUserMessages_ctl00_Messages","IsControlPostBackctl01%24TemplateBody%24ContentPage1":"1","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml%24ciNewContentHtml":"1","ctl01_ObjectBrowser_ClientState":"","IsControlPostBackctl01%24FooterCommunications%24NewContentHtml":"1","IsControlPostBackctl01%24FooterCopyright%24FooterCopyright":"1"}
headers = {"X-MicrosoftAjax":"Delta=true","Accept":"*/*","X-Requested-With":"XMLHttpRequest","Cache-Control":"no-cache","User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0","Referer":"https://www.mtna.org/MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be","Connection":"close","Accept-Encoding":"gzip, deflate","Accept-Language":"en-US,en;q=0.5","Content-Type":"application/x-www-form-urlencoded; charset=utf-8"}
cookies = {"AnonymousCartId":"00000000-0000-0000-0000-000000000000","ASP.NET_SessionId":"cxqktlukl4zprfps4wfyex44","_gat":"1","Asi.Web.Browser.CookiesEnabled":"true","_ga":"GA1.2.1639735034.1545046479","_gid":"GA1.2.6620465.1545046479"}
response = session.post("https://www.mtna.org/MTNA/FindATeacherSearch.aspx", data=paramsPost, params=paramsGet, headers=headers, cookies=cookies)
print("Status code: %i" % response.status_code)
print("Response body: %s" % response.content)
But the results leave me on the same page.
Any idea why ?
What cause its different ?
Updated code - I'm using this code to send the request:
headers = {
"Host": "www.roblox.com",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0",
"Accept": "application/json, text/plain, */*",
"Accept-Language": "en-US;q=0.7,en;q=0.3",
"Referer": "https://www.roblox.com/users/12345/profile",
"Content-Type": "application/json;charset=utf-8",
"X-CSRF-TOKEN": "some-xsrf-token",
"Content-Length": "27",
"DNT": "1",
"Connection": "close"
}
data = {"targetUserId":"56789"}
url = "http://www.roblox.com/user/follow"
r = requests.post(url, headers=headers, data=data, cookies={"name":"value"})
Response (using r.text):
{"isValid":false,"data":null,"error":""}
The request itself is valid, I sent it using burp and it worked:
POST /user/follow HTTP/1.1
Host: www.roblox.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: application/json, text/plain, */*
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Referer: https://www.roblox.com/users/12345/profile
Content-Type: application/json;charset=utf-8
X-CSRF-TOKEN: Ab1/2cde3fGH
Content-Length: 27
Cookie: some-cookie=;
DNT: 1
Connection: close
{"targetUser":"56789"}
Because it works in Burp but not in Python requests, get a packet sniffer (Wireshark is the simplest IMO) and look to see the difference in the packet sent by Burp that works and the one sent from Python that does not work. I am suspecting that the problem is that the website is HTTPS but you are using http://www.roblox.com . Do try https://www.roblox.com , but I am not sure if it will work.
So I've been trying for the last 6 hours to make this work, but I couldn't and endless searches didn't help, So I guess I'm either doing something very fundamental wrong, or it's just a trivial bug which happens to match my logic so I need extra eyes to help me fix it.
The website url is this.
I wrote a piece of messy python code to just login and read the next page, but All I get is a nasty 500 error saying something on the server went wrong processing my request.
Here is the request made by a browser which works just fine, no problem.
HTTP Response code to this request is 302 (Redirect)
POST /appstatus/index.aspx HTTP/1.1
Host: www.wes.org
Connection: close
Content-Length: 303
Cache-Control: max-age=0
Origin: https://www.wes.org
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Content-Type: application/x-www-form-urlencoded
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Referer: https://www.wes.org/appstatus/index.aspx
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.8,fa;q=0.6
Cookie: ASP.NET_SessionId=bu2gemmlh3hvp4f5lqqngrbp; _ga=GA1.2.1842963052.1473348318; _gat=1
__VIEWSTATE=%2FwEPDwUKLTg3MTMwMDc1NA9kFgICAQ9kFgICAQ8PFgIeBFRleHRkZGRk9rP20Uj9SdsjOKNUBlbw55Q01zI%3D&__VIEWSTATEGENERATOR=189D346C&__EVENTVALIDATION=%2FwEWBQK6lf6LBAKf%2B9bUAgK9%2B7qcDgK8w4S2BALowqJjoU1f0Cg%2FEAGU6r2IjpIPG8BO%2BiE%3D&txtUID=Email%40Removed.com&txtPWD=PASSWORDREMOVED&Submit=Log+In&Hidden1=
and this one is the request made by my script.
POST /appstatus/index.aspx HTTP/1.1
Host: www.wes.org
Connection: close
Accept-Encoding: gzip, deflate, br
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
Origin: https://www.wes.org
Accept-Language: en-US,en;q=0.8,fa;q=0.6
Cache-Control: max-age=0
Referer: https://www.wes.org/appstatus/indexca.aspx
Cookie: ASP.NET_SessionId=nxotmb55jjwf5x4511rwiy45
Content-Length: 303
txtPWD=PASSWORDREMOVED&Submit=Log+In&__EVENTVALIDATION=%2FwEWBQK6lf6LBAKf%2B9bUAgK9%2B7qcDgK8w4S2BALowqJjoU1f0Cg%2FEAGU6r2IjpIPG8BO%2BiE%3D&txtUID=Email%40Removed.com&__VIEWSTATE=%2FwEPDwUKLTg3MTMwMDc1NA9kFgICAQ9kFgICAQ8PFgIeBFRleHRkZGRk9rP20Uj9SdsjOKNUBlbw55Q01zI%3D&Hidden1=&__VIEWSTATEGENERATOR=189D346C
And this is the script making the request, I'm sorry if it's so messy, just need something quick.
import requests
import bs4
import urllib.parse
def main():
session = requests.Session()
headers = {"Origin": "https://www.wes.org",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Cache-Control": "max-age=0", "Upgrade-Insecure-Requests": "1", "Connection": "close",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36",
"Referer": "https://www.wes.org/appstatus/indexca.aspx", "Accept-Encoding": "gzip, deflate, br",
"Accept-Language": "en-US,en;q=0.8,fa;q=0.6", "Content-Type": "application/x-www-form-urlencoded"}
r = session.get('https://www.wes.org/appstatus/index.aspx',headers=headers)
cookies = r.cookies
soup = bs4.BeautifulSoup(r.content, "html5lib")
viewState=urllib.parse.quote(str(soup.select('#__VIEWSTATE')[0]).split('value="')[1].split('"/>')[0])
viewStateGenerator=urllib.parse.quote(str(soup.select('#__VIEWSTATEGENERATOR')[0]).split('value="')[1].split('"/>')[0])
eventValidation=urllib.parse.quote(str(soup.select('#__EVENTVALIDATION')[0]).split('value="')[1].split('"/>')[0])
paramsPost = {}
paramsPost.update({'__VIEWSTATE':viewState})
paramsPost.update({'__VIEWSTATEGENERATOR':viewStateGenerator})
paramsPost.update({'__EVENTVALIDATION':eventValidation})
paramsPost.update({"txtUID": "My#Email.Removed"})
paramsPost.update({"txtPWD": "My_So_Called_Password"})
paramsPost.update({"Submit": "Log In"})
paramsPost.update({"Hidden1": ""})
response = session.post("https://www.wes.org/appstatus/index.aspx", data=paramsPost, headers=headers,
cookies=cookies)
print("Status code:", response.status_code) #Outputs 500.
#print("Response body:", response.content)
if __name__ == '__main__':
main()
Any help would be so much appreciated.
You are doing way too much work and in doing so not passing valid data,you extract value attribute directly i.e .select_one('#__VIEWSTATEGENERATOR')["value"] and the same for all the rest, the cookies will be set in the Session object after your initial get so the logic boils down to:
with requests.Session() as session:
headers = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/52.0.2743.116 Safari/537.36"}
r = session.get('https://www.wes.org/appstatus/index.aspx', headers=headers)
soup = bs4.BeautifulSoup(r.content, "html5lib")
viewState = soup.select_one('#__VIEWSTATE')["value"]
viewStateGenerator = soup.select_one('#__VIEWSTATEGENERATOR')["value"]
eventValidation = soup.select_one('#__EVENTVALIDATION')["value"]
paramsPost = {'__VIEWSTATE': viewState,'__VIEWSTATEGENERATOR': viewStateGenerator,
'__EVENTVALIDATION': eventValidation,"txtUID": "My#Email.Removed",
"txtPWD": "My_So_Called_Password",
"Submit": "Log In","Hidden1": ""}
response = session.post("https://www.wes.org/appstatus/index.aspx", data=paramsPost, headers=headers)
print("Status code:", response.status_code)
Python by convention uses CamelCase for class names and lowercase with underscores to separate multiple words, you might want to consider applying that to your code.
I try to create POST request to ASP site (just like in Firefox), for get JSON response.
But in my code response is html, not JSON.
link to site
Firebug Response Headers:
HTTP/1.1 200 OK
Cache-Control: private
Content-Type: application/json; charset=utf-8
Server: Microsoft-IIS/7.5
X-AspNet-Version: 4.0.30319
X-Powered-By: ASP.NET
Date: Mon, 08 Sep 2014 11:32:22 GMT
Content-Length: 101
Firebug Request Headers:
POST /Portal/WebPageMethods/Playlista/playlist.aspx HTTP/1.1
Host: www.polskieradio.pl
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Referer: http://www.polskieradio.pl/10,Czworka.json
Content-Length: 17
Cookie: cookies-accepted=true; ASP.NET_SessionId=35p3kig5t5cmlikubnlnytlh
Connection: keep-alive
Pragma: no-cache
Cache-Control: no-cache
source code:
import requests
import json
url = "http://www.polskieradio.pl/Portal/WebPageMethods/Playlista/playlist.aspx?program=4&count=1"
payload = { "Host": "www.polskieradio.pl",
"User-Agent": "Mozilla/5.0 (X11; Linux x86_64; rv:30.0) Gecko/20100101 Firefox/30.0",
"Accept": "application/json, text/javascript, */*; q=0.01",
"Accept-Language": "pl,en-US;q=0.7,en;q=0.3",
"Accept-Encoding": "gzip, deflate",
"Content-Type": "application/x-www-form-urlencoded; charset=UTF-8",
"X-Requested-With": "XMLHttpRequest",
"Referer": "http://www.polskieradio.pl/10,Czworka",
"Content-Length": "17",
"Cookie": "cookies-accepted=true; ASP.NET_SessionId=5l1eezrjfdyvvevxushojtc2",
"Connection": "keep-alive",
"Pragma": "no-cache",
"Cache-Control": "no-cache"
}
r = requests.post(url, data=json.dumps(payload))
print(r.headers['content-type'])
print r.content
How to do this properly?
Thanks for answers!
Try a little bit different...
Look at this example:
headers = {'Content-type': 'application/json', 'Accept': 'text/plain'}
r = requests.post(url, data=json.dumps(data), headers=headers)
Accept is a header, not a payload.
Everything you are sending as payload, are, in fact, headers.
Your POST payload may be program=4&count=1, or you can do a GET.
--- ADDITION with final solution
import requests
import json
url = "http://www.polskieradio.pl/Portal/WebPageMethods/Playlista/playlist.aspx"
data = 'program=4&count=1'
headers = {
'User-Agent': 'curl/7.35.0',
'Host': 'www.polskieradio.pl',
'Accept':'*/*',
'Proxy-Connection': 'Keep-Alive',
'Content-Type': 'application/x-www-form-urlencoded'
}
r = requests.post(url, data=data, headers=headers)
print r.content