Respond from POST request on BurpSuite is different than Python - python

I used the following URL to search for Piano with distance of 20 miles.
When I ran it with BurpSuite, the request appeared as:
POST /MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be HTTP/1.1
Host: www.mtna.org
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Referer: https://www.mtna.org/MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be
X-Requested-With: XMLHttpRequest
X-MicrosoftAjax: Delta=true
Cache-Control: no-cache
Content-Type: application/x-www-form-urlencoded
Content-Length: 4107
Connection: close
Cookie: ASP.NET_SessionId=cxqktlukl4zprfps4wfyex44; Asi.Web.Browser.CookiesEnabled=true; AnonymousCartId=00000000-0000-0000-0000-000000000000; _ga=GA1.2.1639735034.1545046479; _gid=GA1.2.6620465.1545046479
ctl01%24ScriptManager1=ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ListerPanel%7Cctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24SubmitButton&__WPPS=s&__CTRLKEY=&__SHIFTKEY=&ctl01_ScriptManager1_TSM=%3B%3BAjaxControlToolkit%2C%20Version%3D4.1.50508%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D28f01b0e84b6d53e%3Aen-US%3A0c8c847b-b611-49a7-8e75-2196aa6e72fa%3Aea597d4b%3Ab25378d2%3BTelerik.Web.UI%2C%20Version%3D2014.1.403.45%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D121fae78165ba3d4%3Aen-US%3A68d9452f-f268-45b2-8db7-8c3bbf305b8d%3A16e4e7cd%3Af7645509%3A24ee1bba%3Ae330518b%3A2003d0b8%3Ac128760b%3A1e771326%3A88144a7a%3Ac8618e41%3A1a73651d%3A16d8629e%3A874f8ea2%3Af46195d3%3A19620875%3Acda80b3%3Aed16cbdc%3A92fe8ea0%3Afa31b949%3A490a9d4e%3Abd8f85e4%3BAjaxControlToolkit%2C%20Version%3D4.1.50508.0%2C%20Culture%3Dneutral%2C%20PublicKeyToken%3D28f01b0e84b6d53e%3Aen-US%3A0c8c847b-b611-49a7-8e75-2196aa6e72fa%3A782b16ab&PageInstanceKey=0428325d-7c92-4972-a40f-a36c36bad719&TemplateUserMessagesID=ctl01_TemplateUserMessages_ctl00_Messages&PageIsDirty=false&IsControlPostBackctl01%24HeaderLogo%24HeaderLogoSpan=1&IsControlPostBackctl01%24SocialNetworking%24SocialNetworking=1&IsControlPostBackctl01%24SearchField=1&__EVENTTARGET=ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24SubmitButton&__EVENTARGUMENT=&NavMenuClientID=ctl01_Primary_NavMenu&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml%24ciNewContentHtml=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewShowConsumerAddressCommon%24ciNewShowConsumerAddressCommon=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml2%24ciNewContentHtml2=1&IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciScriptforadtracking%24ciScriptforadtracking=1&IsControlPostBackctl01%24TemplateBody%24ContentPage1=1&IsControlPostBackctl01%24TemplateBody%24ContentPageFooter1=1&IsControlPostBackctl01%24FooterCommunications%24FooterCommunications=1&IsControlPostBackctl01%24FooterCommunications%24NewContentHtml=1&IsControlPostBackctl01%24FooterCopyright%24FooterCopyright=1&__VIEWSTATE=%2FwEPaA8FDzhkNjYzZjU1MjZmNTFlYhgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WCAUYY3RsMDEkTG9naW5TdGF0dXMxJGN0bDAxBRhjdGwwMSRMb2dpblN0YXR1czEkY3RsMDMFFWN0bDAxJFByaW1hcnkkTmF2TWVudQVXY3RsMDEkVGVtcGxhdGVCb2R5JFdlYlBhcnRNYW5hZ2VyMSRnd3BjaVRlYWNoZXJRdWVyaWVzJGNpVGVhY2hlclF1ZXJpZXMkUXVlcnlMaXN0RGlhbG9nBQtjdGwwMSRjdGwxNgUTY3RsMDEkR2VuZXJpY1dpbmRvdwUTY3RsMDEkT2JqZWN0QnJvd3NlcgUZY3RsMDEkT2JqZWN0QnJvd3NlckRpYWxvZ5YeffipvvNAXInmJ3MH5Z%2FoW9I1&__VIEWSTATEGENERATOR=C56D694D&ctl01%24lastClickedElementId=&ctl01%24SearchField%24SearchTerms=Keyword%20Search&ctl01_Primary_NavMenu_ClientState=&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenCacheQueryId=&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenQueryCached=False&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl01=277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL7&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl04=277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL9&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input0%24DropDown1=Piano&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input1%24DropDown1=20&ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24HiddenKeyField1=&ctl01_TemplateBody_WebPartManager1_gwpciTeacherQueries_ciTeacherQueries_QueryListDialog_ClientState=&ctl01_GenericWindow_ClientState=&ctl01_ObjectBrowser_ClientState=&ctl01_ObjectBrowserDialog_ClientState=&ctl01_ctl16_ClientState=&__ASYNCPOST=true
In the response I get 63 results.
I converted the POST request to Python:
import requests
session = requests.Session()
paramsGet = {"cak":"31022421-c202-431a-b5ef-c6d10c90e6be"}
paramsPost = {"ctl01_ScriptManager1_TSM":";;AjaxControlToolkit, Version=4.1.50508, Culture=neutral, PublicKeyToken=28f01b0e84b6d53e:en-US:0c8c847b-b611-49a7-8e75-2196aa6e72fa:ea597d4b:b25378d2;Telerik.Web.UI, Version=2014.1.403.45, Culture=neutral, PublicKeyToken=121fae78165ba3d4:en-US:68d9452f-f268-45b2-8db7-8c3bbf305b8d:16e4e7cd:f7645509:24ee1bba:e330518b:2003d0b8:c128760b:1e771326:88144a7a:c8618e41:1a73651d:16d8629e:874f8ea2:f46195d3:19620875:cda80b3:ed16cbdc:92fe8ea0:fa31b949:490a9d4e:bd8f85e4;AjaxControlToolkit, Version=4.1.50508.0, Culture=neutral, PublicKeyToken=28f01b0e84b6d53e:en-US:0c8c847b-b611-49a7-8e75-2196aa6e72fa:782b16ab","__VIEWSTATEGENERATOR":"C56D694D","__EVENTARGUMENT":"","IsControlPostBackctl01%24HeaderLogo%24HeaderLogoSpan":"1","__VIEWSTATE":"/wEPaA8FDzhkNjYzZWY4NmM2Y2Y4MBgBBR5fX0NvbnRyb2xzUmVxdWlyZVBvc3RCYWNrS2V5X18WCAUYY3RsMDEkTG9naW5TdGF0dXMxJGN0bDAxBRhjdGwwMSRMb2dpblN0YXR1czEkY3RsMDMFFWN0bDAxJFByaW1hcnkkTmF2TWVudQVXY3RsMDEkVGVtcGxhdGVCb2R5JFdlYlBhcnRNYW5hZ2VyMSRnd3BjaVRlYWNoZXJRdWVyaWVzJGNpVGVhY2hlclF1ZXJpZXMkUXVlcnlMaXN0RGlhbG9nBQtjdGwwMSRjdGwxNgUTY3RsMDEkR2VuZXJpY1dpbmRvdwUTY3RsMDEkT2JqZWN0QnJvd3NlcgUZY3RsMDEkT2JqZWN0QnJvd3NlckRpYWxvZ5t6EvKfQQS4e3kXIpgfVSNxCTNO","__WPPS":"s","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries":"1","__SHIFTKEY":"","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciScriptforadtracking%24ciScriptforadtracking":"1","NavMenuClientID":"ctl01_Primary_NavMenu","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenCacheQueryId":"","ctl01%24SearchField%24SearchTerms":"Keyword Search","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24HiddenKeyField1":"","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24mHiddenQueryCached":"False","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml2%24ciNewContentHtml2":"1","ctl01_GenericWindow_ClientState":"","ctl01_Primary_NavMenu_ClientState":"","PageInstanceKey":"1f89a5d5-ce61-4147-9eb5-32cafa6174a6","ctl01_ObjectBrowserDialog_ClientState":"","__CTRLKEY":"","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl04":"277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL9","ctl01%24ScriptManager1":"ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ListerPanel|ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ResultsGrid\x24Sheet0\x24SubmitButton","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input1%24DropDown1":"20","IsControlPostBackctl01%24SocialNetworking%24SocialNetworking":"1","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24Input0%24DropDown1":"Piano","PageIsDirty":"false","__ASYNCPOST":"true","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewShowConsumerAddressCommon%24ciNewShowConsumerAddressCommon":"1","IsControlPostBackctl01%24TemplateBody%24ContentPageFooter1":"1","__EVENTTARGET":"ctl01\x24TemplateBody\x24WebPartManager1\x24gwpciTeacherQueries\x24ciTeacherQueries\x24ResultsGrid\x24Sheet0\x24SubmitButton","ctl01%24TemplateBody%24WebPartManager1%24gwpciTeacherQueries%24ciTeacherQueries%24ResultsGrid%24Sheet0%24ctl01":"277d8abd-5f3f-4b6d-8991-a624daea0d07.FS1.FL7","ctl01_ctl16_ClientState":"","IsControlPostBackctl01%24SearchField":"1","ctl01_TemplateBody_WebPartManager1_gwpciTeacherQueries_ciTeacherQueries_QueryListDialog_ClientState":"","IsControlPostBackctl01%24FooterCommunications%24FooterCommunications":"1","ctl01%24lastClickedElementId":"","TemplateUserMessagesID":"ctl01_TemplateUserMessages_ctl00_Messages","IsControlPostBackctl01%24TemplateBody%24ContentPage1":"1","IsControlPostBackctl01%24TemplateBody%24WebPartManager1%24gwpciNewContentHtml%24ciNewContentHtml":"1","ctl01_ObjectBrowser_ClientState":"","IsControlPostBackctl01%24FooterCommunications%24NewContentHtml":"1","IsControlPostBackctl01%24FooterCopyright%24FooterCopyright":"1"}
headers = {"X-MicrosoftAjax":"Delta=true","Accept":"*/*","X-Requested-With":"XMLHttpRequest","Cache-Control":"no-cache","User-Agent":"Mozilla/5.0 (Windows NT 10.0; WOW64; rv:64.0) Gecko/20100101 Firefox/64.0","Referer":"https://www.mtna.org/MTNA/FindATeacherSearch.aspx?cak=31022421-c202-431a-b5ef-c6d10c90e6be","Connection":"close","Accept-Encoding":"gzip, deflate","Accept-Language":"en-US,en;q=0.5","Content-Type":"application/x-www-form-urlencoded; charset=utf-8"}
cookies = {"AnonymousCartId":"00000000-0000-0000-0000-000000000000","ASP.NET_SessionId":"cxqktlukl4zprfps4wfyex44","_gat":"1","Asi.Web.Browser.CookiesEnabled":"true","_ga":"GA1.2.1639735034.1545046479","_gid":"GA1.2.6620465.1545046479"}
response = session.post("https://www.mtna.org/MTNA/FindATeacherSearch.aspx", data=paramsPost, params=paramsGet, headers=headers, cookies=cookies)
print("Status code: %i" % response.status_code)
print("Response body: %s" % response.content)
But the results leave me on the same page.
Any idea why ?
What cause its different ?

Related

How to call Laravel API from Python correctly

I am using Laravel 8 and Sanctum and attempting to make an API call from Python into the Laravel app.
I don't seem to be sending the request variables correctly from python as they are not being picked up as $request->variable (see "auction_item_id":null in Laravel log below) however they do show up in the request (see below Laravel log).
Is this a problem with the python request header?
Python:
import requests
import re
import json
############## LOGIN ####################
url = 'http://127.0.0.1:8082/api/apilogin'
data = {
'email': 'user#email.com',
'password': 'password'}
r = requests.post(url, data=data)
user = json.loads(r.text)
############## MAKE REST API CALL using login token from prior call ####################
url2 = 'http://127.0.0.1:8082/api/beginlivebiddingforitemAPI'
token = "Bearer " + user["token"]
headers = {"Authorization": token, 'Accept': 'application/json'}
data = {
'auction_id' : 103,
'auction_item_id' : 1224
}
rr = requests.get(url2, data=data, headers=headers)
AuctionBidsController:
public function beginlivebiddingforitem (Request $request){
Log::info("AuctionBidsController#beginlivebiddingforitem", ['auction_item_id' => $request->auction_item_id]);
Api.php:
Route::post('/apilogin', 'UsersController#APIlogin');
Route::middleware(['auth:sanctum'])->group(function () {
Route::get('/beginlivebiddingforitemAPI', 'AuctionBidsController#beginlivebiddingforitem');
});
Laravel log:
[2020-11-06 16:46:44] local.INFO: AuctionBidsController#beginlivebiddingforitem {"auction_item_id":null,"request":{"Illuminate\\Http\\Request":"GET /api/beginlivebiddingforitemAPI HTTP/1.1
Accept: application/json
Accept-Encoding: gzip, deflate
Authorization: Bearer 100|3EjawNPRinapHXxZvDirzyKM73AzeuJC8OTP6xOZ
Connection: keep-alive
Content-Length: 35
Content-Type: application/x-www-form-urlencoded
Host: 127.0.0.1:8082
User-Agent: python-requests/2.23.0
auction_id=103&auction_item_id=1224"}}
^^^^^^^^^^^^^^^
THEY ARE HERE
** But look at "auction_item_id":null **
Here is what a normal request looks like inside laravel:
[2020-11-06 18:34:56] local.INFO: AuctionBidsController#beginlivebiddingforitem {"auction_item_id":"1223","request":{"Illuminate\\Http\\Request":"GET /beginlivebiddingforitem?auction_id=103&auction_item_id=1223 HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: laravel_cookie_consent=1; _ga=GA1.1.1339345323.1594720189; PGADMIN_INT_KEY=129e52c8-dddd-41a0-bc43-53e91dec5402; pga4_session=1c3a240a-46a1-4a9e-8e25-d489bb8ddddddddNltIXmby1M+34KepOYCsGAsY=; PGADMIN_LANGUAGE=en; _gid=GA1.1.1579327561.1604514920; _gat_gtag_UA_170463940_1=1; XSRF-TOKEN=eyJpdiI6ImIxT00yWW54REE2emVQSEpZZzZRRmc9PSIsInZhbHVlIjoiWmlCZGhia1pxeVJ0TmU4NGRBV2xMUURsS21RT0cyWGZUYddddddddddddddddddddddU9oUHg3cGQrdWZhd2gybnRCZ0xUajcwdEhuREh5OE5FcWkzV2tWb3lKOTRwUlljWld4TE9KL3BiMFp0M1dzeklmY2VhaEUiLCJtYWMiOiI0OWZhYmE3ZWM0ZjFhNTU5OWM1ZWE1MmY0MmIyMWQ3YTU1YzE3MmI0NWM1YjY0NTE0MzdhYTVmOWNmODIzMWVlIn0%3D; icollect_session=eyJpdiI6Ik5xSlAzR3NaZUR4QUhxV0RBRFBvd2c9PSIsInZhbHVlIjoianJLbFdVUDJCL2FKZEo5NTFUMElVUFJMRm9BdddddddddddddddHdYL3Arc3Avc1JTcUE5RmNPZ1J2MUVxblFUbmREcHJJeUZJaHZNVzJHTVNhR2p6Snp6M2JkQXJObE9BQU91Si9RbVIzRXMiLCJtYWMiOiI5MzcwMjc5ZWQ0MDE5ZDk2NTVmYmJjNGQ5NWYzMGY2YTVkMDI5YWQ0YTlkNGU1YTUyZTIyYzhkODU4NjNiNDA2In0%3D
Dnt: 1
Host: 127.0.0.1:8082
Referer: http://127.0.0.1:8082/bidliveauctioneer/103
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/86.0.4240.80 Safari/537.36 Edg/86.0.622.43
X-Csrf-Token: wrdZoCDCVeqddddzVxAxByw183bFMuddddP4QBKy
X-Requested-With: XMLHttpRequest
X-Socket-Id: 9357.547174
Cookie: laravel_cookie_consent=1; _ga=; PGADMIN_INT_KEY=; pga4_session=; PGADMIN_LANGUAGE=; _gid=; _gat_gtag_UA_170463940_1=; XSRF-TOKEN=wrdZoCDCVeqddddzVxAxByw183bFMuddddP4QBKy; icollect_session=Qx2QddddNvlIg3ogQ596BdddducrFk4Bmi0nLA89
**note that variables occur right after the GET
GET /beginlivebiddingforitem?auction_id=103&auction_item_id=1223 **
I was using data = data and it's params = data on requests.get(url2, params=data, headers=headers).
Source: https://requests.readthedocs.io/en/master/user/quickstart

Log into asp.net web application with python

I'm trying to log in into one web application with python but very attempt ends with 500 error and the html body shows the error: [HttpAntiForgeryException]. I tried to apply a few solutions from the other questions here but nothing helped. So now, I'm sucked at first request which response Is giving me 500.
import requests
from bs4 import BeautifulSoup
url = "http://localhost:52053/Account/Login"
username = "test#test.sk"
user_password = "pass"
session = requests.Session()
response = session.get(url)
soup = BeautifulSoup(response.content, features="html.parser")
#print(soup)
states = ["__RequestVerificationToken", "Email", "RememberMe"]
login_data = {"username": username, "password": user_password, "Login": "submit"}
headers = {"Host": "localhost:52053",
"Content-Type": "application/x-www-form-urlencoded",
"Connection": "close",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0)",
"Cookie": str(session.cookies.get_dict())}
for state in states: # search for existing aspnet states and get its values
result = soup.find('input', {'name': state})
if not (result is None): # when existent (some may not be needed!)
if state == "Email":
login_data.update({state: login_data["username"]})
else:
login_data.update({state: result['value']})
post_request = session.post(url, headers=headers, data=login_data)
Successful login attempt looks like this.
POST /Account/Login HTTP/1.1
Host: localhost:52053
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sk,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Content-Length: 193
Origin: http://localhost:52053
Connection: close
Referer: http://localhost:52053/Account/Login
Cookie: __RequestVerificationToken=j9yFGpTFSlH5_aQt0k-Gvz10I16TVXbDk31NKPm1HkcWsksUfKXkjL567yFplCS_VovTR7lVuEgNjwgp-EO3RjNj4gQOvNUXnPkjymZx_jA1
Upgrade-Insecure-Requests: 1
__RequestVerificationToken=LjHuOdKSCr1A7KRDNie4GUnCZ3qRwUCdHyLlPYT40DsEB-GNUvEKxe5nvZWf5gZ4ZflwI43xGWPyYu8GI15wroEg9WRRVtSzZ9-KY9Mu_JA1&Email=test%40test.sk&Password=pass&RememberMe=false
Following response is:
HTTP/1.1 302 Found
Cache-Control: no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Expires: -1
Location: /
Server: Microsoft-IIS/10.0
X-AspNetMvc-Version: 5.2
X-AspNet-Version: 4.0.30319
Set-Cookie: .AspNet.ExternalCookie=; path=/; expires=Thu, 01-Jan-1970 00:00:00 GMT
Set-Cookie: .AspNet.ApplicationCookie=KCLm03FHj8v_6rIpTzBTm7EzEtzpKmIz1Z9_z29wycUSqUVyKbGEmptXUwG41MqNOMR7Vbeq2u576ijazupNLffLP-Ua0n60aLmnVSDsLsdTqYT7jjqyGPw1Ppp8AnIDs3sdefmksazX2UvKTxzxRBufFCoxtCJx51mWtBv7v0JzUeC1hnfu1AIJ7GH_8T59KD3iv0hRSHDqlWHlkWzyN1Xt0m5ixC14e4eC2YxEm3_acy96atB2Jv5u0HREPzssLmywuzj6sLa9cHCllTG2gMVWvHA3IDhCWu7Ojf8BO02Eml3pPM5QTJ-sq540fcj9QyELayUOwBZWffSgsJeq8mlt3FupQcJ-JTJxDzAsDc4Cmk-BcvYSfpAJq4SdR-Y4mTN_6vu-wwAOLZPSgh-5K7guWmZ3VfRitZHXd_rvTEmMiVrgHFTEQAkUYu4zTSupxRplTtKb1VSDs0Nc1uEos2z0_aw-nBbRBrTPpvmqGok
Auth flow continues with this request. I'm not trying to sent this request yet (I put it here just for better imagination):
GET / HTTP/1.1
Host: localhost:52053
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0) Gecko/20100101 Firefox/70.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: sk,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://localhost:52053/Account/Login
Connection: close
Cookie: __RequestVerificationToken=j9yFGpTFSlH5_aQt0k-Gvz10I16TVXbDk31NKPm1HkcWsksUfKXkjL567yFplCS_VovTR7lVuEgNjwgp-EO3RjNj4gQOvNUXnPkjymZx_jA1;
.AspNet.ApplicationCookie=gvv113IJhtdaOhdc0Rz2N--5Ob18W6gS64J3wtOJggRTqE70h-8HyBGQAmLvSM2qCV2e-dXR2Uto-BktD6NmNz6dJtxckIYasPOfqodDNZX33YJxNEDg7a64LPi1bNnmrnvQcOHAceQNqZDykXrhFm55dqoo1oZnJHfZQnltwqAdg7DGO31PZpzu-GAZh2_gzuxd_saJdS09ZZQrc9h7WiU2ONqeya87pSAN7ZyHQ_XvsU5cUwDGq7FWLpzlIeeZWkay6iWVmCSwNEofpdVsb880P3XZnFKEj2SW2PfazdNLfgy86YNjkoD6_3Vb1BLirRoSP0XIQMcs2F_CzgXkxD5GvDray8TPYqcQJ4L2fikReUJHadx9fFnslF2BFcnKYC8D-Xusrda_5r-CQoQ4SzAe2Cqn0h1NYHxS1wsxt35neC5RuQ3geadAEEghjrSSVhSl8jCfACtQtcBeNL2x_m6I9L3XJCjMpzJjtP6up3E
Upgrade-Insecure-Requests: 1
Next response is just kind of 200 - you are in.
So my problem is that the response from the first request is failing. Is someone able to see some mistake or did I forgot something?
Failed response from the first request call:
HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html; charset=utf-8
Server: Microsoft-IIS/10.0
X-AspNet-Version: 4.0.30319
X-SourceFiles: =?UTF-8?B?QzpcVXNlcnNcUENBZG1pbmlzdHJhdG9yXERlc2t0b3BccGVuIHRlc3RpbmdcU2VjdXJpdHlXb3Jrc2hvcC1EVldBLW1hc3RlclxkdndhLXRyYWluaW5nXGR2d2EtdHJhaW5pbmdcQWNjb3VudFxMb2dpbg==?=
If I try to pint request headers and login_data, result is:
print(post_request.request.headers)
{'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:70.0)', 'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Connection': 'close', 'Host': 'localhost:52053', 'Content-Type': 'application/x-www-form-urlencoded', 'Cookie': "{'__RequestVerificationToken': 'yg-7mFRyZiONwsZ2dIVkIIW5tB7gSL_sazgphg-VuW2OpNNRRkxmLH-9SZJXiN9whUC_BYTo8RgsiDrVjcYtLEf9anW56rVwZ2RQPzxHA481'}", 'Content-Length': '249'}
print(login_data)
{'username': 'test#test.sk', 'password': 'pass', 'Login': 'submit', '__RequestVerificationToken': '14OuwaRqldlGKi93C91zf6QD_ouOorHBDe63s4KgfP3gbt85V0QMy2X5OMwWAo1TUrD8zJ-zoZbXLPpgDI_wrxVZv3ceYNos_e5_elFhVt01', 'Email': 'test#test.sk', 'RememberMe': 'true', 'Password': 'pass'}
I just find the solution out.
Requests could handle all headers by itself (and my headers was, for some reason, causing errors), it was redirecting my request to 200 response so I didn't instantly saw, that it is actually working and catching 302 response.
I found out by printing:
print(post_request.history)
Which gave me <[302]>
Now, when I know, there is a redirection, I just have to allow_redirects=False and now I'm able to catch my set-cookie header
Full code, witch is getting an expected responses is:
import requests
from bs4 import BeautifulSoup
url = "http://localhost:52053/Account/Login"
username = "test#test.sk"
user_password = "pass"
session = requests.Session()
response = session.get(url)
soup = BeautifulSoup(response.content, features="html.parser")
#print(soup)
states = ["__RequestVerificationToken", "Email", "RememberMe"]
login_data = {"username": username, "password": user_password, "Login": "submit"}
for state in states: # search for existing aspnet states and get its values
result = soup.find('input', {'name': state})
if not (result is None): # when existent (some may not be needed!)
if state == "Email":
login_data.update({state: login_data["username"]})
else:
login_data.update({state: result['value']})
post_request = session.post(url, data=login_data, allow_redirects=False)
print(login_data)
#the code below is testing, if the HttpAntiForgeryException is in code
if "HttpAntiForgeryException" not in post_request.text:
print(post_request.headers)
else:
print("antiforgery")

Return 302 in Web Crawler

After I simulate to log in, when I try to post the original website, it returns 302. When I open the original website in Chrome, it returns 415.
I tried several ways:
session.post(url,headers = headers,data = data)
requests.post(url,headers = headers,data = data)
urllib.request.urlopen.read(url).decode()
import requets
import json
header = {'Host': 'sty.js118114.com:8080',
'Connection': 'keep-alive',
'Content-Length': '8188',
'Accept': '*/*',
'X-Requested-With': 'XMLHttpRequest',
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36',
'Content-Type': 'text/plain;charset=UTF-8;application/xml',
'Origin': 'http://sty.js118114.com:8080',
'Referer':
'http://sty.js118114.com:8080/Report/report/movecar_list.html',
'Accept-Encoding': 'gzip, deflate',
'Accept-Language': 'zh-CN,zh;q=0.9',
'Cookie': cookie_name + '=' + cookie_value
}
data = {"calling_no":"","begin_time":"","end_time":"","called_car_no":""}
res = requests.post(target,data = json.dumps(data),headers = header)
print(res.content.decode())
I expect the content must be the json version or html version so that I can use re model or xpath to get the infomation I want.(without any redirects
Lastly, I provide the necessary infomation about the problem:
Chrome Network
General
Request URL: http://sty.js118114.com:8080/Report/movecar/list/1/10
Request Method: POST
Status Code: 200 OK
Remote Address: 127.0.0.1:8888
Referrer Policy: no-referrer-when-downgrade
Response Headers
Content-Length: 8150
Content-Type: application/json;charset=UTF-8
Date: Thu, 22 Aug 2019 00:47:51 GMT
Server: Apache-Coyote/1.1
Request Headers
Accept: */*
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9
Content-Length: 66
Content-Type: text/plain;charset=UTF-8;
Cookie: JSESSIONID=0A474B00017BFFD89A515B336F482905
Host: sty.js118114.com:8080
Origin: http://sty.js118114.com:8080
Proxy-Connection: keep-alive
Referer: http://sty.js118114.com:8080/Report/report/movecar_list.html
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36
X-Requested-With: XMLHttpRequest
Request Payload
{calling_no: "", begin_time: "", end_time: "", called_car_no: ""}
begin_time: ""
called_car_no: ""
calling_no: ""
end_time: ""
Fiddler Inspectors Raw
POST http://sty.js118114.com:8080/Report/movecar/list/1/10 HTTP/1.1
Host: sty.js118114.com:8080
Connection: keep-alive
Content-Length: 66
Accept: */*
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36
(KHTML, like Gecko) Chrome/76.0.3809.100 Safari/537.36
Content-Type: text/plain;charset=UTF-8;
Origin: http://sty.js118114.com:8080
Referer: http://sty.js118114.com:8080/Report/report/movecar_list.html
Accept-Encoding: gzip, deflate
Accept-Language: zh-CN,zh;q=0.9
Cookie: JSESSIONID=0A474B00017BFFD89A515B336F482905
{"calling_no":"","begin_time":"","end_time":"","called_car_no":""}
Response Raw
HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Content-Type: application/json;charset=UTF-8
Date: Thu, 22 Aug 2019 00:27:59 GMT
Content-Length: 8150

Post data send in python using socket

I write a code who has to click a button on website, by sending a POST request. I use sockets to this. Here is my code:
import sys
import socket
import time
import socks
red = "\033[1;31m"
green = "\033[1;32m"
yellow = "\033[1;33m"
blue = "\033[1;34m"
defcol = "\033[0m"
s = socks.socksocket()
s.set_proxy(socks.SOCKS5, "209.33.198.133", 9999)
ipaddr = socket.gethostbyname('mclista.pl')
#ipaddr = 'mclista.pl'
s.connect(("149.202.218.96", 80))
header = ("""
POST /json/daj_diax/ HTTP/1.1
Host: www.mclista.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: https://www.google.com
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 68
Connection: keep-alive
Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=65d10295bd238ea3c81f19b3e02d073c; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22d1e9deec40aa713329f98ddd11ce042f%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541242717%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D913ae0005d8fb07d1b6dc486b7fc9297; _gat=1
""")
print(yellow + header)
request = header
s.sendall(request.encode('utf-8'))
response = s.recv(4096)
time.sleep(2)
print('\n\n\n\n\n\n\n' + blue + str(response) + '\n')
s.close()
My question is, why on send i getting it response:
HTTP/1.1 400 Bad Request
Date: Sat, 03 Nov 2018 11:20:02 GMT
Server: Apache
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
Anyone know, how to fix it? Header is good, i copy it from firefox network tools. I need to send POST request, not GET. Code written in python 3.7
On i send it post from firefox, i getting normal response:
status: "ok"
Now i change a header to:
header = ("""POST /json/daj_diax/ HTTP/1.1 Host: www.mclista.pl User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0 Accept: application/json, text/javascript, */*; q=0.01 Accept-Language: pl,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Referer: http://www.mclista.pl/42204 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Requested-With: XMLHttpRequest Content-Length: 68 Connection: keep-alive Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1""")
and to:
header = ("""POST /json/daj_diax/ HTTP/1.1
Host: www.mclista.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://www.mclista.pl/42204
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 68
Connection: keep-alive
Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1""")
doesn't working now too. On first header the same response, on secound header no response, recive sending a empty byte (b'')
You may need to use CRLF \r\n:
header = ("POST /json/daj_diax/ HTTP/1.1\r\n"
"Host: www.mclista.pl\r\n"
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0\r\n"
"Accept: application/json, text/javascript, */*; q=0.01\r\n"
"Accept-Language: pl,en-US;q=0.7,en;q=0.3\r\n"
"Accept-Encoding: gzip, deflate\r\n"
"Referer: http://www.mclista.pl/42204\r\n"
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8\r\n"
"X-Requested-With: XMLHttpRequest\r\n"
"Content-Length: 68\r\n"
"Connection: keep-alive\r\n"
"Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1\r\n\r\n")

ssl connection using python 2.7.13 requests

I tried to use python requests package to connect over SSL and i used below code.
resp = requests.get(addr , auth=HttpNtlmAuth(userFile[unum],passFile[pnum]) , headers = {'User-Agent':'Mozilla/4.01 (compatible; MSIE 6.0; Windows NT 5.1)'} , verify = False)
but every time i got below error.
("bad handshake: SysCallError(-1, 'Unexpected EOF')",)
UPDATE 1:
I dump host certificate and save that to c.cert and used below code.
resp = requests.get(addr , auth=HttpNtlmAuth(userFile[unum],passFile[pnum]) , headers = {'User-Agent':'Mozilla/4.01 (compatible; MSIE 6.0; Windows NT 5.1)'} , cert = '/home/user/Desktop/c.cert' , verify = '/home/user/Desktop/c.cert')
and new error.
[('PEM routines', 'PEM_read_bio', 'no start line'), ('SSL routines', 'SSL_CTX_use_PrivateKey_file', 'PEM lib')]
UPDATE 2 :
Here is what Burp Suite captured.
(First , session creation):
GET /PATH HTTP/1.1
Host: HOST
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Cookie: ASPSESSIONIDCQCTCTCQ=EPEHGDDAKOINODAJJKCELEOK
DNT: 1
Connection: close
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
(Second , when use credentials):
GET /PATH HTTP/1.1
Host: HOST
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Cookie: ASPSESSIONIDCQCTCTCQ=EPEHGDDAKOINODAJJKCELEOK
DNT: 1
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
Authorization: NTLM TlRMTVNTUAABAAAAB4IIAAAAAAAAAAAAAAAAAAAAAAA=
Connection: close
(Third , credentials was wrong)
GET /PATH HTTP/1.1
Host: HOST
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Firefox/52.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Cookie: ASPSESSIONIDCQCTCTCQ=EPEHGDDAKOINODAJJKCELEOK
DNT: 1
Upgrade-Insecure-Requests: 1
Cache-Control: max-age=0
Authorization: NTLM TlRMTVNTUAADAAAAGAAYAGAAAACIAIgAeAAAAAAAAABAAAAACgAKAEAAAAAWABYASgAAAAAAAAAAAAAABYIIAGEAZABtAGkAbgBXAE8AUgBLAFMAVABBAFQASQBPAE4Ao8+kG6lMZcLmQys5IUkpayq+W1VFZPkjUQjxBhzugt88vU6RR3wahQEBAAAAAAAAgPonWPz30gGr58MfeUgvgQAAAAACABIAVwBFAEIAUwBFAFIAVgBFAFIAAQASAFcARQBCAFMARQBSAFYARQBSAAQAEgBXAEUAQgBTAEUAUgBWAEUAUgADABIAVwBFAEIAUwBFAFIAVgBFAFIAAAAAAA==
Connection: close
when using URL without HTTPS it gives me page with below custom error
Bad Request (Invalid Hostname)
Maybe you need to create a SSL context, and pass it to requests.
import ssl
context = ssl.SSLContext(ssl.PROTOCOL_TLSv1_2) #if you need TLS v1.2
and then, pass this context var to the requests method. If an optional keyword argument, name context. So, the call will be like this:
resp = requests.get(addr, ...., context=context)

Categories