Similar to a question asked here: Http Redirection code 3XX in python requests. I do also not receive redirection when I'm trying to post a form with python's requests.
To bypass same origin policy, my goal is it to proxy (redirect) an internal site with my flask application through the following code:
method_requests_mapping = {
'GET': requests.get,
'HEAD': requests.head,
'POST': requests.post,
'PUT': requests.put,
'DELETE': requests.delete,
'PATCH': requests.patch,
'OPTIONS': requests.options,
}
#bp.route('/<path:url>', methods=method_requests_mapping.keys())
def proxy(url):
url='https://intern.something.com/'+url
username=session['username']
password=session['password']
requests_function = method_requests_mapping[flask.request.method]
request = requests_function(url, stream=True, params=flask.request.args,auth=(username, password),allow_redirects=False)
response = flask.Response(flask.stream_with_context(request.iter_content()),
content_type=request.headers['content-type'],
status=request.status_code, )
response.headers['Access-Control-Allow-Origin'] = '*'
print(request.history)
print(request.cookies)
print(request.status_code)
return response
If I am trying to use the site without my flask proxy network analysis shows me this:
Request:
Host: intern.something.com
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate, br
Referer: https://intern.something.com/contract_config_edit.php4?Contract_ID=1463234
Content-Type: application/x-www-form-urlencoded
Content-Length: 4024
Authorization: Basic YWhvZWhuZTpLYXR6ZTc0MzYh
Connection: keep-alive
Cookie: PHPSESSID=kr9am6tpid67ikct3up67f03h0
Upgrade-Insecure-Requests: 1
Answer:
HTTP/1.1 302 Found
Date: Wed, 02 Jan 2019 07:50:31 GMT
Server: Apache/2.2.3 (Red Hat)
X-Powered-By: PHP/5.1.6
Expires: Thu, 19 Nov 1981 08:52:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre- check=0
Pragma: no-cache
Location: https://intern.something.com /contract_show.php4?Contract_ID=1463234
Content-Length: 0
Connection: close
Content-Type: text/html
But if I do it with the proxy it seems not to work correctly:
Request:
Host: 10.146.177.18:7000
User-Agent: Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:64.0) Gecko/20100101 Firefox/64.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: de,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://10.146.177.18:7000/backoffice/contract /contract_config_edit.php4?Contract_ID=1463234
Content-Type: application/x-www-form-urlencoded
Content-Length: 4024
Authorization: Basic RWluaG9ybjpGZXVlcnphbmdlbmJvaGxlNTU0ISE/
Connection: keep-alive
Cookie: _pk_id.7.1c19=5f552d1eb2170bab.1546180080.2.1546185355.1546184002.; session=.eJwtj1FKxTAQRddivt9Hkk5mJm8LLqJMJjdUxFbaPgTFvVvRz3PhwD1fYR47jiXcz_2BW5hfergHjTrIMlHxOrgSWh- NxNU0e67iEch5SpqaQaRxSz4oo1dzcRLNXcQ5Ugd4yMhVS8m9oVMt3pJpacw2UUEtrUfXaNQ7C DJaEw234Mc-5nN7xXr9YWdTBpJAY-KRMBVCKYYqrPEyJFav-fLe7Tg- tv234tnOTwhN_HTtjwP7X1z6p9XecKEtG5YV4fsHxkJOZg.Dw34rg.p2bNxLLF26aIXxth9VN7 BHA5x4U
Upgrade-Insecure-Requests: 1
Answer:
HTTP/1.0 200 OK
Content-Type: text/html
Access-Control-Allow-Origin: *
Vary: Cookie
Connection: close
Server: Werkzeug/0.14.1 Python/3.5.2
Date: Wed, 02 Jan 2019 08:15:38 GMT
Maybe it could be a problem with the cookies though it seems in the console it sends the correct cookie:
10.146.177.49 - - [02/Jan/2019 09:15:38] "POST /backoffice/contract/contract_config_edit.php4?Contract_ID=1463234 HTTP/1.1" 200 -
<RequestsCookieJar[<Cookie PHPSESSID=saqjj7n6m61aee19k3pe6moaf4 for intern.something.com/>]>
Does anyone know what the problem is here?
Related
I want to build a simple python program that gets updates for tracking numbers from UPS, I couldn't get an account number with them so I can't use their API. I decided to try web scraping.
Here's an example of a tracking number:
https://www.ups.com/track?loc=en_US&tracknum=1Z0X118AYW08592000&requester=WT/trackdetails
I want to get the scheduled delivery date, the problem is that what the requests module scrapes and what shows when I view the page source doesn't get all the information inside a tag called app-root. That's where the delivery date is.
I found a similar post that solves this problem with FedEx, but I can't get it to work with the ups website: Parsing HTML does not output desired data(tracking info for FedEx)
I installed an extension called HTTP Trace that shows all the requests that go through my server, I can't find the one that matches UPS, this is what I got from the extension when I searched for the tracking number, any ideas what I can do here?
https://wwwapps.ups.com/WebTracking/track?loc=en_IL
HTMLVersion: 5.0
loc: en_IL
track.x: Track
trackNums: 1Z0X118AYW08592000
ups-search: 1Z0X118AYW08592000
POST https://wwwapps.ups.com/WebTracking/track?loc=en_IL
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
HTTP/1.1 302 Moved Temporarily
Redirect to: https://www.ups.com/track?loc=en_IL&tracknum=1Z0X118AYW08592000&requester=WT
Server: Apache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Cache-Control: no-store, no-cache
Pragma: no-cache
Location: https://www.ups.com/track?loc=en_IL&tracknum=1Z0X118AYW08592000&requester=WT
Content-Length: 365
Content-Type: text/html
Date: Thu, 14 Jan 2021 00:32:34 GMT
Connection: keep-alive
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=164
Server-Timing: origin; dur=23
Debug-AK-TLS: No bypass
GET https://www.ups.com/track?loc=en_IL&tracknum=1Z0X118AYW08592000&requester=WT
Upgrade-Insecure-Requests: 1
Content-Type: application/x-www-form-urlencoded
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
HTTP/1.1 200 OK
Server: Apache
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Strict-Transport-Security: max-age=31536000; includeSubDomains
Cache-Control: no-store, no-cache
Pragma: no-cache
Content-Type: text/html; charset=utf-8
Content-Encoding: gzip
Debug-AK-TLS: No bypass
X-Akamai-Transformed: 9 9152 0 pmb=mTOE,1mRUM,1
Date: Thu, 14 Jan 2021 00:32:34 GMT
Content-Length: 10947
Connection: keep-alive
Vary: Accept-Encoding
Server-Timing: cdn-cache; desc=MISS
Server-Timing: edge; dur=182
Server-Timing: origin; dur=201
https://www.facebook.com/tr/?id=969628123173894&ev=PageView&dl=https%3A%2F%2Fwww.ups.com%2Ftrack%3Floc%3Den_IL%26tracknum%3D1Z0X118AYW08592000%26requester%3DWT%2Ftrackdetails&rl=https%3A%2F%2Fwww.ups.com%2F&if=false&ts=1610584355509&sw=1920&sh=1080&v=2.9.32&r=stable&a=tmtealium&ec=0&o=30&fbp=fb.1.1598067407332.38393503&it=1610584355413&coo=false&dpo=LDU&dpoco=0&dpost=0&rqm=GET
GET https://www.facebook.com/tr/?id=969628123173894&ev=PageView&dl=https%3A%2F%2Fwww.ups.com%2Ftrack%3Floc%3Den_IL%26tracknum%3D1Z0X118AYW08592000%26requester%3DWT%2Ftrackdetails&rl=https%3A%2F%2Fwww.ups.com%2F&if=false&ts=1610584355509&sw=1920&sh=1080&v=2.9.32&r=stable&a=tmtealium&ec=0&o=30&fbp=fb.1.1598067407332.38393503&it=1610584355413&coo=false&dpo=LDU&dpoco=0&dpost=0&rqm=GET
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept: image/avif,image/webp,image/apng,image/*,*/*;q=0.8
HTTP/1.1 302
Redirect to: https://cx.atdmt.com/?c=1479770850078954307&f=AYzL_IHfyiIJ9HIa7oqq8XcmRPtLo6M0aKkForULuTS_d5qgkpmUtO1x4Rmi3jkdZ4EPRHG7qxKZDTiWb-BA5MYf&id=969628123173894&l=3&v=0
cache-control: no-cache, no-store, must-revalidate
pragma: no-cache
expires: 0
date: Thu, 14 Jan 2021 00:32:36 GMT
location: https://cx.atdmt.com/?c=1479770850078954307&f=AYzL_IHfyiIJ9HIa7oqq8XcmRPtLo6M0aKkForULuTS_d5qgkpmUtO1x4Rmi3jkdZ4EPRHG7qxKZDTiWb-BA5MYf&id=969628123173894&l=3&v=0
content-type: text/plain
content-length: 0
server: proxygen-bolt
alt-svc: h3-29=":443"; ma=3600,h3-27=":443"; ma=3600
GET https://cx.atdmt.com/?c=1479770850078954307&f=AYzL_IHfyiIJ9HIa7oqq8XcmRPtLo6M0aKkForULuTS_d5qgkpmUtO1x4Rmi3jkdZ4EPRHG7qxKZDTiWb-BA5MYf&id=969628123173894&l=3&v=0
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/87.0.4280.88 Safari/537.36
Accept: image/avif,image/webp,image/apng,image/*,*/*;q=0.8
HTTP/1.1 200
x-fb-rlafr: 0
content-type: image/gif
date: Wed, 13 Jan 2021 16:32:37 PST
x-content-type-options: nosniff
report-to: {"group":"coep_report","max_age":86400,"endpoints":[{"url":"https:\/\/www.facebook.com\/browser_reporting\/"}]}
cache-control: public, max-age=0
content-encoding: br
x-frame-options: DENY
cross-origin-resource-policy: cross-origin
expires: Wed, 13 Jan 2021 16:32:37 PST
vary: Accept-Encoding
cross-origin-embedder-policy-report-only: require-corp;report-to="coep_report"
pragma: public
x-fb-debug: ySiesinmQSMWtIWGg5+rMp+g66R70GGiqJJC3M0DowZMGuFf14OidRiX02DfG99gXxjUSjCaEtHosxh/9tl/hQ==
I honestly do not believe this is possible. I checked how UPS loads its sites, and it seems to load the frontend first like this
Get request to website preview
then goes in to the api to grab the dates. For example, the delivered on date is stored in this api link ("https://www.ups.com/track/api/Track/GetStatus?loc=en_US") which needs a bunch of headers and has some akamai/security cookies (which may prevent you from scraping it).
If you really do not want to use an api, I would suggest using something like Selenium if you do not need it to be quick/do not have many links to work with.
Your only choice is either Selenium OR API but that comes with a hitch. For what I can tell on the UPS website, their API only allows for queries at night. They only want "emergencies" to be hitting their API, which is preposterous since I would imagine web requests are hitting the same API.
The essence of the problem is that I used to connect to websocket by sending Origin, User-Agent, Cookies and the connection worked, now the domain owner decided to change it to the domain of the websocket and put cloudflare protection there, after which my connection method does not work . Advise some method, or information on how to connect to a web socket with cloudflare. Help me pls!!
Example of my code:
import websocket
import json
import time
import traceback
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173', 'cookie': '__cfduid=da97b059db0292806e2affdf9c3f4fd8b1593022325; _csrf=i8W6njc7hUXMOf4iQjiAxKg1; language=en; theme=darkTheme; pro_version=false; csgo_ses=1489162147d69debd9fe5d0ea2e445c87a117578d774502172d7151b89b82f7f; steamid=76561199068891508; avatar=https://steamcdn-a.akamaihd.net/steamcommunity/public/images/avatars/fe/fef49e7fa7e1997310d705b2a6158ff8dc1cdfeb_medium.jpg; username=andrewcrook232; thirdparty_token=06d04856ce6e334aa1368696df775e7ba0b1b898db135b0af0b5dc0fe001dd55; user_type=old; sellerid=6721648; type_device=desktop', 'origin': 'https://cs.money'}
def start_ws():
try:
ws = websocket.WebSocketApp("wss://ws.cs.money/ws", on_message = on_message, cookie = json.dumps(headers))
print("Connected")
while True:
ws.run_forever(ping_timeout=20)
print("Reload")
time.sleep(20)
except:
print(traceback.format_exc())
def on_message(ws, message):
try:
print(message)
except:
print(traceback.format_exc())
if __name__ == "__main__":
start_ws()
Below is all the information that I got with Chrome Inspector (f12) -> Network -> WS -> headers, this information should be more than enough to successfully join WSS.
Request URL: wss://ws.cs.money/ws
Request Method: GET
Status Code: 101 Switching Protocols
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
CF-Cache-Status: DYNAMIC
CF-RAY: 5a886ad37f4b8ac6-KBP
cf-request-id: 038921182700008ac6798a2200000001
Connection: upgrade
Date: Wed, 24 Jun 2020 18:12:29 GMT
Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
Sec-WebSocket-Accept: zrH4CEKXm3BY5z77HroJDqGgYSc=
Server: cloudflare
Strict-Transport-Security: max-age=31536000; includeSubDomains; preload
Upgrade: websocket
X-Content-Type-Options: nosniff
Accept-Encoding: gzip, deflate, br
Accept-Language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: no-cache
Connection: Upgrade
Host: ws.cs.money
Origin: https://cs.money
Pragma: no-cache
Sec-WebSocket-Extensions: permessage-deflate; client_max_window_bits
Sec-WebSocket-Key: GXVT8QewAgPEZDEZZ+x3dA==
Sec-WebSocket-Version: 13
Upgrade: websocket
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173
Also additional page data:
Request URL: https://cs.money/
Request Method: GET
Status Code: 200
Remote Address: 104.20.76.156:443
Referrer Policy: no-referrer-when-downgrade
alt-svc: h3-27=":443"; ma=86400, h3-28=":443"; ma=86400, h3-29=":443"; ma=86400
cf-cache-status: DYNAMIC
cf-ray: 5a886ab5adac8aea-KBP
cf-request-id: 038921058800008aea96109200000001
content-encoding: br
content-security-policy: script-src 'self' cs.money dev.csgo.trade gleam.io www.am4charts.com translate.google.com translate.googleapis.com www.googletagmanager.com www.googleoptimize.com www.google-analytics.com connect.facebook.net https://vk.com 'unsafe-inline' top-fwz1.mail.ru 'unsafe-eval' api.usersnap.com cdn.usersnap.com cs.money mc.yandex.ru diffuser-cdn.app-us1.com diffuser-cdn.app-us1.com prism.app-us1.com trackcmp.net api.basisid.com https://cdn.amplitude.com sc-static.net support.cs.money embed-sandbox.bridgerpay.com embed.bridgerpay.com cs.money; worker-src 'self' data: blob: cs.money; object-src cs.money dota.money; media-src cs.money dota.money; frame-src cs.money dota.money onesignal.com https://*.com https://*.ru https://*.ua http://www.youtube.com
content-type: text/html; charset=utf-8
date: Wed, 24 Jun 2020 18:12:25 GMT
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
set-cookie: user_type=old; Path=/
set-cookie: language=en; Max-Age=8640000; Domain=cs.money; Path=/; Expires=Fri, 02 Oct 2020 18:12:25 GMT
set-cookie: language=en; Max-Age=8640000; Domain=.cs.money; Path=/; Expires=Fri, 02 Oct 2020 18:12:25 GMT
set-cookie: sellerid=6721648; Max-Age=8640000; Domain=cs.money; Path=/; Expires=Fri, 02 Oct 2020 18:12:25 GMT
set-cookie: pro_version=false; Max-Age=8640000; Domain=cs.money; Path=/; Expires=Fri, 02 Oct 2020 18:12:25 GMT
status: 200
strict-transport-security: max-age=31536000; includeSubDomains; preload
x-cache-status: BYPASS
x-content-type-options: nosniff
x-dns-prefetch-control: off
x-download-options: noopen
x-frame-options: SAMEORIGIN
x-powered-by: PHP 4.1.0
x-xss-protection: 1; mode=block
:authority: cs.money
:method: GET
:path: /
:scheme: https
accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
accept-encoding: gzip, deflate, br
accept-language: ru-RU,ru;q=0.9,en-US;q=0.8,en;q=0.7
cache-control: max-age=0
cookie: __cfduid=da97b059db0292806e2affdf9c3f4fd8b1593022325; _csrf=i8W6njc7hUXMOf4iQjiAxKg1; language=en; theme=darkTheme; pro_version=false; csgo_ses=1489162147d69debd9fe5d0ea2e445c87a117578d774502172d7151b89b82f7f; steamid=76561199068891508; avatar=https://steamcdn-a.akamaihd.net/steamcommunity/public/images/avatars/fe/fef49e7fa7e1997310d705b2a6158ff8dc1cdfeb_medium.jpg; username=andrewcrook232; thirdparty_token=06d04856ce6e334aa1368696df775e7ba0b1b898db135b0af0b5dc0fe001dd55; user_type=old; sellerid=6721648; type_device=desktop
referer: https://steamcommunity.com/openid/login?openid.mode=checkid_setup&openid.ns=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0&openid.identity=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.claimed_id=http%3A%2F%2Fspecs.openid.net%2Fauth%2F2.0%2Fidentifier_select&openid.return_to=https%3A%2F%2Fauth.dota.trade%2Flogin%2Fcallback%3FredirectUrl%3Dhttps%3A%2F%2Fcs.money%26callbackUrl%3Dhttps%3A%2F%2Fcs.money%2Flogin&openid.realm=https%3A%2F%2Fauth.dota.trade
sec-fetch-dest: document
sec-fetch-mode: navigate
sec-fetch-site: cross-site
sec-fetch-user: ?1
upgrade-insecure-requests: 1
user-agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.138 Safari/537.36 OPR/68.0.3618.173
I'm not sure about the real reason, but it seems that your code has some bug.
If you need to build a websocket connection with customized header, you pass it to header parameter, instead of json dump it.
ws = websocket.WebSocketApp("wss://ws.cs.money/ws",
on_message = on_message,
cookie = json.dumps(headers))
should be
cookie_string = headers['cookie']
del headers['cookie']
header_without_cookie = headers
ws = websocket.WebSocketApp("wss://ws.cs.money/ws",
on_message = on_message,
header = header_without_cookie,
cookie = cookie_string)
websocket-client documentation is missing, maybe you can read source code about usage
https://github.com/websocket-client/websocket-client/blob/2222f2c49d71afd74fcda486e3dfd14399e647af/websocket/_app.py
I'm trying to download a file from a website using Python's request module.
However the site will allow me to download the file only if the download link is clicked directly from the download page.
So using requests, I tried hitting the download page's URL first using requests.get() then proceeding to download the file. But unfortunately this doesn't seem to work. A text asking me to open the download page first simply gets written into file.torrent"
import requests
def download(username, password):
with requests.Session() as session:
session.post('https://website.net/forum/login.php', data={'login_username': username, 'login_password': password})
# Download page URL
requests.get('https://website.net/forum/viewtopic.php?t=2508126')
# The download URL itself
response = requests.get('https://website.net/forum/dl.php?t=2508126')
with open('file.torrent', 'wb') as f:
f.write(response.content)
download(username='XXXXX', password='YYYYY')
Response when downloading directly from the download page (works) :
General :
Request URL: https://website.net/forum/dl.php?t=2508126
Request Method: GET
Status Code: 200 OK
Remote Address: 185.37.128.136:443
Referrer Policy: no-referrer-when-downgrade
Response Headers :
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Content-Disposition: attachment; filename="[website.net].t2508126.torrent"
Content-Length: 33641
Content-Type: application/x-bittorrent; name="[website.net].t2508126.torrent"
Date: Thu, 14 Feb 2019 07:57:08 GMT
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Thu, 14 Feb 2019 07:57:09 GMT
Pragma: no-cache
Server: nginx
Set-Cookie: bb_dl=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; path=/forum/; domain=.website.net
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: bb_t=a%3A3%3A%7Bi%3A2507902%3Bi%3A1550052944%3Bi%3A2508011%3Bi%3A1550120230%3Bi%3A2508126%3Bi%3A1550125516%3B%7D; bb_data=1-27969311-wXVPJGcedLE1I2mM9H0u-3106784170-1550128652-1550131012-3061288864-1; bb_dl=2508126
Host: website.net
Referer: https://website.net/forum/viewtopic.php?t=2508126
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3701.0 Safari/537.36
Query String Parameters :
t: 2508126
Response when opening the download link on it's own (doesn't work) :
General :
Request URL: https://website.net/forum/dl.php?t=2508126
Request Method: GET
Status Code: 200 OK
Remote Address: 185.37.128.136:443
Referrer Policy: no-referrer-when-downgrade
Response Headers :
Cache-Control: no-store, no-cache, must-revalidate
Cache-Control: post-check=0, pre-check=0
Content-Type: text/html; charset=windows-1251
Date: Thu, 14 Feb 2019 08:03:29 GMT
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Last-Modified: Thu, 14 Feb 2019 08:03:29 GMT
Pragma: no-cache
Server: nginx
Transfer-Encoding: chunked
Request Headers :
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Connection: keep-alive
Cookie: bb_t=a%3A3%3A%7Bi%3A2507902%3Bi%3A1550052944%3Bi%3A2508011%3Bi%3A1550120230%3Bi%3A2508126%3Bi%3A1550125516%3B%7D; bb_data=1-27969311-wXVPJGcedLE1I2mM9H0u-3106784170-1550128652-1550131390-3061288864-1
Host: website.net
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3701.0 Safari/537.36
Query String Parameters :
t: 2508126
This works for me:
data={'login_username': username, 'login_password': password, 'login': ''}
and using session.get() instead of requests.get()
I write a code who has to click a button on website, by sending a POST request. I use sockets to this. Here is my code:
import sys
import socket
import time
import socks
red = "\033[1;31m"
green = "\033[1;32m"
yellow = "\033[1;33m"
blue = "\033[1;34m"
defcol = "\033[0m"
s = socks.socksocket()
s.set_proxy(socks.SOCKS5, "209.33.198.133", 9999)
ipaddr = socket.gethostbyname('mclista.pl')
#ipaddr = 'mclista.pl'
s.connect(("149.202.218.96", 80))
header = ("""
POST /json/daj_diax/ HTTP/1.1
Host: www.mclista.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: https://www.google.com
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 68
Connection: keep-alive
Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=65d10295bd238ea3c81f19b3e02d073c; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22d1e9deec40aa713329f98ddd11ce042f%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541242717%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D913ae0005d8fb07d1b6dc486b7fc9297; _gat=1
""")
print(yellow + header)
request = header
s.sendall(request.encode('utf-8'))
response = s.recv(4096)
time.sleep(2)
print('\n\n\n\n\n\n\n' + blue + str(response) + '\n')
s.close()
My question is, why on send i getting it response:
HTTP/1.1 400 Bad Request
Date: Sat, 03 Nov 2018 11:20:02 GMT
Server: Apache
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>
Anyone know, how to fix it? Header is good, i copy it from firefox network tools. I need to send POST request, not GET. Code written in python 3.7
On i send it post from firefox, i getting normal response:
status: "ok"
Now i change a header to:
header = ("""POST /json/daj_diax/ HTTP/1.1 Host: www.mclista.pl User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0 Accept: application/json, text/javascript, */*; q=0.01 Accept-Language: pl,en-US;q=0.7,en;q=0.3 Accept-Encoding: gzip, deflate Referer: http://www.mclista.pl/42204 Content-Type: application/x-www-form-urlencoded; charset=UTF-8 X-Requested-With: XMLHttpRequest Content-Length: 68 Connection: keep-alive Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1""")
and to:
header = ("""POST /json/daj_diax/ HTTP/1.1
Host: www.mclista.pl
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0
Accept: application/json, text/javascript, */*; q=0.01
Accept-Language: pl,en-US;q=0.7,en;q=0.3
Accept-Encoding: gzip, deflate
Referer: http://www.mclista.pl/42204
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
X-Requested-With: XMLHttpRequest
Content-Length: 68
Connection: keep-alive
Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1""")
doesn't working now too. On first header the same response, on secound header no response, recive sending a empty byte (b'')
You may need to use CRLF \r\n:
header = ("POST /json/daj_diax/ HTTP/1.1\r\n"
"Host: www.mclista.pl\r\n"
"User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:63.0) Gecko/20100101 Firefox/63.0\r\n"
"Accept: application/json, text/javascript, */*; q=0.01\r\n"
"Accept-Language: pl,en-US;q=0.7,en;q=0.3\r\n"
"Accept-Encoding: gzip, deflate\r\n"
"Referer: http://www.mclista.pl/42204\r\n"
"Content-Type: application/x-www-form-urlencoded; charset=UTF-8\r\n"
"X-Requested-With: XMLHttpRequest\r\n"
"Content-Length: 68\r\n"
"Connection: keep-alive\r\n"
"Cookie: _ga=GA1.2.942145996.1539109230; _gid=GA1.2.1111512675.1541185659; csrf_mclista=8cb69323a6f12ff338c06d4bea489c7f; ci_session=a%3A5%3A%7Bs%3A10%3A%22session_id%22%3Bs%3A32%3A%22b3df7ad8aef06a7428e497f90361eaef%22%3Bs%3A10%3A%22ip_address%22%3Bs%3A10%3A%225.184.63.6%22%3Bs%3A10%3A%22user_agent%22%3Bs%3A78%3A%22Mozilla%2F5.0+%28Windows+NT+10.0%3B+Win64%3B+x64%3B+rv%3A63.0%29+Gecko%2F20100101+Firefox%2F63.0%22%3Bs%3A13%3A%22last_activity%22%3Bi%3A1541265085%3Bs%3A9%3A%22user_data%22%3Bs%3A0%3A%22%22%3B%7D14f5f5334fa1876215c215f1f81889c6; _gat=1\r\n\r\n")
i need to create .ASPXAUTH cookie on python. i programing to desktop client. and first request not need .ASPXAUTH cookie but second request is need.
My First Request Headers:
User-Agent: WebPolicy
Host: xxx.host
Cache-Control: no-cache
My First Response Headers:
reply: 'HTTP/1.1 200 OK\r\n'
header: Cache-Control: private
header: Transfer-Encoding: chunked
header: Content-Type: text/xml; charset=utf-8
header: Server: Microsoft-IIS/7.5
header: Set-Cookie: tivi_=3tnihi55ezuk50zyrrpuwv45; path=/; HttpOnly
header: X-AspNet-Version: 2.0.50727
header: X-Powered-By: ASP.NET
header: Date: Thu, 14 Oct 2010 13:05:50 GMT
And i need second send headers :
Accept: */*
Content-Length: 259
Content-Type: text/xml; charset=utf-8
SOAPAction: "http://tempuri.org/IMiddlewareServices/Login"
UA-CPU: x86
Accept-Encoding: gzip, deflate
User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648; .NET CLR 3.5.21022; .NET CLR 1.1.4322)
Host: mw.webtv.ttnet.com.tr
Connection: Keep-Alive
Cache-Control: no-cache
Cookie: .ASPXAUTH=090F2718E32AF3F9B1C9E5A15BA54CFD8D4430C44A91029719953D4A6D38DD1D9164D86D772E2645C0C0545A71C12EA80AE5A8F725FD6037BD00DB291A863DD577735E16D8745E2833979F337935F29A37C509FB0350F1180DA0D2C1C44F97D0F081B13D33984C198ECD695C34B2E79A3E7CFBDD2D67D630C019714C3A70280E; tivi_=nqkngs45drnsh4z4y4b30g55
please help me! how to create ".ASPXAUTH" cookie ?
Python requests module does the job for me with .ASPXAUTH cookie-based authentication. In a simple example below .ASPXAUTH cookie is received after the first session.get call and used for the second call:
import requests
session = requests.Session()
session.get('https://host/user/login?username=###&password=###')
print session.cookies
session.get('https://host/apicall')
print session.cookies