I am trying to fill a form like that and submit it automaticly. To do that, I sniffed the packets while logging in.
POST /?pg=ogrgiris HTTP/1.1
Host: xxx.xxx.com
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
Accept-Language: en-us
Accept-Encoding: gzip, deflate
Content-Type: application/x-www-form-urlencoded
Origin: http://xxx.xxx.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Safari/605.1.15
Referer: http://xxx.xxx.com/?pg=ogrgiris
Upgrade-Insecure-Requests: 1
DNT: 1
Content-Length: 60
Connection: close
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
I repeated that packet by burp suite and saw works porperly. the response was the html of the member page.
Now I tried to do that on python. The code is below:
import requests
url = 'http://xxx.xxx.com/?pg=ogrgiris'
headers = {'Host':'xxx.xxx.com',
'Accept':'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
'Accept-Encoding':'gzip, deflate',
'Content-Type':'application/x-www-form-urlencoded',
'Referer':'http://xxx.xxx.com/?pg=ogrgiris',
'Content-Lenght':'60','Connection':'close'}
credentials = {'seviye': '700','ilkodu': '34','kurumkodu': '317381','ogrecino': '40','isim': 'ahm'}
r = requests.post(url,headers=headers, data=credentials)
print(r.content)
the problem is, that code prints the html of the login page even I send all of the credentials enough to log in. How can I get the member page? thanks.
If the POST request displays a page with the content you want, then the problem is only that you are sending data as JSON, not in "form" data format (application/x-www-form-urlencoded).
If a session is created at the request base and you have to make another request for the requested data, then you have to deal with cookies.
Problem with data format:
r = requests.post(url, headers=headers, data=credentials)
Kwarg json = creates a request body as follows:
{"ogrecino": "40", "ilkodu": "34", "isim": "ahm", "kurumkodu": "317381", "seviye": "700"}
While data= creates a request body like this:
seviye=700&ilkodu=34&kurumkodu=317381&ogrencino=40&isim=ahm
You can try https://httpbin.org:
from requests import post
msg = {"a": 1, "b": True}
print(post("https://httpbin.org/post", data=msg).json()) # Data as Form data, look at key `form`, it's object in JSON because it's Form data format
print(post("https://httpbin.org/post", json=msg).json()) # Data as json, look at key `data`, it's string
If your goal is to replicate the sample request, you are missing a lot of the headers; this in particular is very important Content-Type: application/x-www-form-urlencoded because it will tell your HTTP client how to format/encode the payload.
Check the documentation for requests so see how these form posts can work.
Related
I am currently developing a program in Python 3 that shall send a chunked encoded message in a POST Request to an arbitrary URL. As described in the docs of the library Requests (see "chunked-encoded requests" under https://requests.readthedocs.io/en/master/user/advanced/), one can use a Generator to specify a chunked encoded message. At the moment, I can not verify whether my Generator gets the job done, because everytime i try to print out the message body of the request with
print(response.request.body)
i get the following:
<generator object generator at 0x03FEB530>
For example, I want to send the following message in a request. (with Content-Type: application/x-www-form-urlencoded)
x=FOO
In a chunked encoded message, the message body shall look like this:
5\r\n
x=FOO\r\n
0\r\n
\r\n
In a whole request, it should look like this:
POST / HTTP/1.1
Host: example.com
Transfer-Encoding: chunked
5
x=FOO
0
I wrote my own Generator and wanted to ask whether this is the correct way of doing it. You can see it in the code snippet below.
def generator():
var1 = "5\r\n"
var2= "x=FOO\r\n"
var3= "0\r\n\r\n"
x = var1.encode('utf8')
y = var2.encode('utf8')
z = var3.encode('utf8')
yield x
yield y
yield z
url = 'https://example.com'
headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:71.0) Gecko/20100101 Firefox/71.0',
'Connection': 'keep-alive', 'Content-Type': 'application/x-www-form-urlencoded',
'Transfer-Encoding': 'chunked'}
response = requests.post(url=url, data=generator(), headers=headers)
Disclaimer: I do not want my generator to automatically generate the chunked encoding of a message, I want to send a request with a specific chunked encoded message like shown above.
The url I'd like to send post request to is http://www.hkexnews.hk/sdw/search/searchsdw.aspx
The search I'd like to do (manually) is simply input "1" in "Stock Code" and click "Search"
I have tried many time with python and chrome extension "Postman" sending post request with the following header:
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8
Accept-Encoding: gzip, deflate
Accept-Language: zh-TW,zh;q=0.9,en-US;q=0.8,en;q=0.7
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 1844
Content-Type: application/x-www-form-urlencoded
Cookie: TS0161f2e5=017038eb490da17e158ec558c902f520903c36fad91e96a3b9ca79b098f2d191e3cac56652
Host: www.hkexnews.hk
Origin: http://www.hkexnews.hk
Referer: http://www.hkexnews.hk/sdw/search/searchsdw.aspx
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/67.0.3396.62 Safari/537.36
and the following as params:
today: 20180624
sortBy:
selPartID:
alertMsg:
ddlShareholdingDay: 23
ddlShareholdingMonth: 06
ddlShareholdingYear: 2018
txtStockCode: 00001
txtStockName:
txtParticipantID:
txtParticipantName:
btnSearch.x: 35
btnSearch.y: 8
but it doesn't work.
Try the below way. It should fetch you the required response along with the tabular data available in that site generated according to the search criteria.
import requests
from bs4 import BeautifulSoup
URL = "http://www.hkexnews.hk/sdw/search/searchsdw.aspx"
with requests.Session() as s:
s.headers={"User-Agent":"Mozilla/5.0"}
res = s.get(URL)
soup = BeautifulSoup(res.text,"lxml")
payload = {item['name']:item.get('value','') for item in soup.select("input[name]")}
payload['__EVENTTARGET'] = 'btnSearch'
payload['txtStockCode'] = '00001'
payload['txtParticipantID'] = 'A00001'
req = s.post(URL,data=payload,headers={"User-Agent":"Mozilla/5.0"})
soup_obj = BeautifulSoup(req.text,"lxml")
for items in soup_obj.select("#pnlResultSummary .ccass-search-datarow"):
data = [item.get_text(strip=True) for item in items.select("div")]
print(data)
If the news site provides a search API and you have the access then you can use something like Postman to get the search results. Otherwise, you will have scrape the results.
The use case you mentioned is typical of scraping. See if there is a search API, if not use something like selenium to scrape the results.
I'm trying to send a post request via python, but it goes through badly.
I want my code to approve my selected seats and continue to payment.
I took this url,data and token from the post request after putting the selected cinema place time and seats.
import urllib.parse, urllib.request
url = "https://tickets.yesplanet.co.il/YPR/SelectSeatPageRes.aspx/SetSelectedSeats?ec=10725013018-246564"
data = urllib.parse.urlencode(dict(
seats = "11,19#11,20#11,21#11,22",
token ="246564#5#1"
))
res = urllib.request.urlopen(url, data.encode("utf8"))
print (res.read())
the link has an expiration but this is the result:
Session Ended It appears that the session has ended before you were able to complete your purchase.
a link to the main site : https://www.yesplanet.co.il
how do i know if my request is complete?
for your convince info from the headers and response tabs from the development tool:
response headers:
Cache-Control:private, max-age=0
Content-Length:170
Content-Type:application/json; charset=utf-8
Date:Tue, 30 Jan 2018 01:27:26 GMT
P3P:CP="NOI ADM DEV COM NAV OUR STP"
Server:Microsoft-IIS/8.5
X-AspNet-Version:4.0.30319
X-Powered-By:ASP.NET
request headers:
**Accept:application/json, text/javascript, */*; q=0.01
Accept-Encoding:gzip, deflate, br
Accept-Language:he-IL,he;q=0.9,en-US;q=0.8,en;q=0.7
Connection:keep-alive
Content-Length:44
Content-Type:application/json; charset=UTF-8
Cookie:ASP.NET_SessionId=p4citijvw3vrqxuoekqnlrhw; _ga=GA1.3.525452416.1517275557; _gid=GA1.3.1168599094.1517275557; _gat_tealium_0=1; utag_main=v_id:016144aba503001d7d72fa299b0904072001c06a00868$_sn:1$_ss:0$_st:1517277365866$ses_id:1517275555076%3Bexp-session$_pn:2%3Bexp-session; hfOIKey=CXCFcTD1; SS#246564#5#1=; SS%23246564%235%231=17%2C12%2317%2C13; hfSKey=%7C%7C%7C%7C%7C%7C%7C%7C%7C1072_res%7C10725013018-246564%7C20
Host:tickets.yesplanet.co.il
Origin:https://tickets.yesplanet.co.il
Referer:https://tickets.yesplanet.co.il/YPR/SelectSeatPageRes.aspx?dtticks=636528796178961691&cf=1004&ec=10725013018-246564
User-Agent:Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.119 Safari/537.36
X-Requested-With:XMLHttpRequest**
request payload
{seats: "16,10#16,11", token: "246564#5#1"}
seats
:
"16,10#16,11"
token
:
"246564#5#1"
and the response tab:
{"d":"{\"ReturnCode\":0,\"Message\":null,\"Redirect\":\"/YPR/OrderFormPageRes.aspx?dtticks=636528796470870119\\u0026cf=1005\\u0026ec=10725013018-246564\",\"Data\":null}"}
The cookie header is the key. When you send a request from xhr (aka your browser), relevant cookies are automatically appended to your request.
These cookies are how sessions are usually managed, and the response message indicates that the server did not find a valid session cookie in your request.
You will need to "authorize", via logging in or otherwise beginning this session, and then insert that session cookie into your request before sending it.
After rereading, the token header is most likely not static either. My guess would be that this is engineered to prevent automated requests, and so may be difficult to circumvent.
Update in response to OP comment:
Use cookiejar or just read the urllib docs and figure out how to extract and then insert cookies.
how to send cookies inside post request
You will need to study the website’s behavior in your developer tools and see which request triggers a session cookie update, and then simulate that request before you simulate your post request.
You’ve been provided three answers. Mark the question as correct and post another, more specific if you still have trouble.
I'm trying to upload an image using requests on python.
This is what I send using browser
POST /upload-photo/{res1}/{res2}/{res3}/ HTTP/1.1
Host: tgt.tgdot.com
Connection: keep-alive
Content-Length: 280487
Authorization: Basic {value}=
Accept: */*
Origin: http://tgt.tgdot.com
X-Requested-With: XMLHttpRequest
User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.134 Safari/537.36
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA8sGeB48ZZCvG127
Referer: http://tgt.tgdot.com/{res1}/
Accept-Encoding: gzip, deflate
Accept-Language: en-US,en;q=0.8,es;q=0.6
Cookie: fttoken={cookie_value}
This is my code
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
encoded_image = base64.b64encode(image_file.read())
headers = {"Content-Type":"multipart/form-data", "Authorization":"Basic " + authvalue}
cookie = {cookiename: token.value}
r = requests.post(url, headers =headers, cookies = cookie, params=encoded_image)
print r.request.headers
print r.status_code
print r.text
I keep getting 414 Request-URI Too Large
I'm not sure what's missing here. I would really appreciate help
You are encoding the whole image into the request parameters, effectively extending the URL by the length of the image.
If you already encoded the image data, use the data parameter:
r = requests.post(url, headers=headers, cookies=cookie, data=encoded_image)
Note that requests can encode multipart/form-data POST bodies directly, there is no need for you to encode it yourself. Use the files parameter in that case, passing in a dictionary or sequence of tuples. See the POST Multiple Multipart-Encoded Files section of the documentation.
The library can also handle a username and password pair to handle the Authorization header; simply pass in a (username, password) tuple for the auth keyword argument.
Encoding an image to Base64 is not sufficient however. Your content-type header and your POST payload are not matching. You'd instead post the file with a field name:
with open(os.getcwd()+"/images/thee1.JPG", "rb") as image_file:
files = {'field_name': image_file}
cookie = {cookiename: token.value}
r = requests.post(url, cookies = cookie, files=files, auth=(username, password)
I'm trying to connect to the twiiter API through a POST request as the docs say but I always get a 403 forbidden error.
This is my code. I'm using urlib2 in python 2.7:
def auth_API():
url = 'https://api.twitter.com/oauth2/token'
header = {}
values = {}
header['User-Agent'] = 'Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1'
header['Authorization'] = 'Basic ' + B64BEARERTOKENCREDS
header['Content-Type'] = 'application/x-www-form-urlencoded;charset=UTF-8'
header['Accept-Encoding'] = 'gzip'
values['grant_type'] = 'client_credentials'
data = urllib.urlencode(values)
req = urllib2.Request(url, data, header)
try:
response = urllib2.urlopen(req)
response.read()
except urllib2.HTTPError as e:
print e
Checking the docs I found an example request wich is the same as mine:
Twitter example:
POST /oauth2/token HTTP/1.1
Host: api.twitter.com
User-Agent: My Twitter App v1.0.23
Authorization: Basic NnB1[...]9JM29jYTNFOA==
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Content-Length: 29
Accept-Encoding: gzip
grant_type=client_credentials
My request:
POST /oauth2/token HTTP/1.1
Content-Length: 29
Accept-Encoding: gzip
Connection: close
User-Agent: Mozilla/6.0 (Windows NT 6.2; WOW64; rv:16.0.1) Gecko/20121011 Firefox/16.0.1
Host: api.twitter.com
Content-Type: application/x-www-form-urlencoded;charset=UTF-8
Authorization: Basic NnB1[...]YTNFOA==
grant_type=client_credentials
Any idea with what could be wrong with this?
Regards.
PS: I know that there are some third party libs for this but I want to make it by myself.
I solved my problem, the error was with base64.encodestring() which adds an \n at the end of the string messing up the request.
Using base64.b64encode() instead worked fine.
Regards