Before downvoting/marking as duplicate, please note:
I have already tried out this, this, this, this,this, this - basically almost all the methods I could find pointed out by the Requests documentation but do not seem to find any solution.
Problem:
I want to make a POST request with a set of headers and form data.
There are no files to be uploaded. As per the request body in Postman, we set the parameters by selecting 'form-data' under the 'Body' section for the request.
Here is the code I have:
headers = {'authorization': token_string,
'content-type':'multipart/form-data; boundary=----WebKitFormBoundaryxxxxxXXXXX12345'} # I get 'unsupported application/x-www-form-url-encoded' error if I remove this line
body = {
'foo1':'bar1',
'foo2':'bar2',
#... and other form data, NO FILE UPLOADED
}
#I have also tried the below approach
payload = dict()
payload['foo1']='bar1'
payload['foo2']='bar2'
page = ''
page = requests.post(url, proxies=proxies, headers=headers,
json=body, files=json.dump(body)) # also tried data=body,data=payload,files={} when giving data values
Error
{"errorCode":404,"message":"Required String parameter 'foo1' is not
present"}
EDIT:
Adding a trace of the network console. I am defining it in the same way in the payload as mentioned on the request payload.
There isn't any gui at all? You could get the network data from chrome, although:
Try this:
headers = {'authorization': token_string}
Probably there is more authorization? Or smthng else?
You shouldn't add Content-Type as requests will handle it for you.
Important, you could see the content type as WebKitFormBoundary, so for the payload you must take, the data from the "name" variable.
Example:
(I know you won't upload any file, it just an example) -
So in this case, for my payload would look like this: payload = {'photo':'myphoto'} (yea there would be an open file etc etc, but I try to keep it simple)
So your payload would be this-> (So always use name from the WebKit)
payload = {'foo1':'foo1data',
'foo2':'foo2data'}
session.post(url,data = payload, proxies etc...)
Important! As I can see you use the method from requests library. Firstly you always should create a session like this
session = requests.session() -> it will handle cookies, headers, etc, and won't open a new session, or plain requests with every requests.get/post.
Related
I have to send a POST request to the /batch endpoint of : 'https://www.google-analytics.com'.
As mentioned in the Documentation I have to send the request to /batch endpoint and specify each payload on its own line.
I was able to achieve this using POSTMAN as follows:
My query is to make a POST request using Python's requests library
I tried something like this :
import requests
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
response = requests.post('https://www.google-analytics.com/batch', data=text)
but it doesn't works.
UPDATE
I Tried this and it works !
import http.client
conn = http.client.HTTPSConnection("www.google-analytics.com")
payload = "v=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=63\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=11\r\nv=1&cid=43223523&tid=UA-200248207-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=23"
headers = {
'Content-Type': 'text/plain'
}
conn.request("POST", "/batch", payload, headers)
res = conn.getresponse()
But the question remains open, what's the issue with requests here.
You don't need to double-escape the newline symbol.
Moreover, you don't need the newline symbol at all for the multi-line string.
And also the indentations you put in your multi-line string are counted:
test = '''abc
def
ghi'''
print(test)
Here's an SO answer that explains this with some additional ways to make long stings: https://stackoverflow.com/a/10660443/4570170
Now the request body.
The documentation says
payload_data – The BODY of the post request. The body must include exactly 1 URI encoded payload and must be no longer than 8192 bytes.
So try uri-encoding your payload:
text = '''v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=bookmarks&ev=13
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=upvotes&ev=65
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=questions&ev=15
v=1&cid=43223523&tid=UA-XXXXXX-1&t=event&ec=aggregated_stats&ea=daily_kpi&el=postviews&ev=95'''
text_final = requests.utils.quote(text)
response = requests.post('https://www.google-analytics.com/batch', data=text_final)
Finally , I figured out the solution myself.
Updating for others help.
The problem was I was working on AWS Cloud9 and as mentioned in the documentation
Some environments are not able to send hits to Google Analytics directly. Examples of this are older mobile phones that can't run JavaScript or corporate intranets behind a firewall.
So we just need to include the User Agent parameter
ua=Opera/9.80
in each of our payloads
It works !
I am using am attempting to do a bulk download of a series of PDFs from a site that requires login authentication. I am able to successfully log in, however, when I attempt a GET request for '/transcripts/transcript.pdf?user_id=3007' but, the request returns the content for '/transcripts/transcript.pdf'.
Does anyone have any idea why the URL param is not sending? Or why it would be rerouted?
I have tried passing the parameter 'user_id' as data, params, and hardcoded in the URL.
I have removed the actual domain from the strings below just for privacy
with requests.Session() as s:
login = s.get('<domain>/login/canvas')
# print the html returned or something more intelligent to see if it's a successful login page.
print(login.text)
login_html = lxml.html.fromstring(login.text)
hidden_inputs = login_html.xpath(r'//form//input[#type="hidden"]')
form = {x.attrib["name"]: x.attrib["value"] for x in hidden_inputs}
print("form: ",form)
form['pseudonym_session[unique_id]']= username
form['pseudonym_session[password]']= password
response = s.post('<domain>/login/canvas',data=form)
print(response.url, response.status_code) # gets <domain>?login_success=1 200
# An authorised request.
data = { 'user_id':'3007'}
r = s.get('<domain>/transcripts/transcript.pdf?user_id=3007', data=data)
print(r.url) # gets <domain>/transcripts/transcript.pdf
print(r.status_code) # gets 200
with open('test.pdf', 'wb') as f:
f.write(r.content)
GET response returns /transcripts/transcript.pdf and not /transcripts/transcript.pdf?user_id=3007
From the looks of it, you are trying to use canvas. I'm pretty sure in canvas, you can bulk download all test attachments.
If that's not the case, There are a few things to try:
after logging in, try typing the url with user_id into a browser. Does that take you directly to the PDF file or links to one?
if so, look at the url, it may simply not display the parameters; some websites do this, don't worry about it
If not, GET may not be enough; perhaps the site uses javascript, etc.
after looking through the '.history' of the request I found a series of 302 redirects.
The first was to '/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf'
In a desperate attempt, I tried: s.get('/login?force_login=0&target_uri=%2Ftranscripts%2Ftranscript.pdf%3Fuser_id%3D3007') and this still rerouted me a few times but ultimately got me the file I wanted!
If anyone has a more elegant solution to this or any resources that I can read I would greatly appreciate it!
I'm trying to create a super-simplistic Virtual In / Out Board using wx/Python. I've got the following code in place for one of my requests to the server where I'll be storing the data:
data = urllib.urlencode({'q': 'Status'})
u = urllib2.urlopen('http://myserver/inout-tracker', data)
for line in u.readlines():
print line
Nothing special going on there. The problem I'm having is that, based on how I read the docs, this should perform a Post Request because I've provided the data parameter and that's not happening. I have this code in the index for that url:
if (!isset($_POST['q'])) { die ('No action specified'); }
echo $_POST['q'];
And every time I run my Python App I get the 'No action specified' text printed to my console. I'm going to try to implement it using the Request Objects as I've seen a few demos that include those, but I'm wondering if anyone can help me explain why I don't get a Post Request with this code. Thanks!
-- EDITED --
This code does work and Posts to my web page properly:
data = urllib.urlencode({'q': 'Status'})
h = httplib.HTTPConnection('myserver:8080')
headers = {"Content-type": "application/x-www-form-urlencoded",
"Accept": "text/plain"}
h.request('POST', '/inout-tracker/index.php', data, headers)
r = h.getresponse()
print r.read()
I am still unsure why the urllib2 library doesn't Post when I provide the data parameter - to me the docs indicate that it should.
u = urllib2.urlopen('http://myserver/inout-tracker', data)
h.request('POST', '/inout-tracker/index.php', data, headers)
Using the path /inout-tracker without a trailing / doesn't fetch index.php. Instead the server will issue a 302 redirect to the version with the trailing /.
Doing a 302 will typically cause clients to convert a POST to a GET request.
I'm starting to learn how to use the python requests module. For practicing I tried to manage a challenge/response problem: I want to access the data on http://lema.rae.es/drae/srv/search?val=hacer
With the "Tamper Data" plugin for Firefox I inspected the necessary HTTP requests:
GET http://lema.rae.es/drae/srv/search?val=hacer
POST http://lema.rae.es/drae/srv/search?val=hacer
I copied the exact headers that are sent by Firefox in the two HTTP requests and implemented the JavaScript "challenge" function in Python. Then I'm doing the following:
url = "http://lema.rae.es/drae/srv/search?val=hacer"
headers = { ... }
r1 = requests.get(url=url, headers=headers)
html = r1.content.decode("utf-8")
formdata = challenge(html)
headers = { ... }
r2 = requests.post(url=url, data=formdata, headers=headers)
Unfortunately, the server will not answer in the expected way. I checked all the headers I'm sending via "r.request.headers" and they agree perfectly with the headers that firefox sends (according to Tamper Data)
What am I doing wrong?
You can inspect my full code here: http://pastebin.com/7JAZ9B4s
This is the response header I should be getting:
Date[Tue, 10 Feb 2015 17:13:53 GMT]
Vary[Accept-Encoding]
Content-Encoding[gzip]
Cache-Control[max-age=0, no-cache]
Keep-Alive[timeout=5, max=100]
Connection[Keep-Alive]
Content-Type[text/html; charset=UTF-8]
Set-Cookie[TS014dfc77=017ccc203c29467c4d9b347fb56ea0e89a7182e52b9d7b4a1174efbf134768569a005c7c85; Path=/]
Transfer-Encoding[chunked]
And this is the response header I really get:
Content-Length[5798]
Content-Type[text/html]
Pragma[no-cache]
Cache-Control[no-cache]
I found the reason why my code doesn't work:
The server expects the POSTDATA in exactly the same order in which the entries appear as input-elements of the form. In my code the values of the input-elements were stored in a python dict. But this data type does not preserve the order in which values have been declared!
The ruby script (referred to in the comments) however does work because the ruby dict data type seems to preserve the order of declaration!
Furthermore, reimplementing the javascript challenge() function in python was not necessary at all, because the server will be happy to accept any response string (that worked in the past) over and over again!
With the firefox plugin "HttpFox" i'm getting the POST request which looks like this:
{'json':'{"command":"SEARCH","data":{"someData":"someValue","otherData":"otherData"}}'}
Now i need to send a http request build with python to get the same data as i would get via browser. See code:
headers = {'Content-type': 'application/json; charset=utf-8'}
payload = ?
req = requests.post(url, data=json.dumps(payload), headers = headers)
My problem is:
I'm not sure how to build the payload. It should be a dictionary as well, but im confused because of the POST type which is delivered with HttpFox. There are two dictionaries inside the main dictionary.
How should i handle this ?
Appreciate any help.
Ok, i found the solution:
It was necessary to build a dict like this:
valueString = """{"command":"SEARCH","data":{"someData":"someValue","otherData":"otherData"}}"""
/// the """ ensures that the whole text between """ is handled as a string.
payload = {'json': valueString}
The key 'json' requieres a string. In this case the string looks like a dictionary.
That's it.