I am trying to upload a file to Amazon S3 with Python Requests (Python is v2.7.9 and requests is v2.7). Following the curl command which works perfectly:
curl --request PUT --upload-file img.png https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ
But when I do same with requests, it fails. Here's what I have tried:
url = https://mybucket-dev.s3.amazonaws.com/6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ
requests.put(url, files={'file': base64_encoded_image})
requests.put(url, files={'upload_file': base64_encoded_image})
It fails with 403 and response I am getting is:
<Error><Code>SignatureDoesNotMatch</Code><Message>The request signature we calculated does not match the signature you provided. Check your key and signing method.</Message>
Then I ran curl in verbose mode:
* Hostname was NOT found in DNS cache
* Trying 54.231.168.134...
* Connected to mybucket-dev.s3.amazonaws.com (54.231.168.134) port 443 (#0)
* TLS 1.2 connection using TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
* Server certificate: *.s3.amazonaws.com
* Server certificate: VeriSign Class 3 Secure Server CA - G3
* Server certificate: VeriSign Class 3 Public Primary Certification Authority - G5
> PUT /6b89e187-26fa-11e5-a04f-a45e60d45b53?Signature=Ow%3D&Expires=1436595966&AWSAccessKeyId=AQ HTTP/1.1
> User-Agent: curl/7.37.1
> Host: mybucket-dev.s3.amazonaws.com
> Accept: */*
> Content-Length: 52369
> Expect: 100-continue
>
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< x-amz-id-2: 5lLCQ3FVrTBg2vkyk44E+MecQJb2OGiloO0+2pKePtxPgZptKECNlUyYN43sl4LBNe9f8idh/cc=
< x-amz-request-id: 636A24D53DEB5215
< Date: Fri, 10 Jul 2015 12:04:44 GMT
< ETag: "5802130d4320b56a72afe720e2c323a7"
< Content-Length: 0
* Server AmazonS3 is not blacklisted
< Server: AmazonS3
<
* Connection #0 to host mybucket-dev.s3.amazonaws.com left intact
So then I added headers
headers = {'Content-Length': '52369', 'Host': 'mybucket-dev.s3.amazonaws.com', 'Expect': '100-continue', 'Accept': '*/*', 'User-Agent': 'curl/7.37.1'}
requests.put(url, files={'file': base64_encoded_image}, headers=headers)
I tried with different combinations of header, still it throws same error. Then I tried to send query parameters too:
payload={'Expires': '1436595966', 'AWSAccessKeyId': 'AQ', 'Signature': 'Ow%3D'}
requests.put(url, files={'file': base64_encoded_image}, headers=headers, data=payload)
It still fails and same error. I tried URL without query parameters and sending them as data=payload to requests, it fails with same error.
Engineers at requests helped me:
with open('img.png', 'rb') as data:
requests.put(url, data=data)
Acording to this documentation you have pass the files argument to post method also need send the key name for S3
import requests
url = 'https://s3.amazonaws.com/<some-bucket-name>'
data = { 'key': 'test/test.jpeg' }
files = { 'file': open('test.jpeg', 'rb') }
r = requests.post(url, data=data, files=files)
print "status %s" % r.status_code
Related
I am trying to send through an image using the following code: This is just a part of my code, I didn't include the headers here but they are set up correctly, with content-type as content-type: multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe
img = "piano.jpg"
f = open(img,'rb')
out = f.read()
files = {'file':out}
p = requests.post("https://ecg-api.gumtree.com.au/api/pictures",headers=headers, data=files)
f.close()
I get a 400 error incorrect multipart/form-data format
How do I send the image properly?
Extra Details:
Network analysis shows the following request been sent:
POST https://ecg-api.gumtree.com.au/api/pictures HTTP/1.1
host: ecg-api.gumtree.com.au
content-type: multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe
authorization: Basic YXV5grehg534
accept: */*
x-ecg-ver: 1.49
x-ecg-ab-test-group: gblios_9069_b;gblios-8982-b
accept-encoding: gzip
x-ecg-udid: 73453-7578p-8657
x-ecg-authorization-user: id="1635662", token="ee56hgjfjdghgjhfj"
accept-language: en-AU
content-length: 219517
user-agent: Gumtree 12.6.0 (iPhone; iOS 13.3; en_AU)
x-ecg-original-machineid: Gk435454-hhttehr
Form data:
file: ����..JFIF.....H.H..��.LExif..MM.*..................�i.........&......�..
I cut off the the formdata part for file as its too long. My headers are written as follows (I have made up the actual auth values here):
idd = "1635662"
token = "ee56hgjfjdghgjhfj"
headers = {
"authority":"ecg-api.gumtree.com.au",
"content-type":"multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe",
"authorization":"Basic YXV5grehg534",
"accept":"*/*",
"x-ecg-ver":"1.49",
"x-ecg-ab-test-group":"gblios_9069_b;gblios-8982-b",
"accept-encoding":"gzip",
"x-ecg-udid":"73453-7578p-8657",
"x-ecg-authorization-user":f"id={idd}, token={token}",
"accept-language":"en-AU",
"content-length":"219517",
"user-agent":"Gumtree 12.6.0 (iPhone; iOS 13.3; en_AU)",
"x-ecg-original-machineid":"Gk435454-hhttehr"
}
Maybe its the way I have written the headers? I suspect its the way I have written the x-ecg-authorization-user part in headers? Because I realise even putting random values for the token or id gives me the same 400 error incorrect multipart/form-data format
You can try the following code. Don't set content-type in the headers.Let Pyrequests do that for you
files = {'file': (os.path.basename(filename), open(filename, 'rb'), 'application/octet-stream')}
upload_files_url = "url"
headers = {'Authorization': access_token, 'JWTAUTH': jwt}
r2 = requests.post(parcels_files_url, files=files, headers=headers)
I am trying to send a raw POST request to a chromedriver server.
Here is what I try to initiate a new session:
import socket
s = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 9515))
s.send(b'POST /session HTTP/1.1\r\nContent-Type:application/json\r\n{"capabilities": {}, "desiredCapabilities": {}}\r\n\r\n')
response = s.recv(4096)
print(response)
Output:
b'HTTP/1.1 200 OK\r\nContent-Length:270\r\nContent-Type:application/json; charset=utf-8\r\nConnection:close\r\n\r\n{"sessionId":"b26166c2aac022566917db20260500bb","status":33,"value":{"message":"session not created exception: Missing or invalid capabilities\\n (Driver info: chromedriver=2.31.488763 (092de99f48a300323ecf8c2a4e2e7cab51de5ba8),platform=Linux 4.4.0-91-generic x86_64)"}}'
Summary of error: json object I am sending is not getting parsed correctly
When I use the same json object but send it by requests library, everything is OK:
import requests
params = {
'capabilities': {},
'desiredCapabilities': {}
}
headers = {'Content-type': 'application/json'}
URL = "http://127.0.0.1:9515"
r = requests.post(URL + "/session", json=params)
print("Status: " + str(r.status_code))
print("Body: " + str(r.content))
Output:
Status: 200
Body: b'{"sessionId":"e03189a25d099125a541f3044cb0ee42","status":0,"value":{"acceptSslCerts":true,"applicationCacheEnabled":false,"browserConnectionEnabled":false,"browserName":"chrome","chrome":{"chromedriverVersion":"2.31.488763 (092de99f48a300323ecf8c2a4e2e7cab51de5ba8)","userDataDir":"/tmp/.org.chromium.Chromium.LBeQkw"},"cssSelectorsEnabled":true,"databaseEnabled":false,"handlesAlerts":true,"hasTouchScreen":false,"javascriptEnabled":true,"locationContextEnabled":true,"mobileEmulationEnabled":false,"nativeEvents":true,"networkConnectionEnabled":false,"pageLoadStrategy":"normal","platform":"Linux","rotatable":false,"setWindowRect":true,"takesHeapSnapshot":true,"takesScreenshot":true,"unexpectedAlertBehaviour":"","version":"60.0.3112.90","webStorageEnabled":true}}'
Summary of output: json object is parsed successfully by the chromedriver and new session is created
Do you, guys, have an idea why sending the raw POST request using socket is not working as expected?
There are several issues about your HTTP request:
The body of the HTTP request should be separated from the head by \r\n\r\n.
You need the Content-Length field, otherwise the remote host does not know when your body is complete.
The Host field is mandatory in HTTP 1.1. (Since you receive 200 with your first request, your server probably doesn't insist.)
I've got your example to work (with an apache webserver) by using:
s.send(b'POST /session HTTP/1.1\r\nHost: 127.0.0.1:9515\r\nContent-Type: application/json\r\nContent-Length: 47\r\n\r\n{"capabilities": {}, "desiredCapabilities": {}}')
To be visually more clear, the valid HTTP request looks like
POST /session HTTP/1.1
Host: 127.0.0.1:9515
Content-Type: application/json
Content-Length: 47
{"capabilities": {}, "desiredCapabilities": {}}
I am trying to post data to my server from my microcontroller. I need to send raw http data from my controller and this is what I am sending below:
POST /postpage HTTP/1.1
Host: https://example.com
Accept: */*
Content-Length: 18
Content-Type: application/json
{"cage":"abcdefg"}
My server requires JSON encoding and not form encoded request.
For the above request sent, I get an 400 error from the server, HTTP/1.1 400 Bad Request
However, when I try to reach the post to my server via a python script via my laptop, I am able to get a proper response.
import requests
url='https://example.com'
mycode = 'abcdefg'
def enter():
value = requests.post('url/postpage',
params={'cage': mycode})
print vars(value)
enter()
Can anyone please let me know where I could be going wrong in the raw http data I'm sending above ?
HTTP specifies the separator between headers as a single newline, and requires a double newline before the content:
POST /postpage HTTP/1.1
Host: https://example.com
Accept: */*
Content-Length: 18
Content-Type: application/json
{"cage":"abcdefg"}
If you don’t think you’ve got all of the request right, try seeing what was sent by Python:
response = ...
request = response.request # request is a PreparedRequest.
headers = request.headers
url = request.url
Read the docs for PreparedRequest for more information.
To pass a parameter, use this Python:
REQUEST = 'POST /postpage%s HTTP/1.1\r\nHost: example.com\r\nContent-Length: 0\r\nConnection: keep-alive\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nUser-Agent: python-requests/2.4.3 CPython/2.7.9 Linux/4.4.11-v7+\r\n\r\n';
query = ''
for k, v in params.items():
query += '&' + k + '=' + v # URL-encode here if you want.
if len(query): query = '?' + query[1:]
return REQUEST % query
I'm using Observium to pull Nginx stats on localhost however it returns '405 Not Allowed':
# curl -I localhost/nginx_status
HTTP/1.1 405 Not Allowed
Server: nginx
Date: Wed, 19 Jun 2013 22:12:37 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 166
Connection: keep-alive
Keep-Alive: timeout=5
# curl -I -H "Host: example.com" localhost/nginx_status
HTTP/1.1 200 OK
Server: nginx
Date: Wed, 19 Jun 2013 22:12:43 GMT
Content-Type: text/plain
Connection: keep-alive
Keep-Alive: timeout=5
Could you please advise how to add Host header with 'urllib2.urlopen' in Python (Python 2.6.6
):
Current script:
#!/usr/bin/env python
import urllib2
import re
data = urllib2.urlopen('http://localhost/nginx_status').read()
params = {}
for line in data.split("\n"):
smallstat = re.match(r"\s?Reading:\s(.*)\sWriting:\s(.*)\sWaiting:\s(.*)$", line)
req = re.match(r"\s+(\d+)\s+(\d+)\s+(\d+)", line)
if smallstat:
params["Reading"] = smallstat.group(1)
params["Writing"] = smallstat.group(2)
params["Waiting"] = smallstat.group(3)
elif req:
params["Requests"] = req.group(3)
else:
pass
dataorder = [
"Active",
"Reading",
"Writing",
"Waiting",
"Requests"
]
print "<<<nginx>>>\n";
for param in dataorder:
if param == "Active":
Active = int(params["Reading"]) + int(params["Writing"]) + int(params["Waiting"])
print Active
else:
print params[param]
You might want to check out the urllib2 missing manual for more information, but basically you create a dictionary of your header labels and values and pass it to the urllib2.Request method. A (slightly) modified version of the code from the linked manual:
from urllib import urlencode
from urllib2 import Request urlopen
# Define values that we'll pass to our urllib and urllib2 methods
url = 'http://www.something.com/blah'
user_host = 'example.com'
values = {'name' : 'Engineero', # dict of keys and values for our POST data
'location' : 'Interwebs',
'language' : 'Python' }
headers = { 'Host' : user_host } # dict of keys and values for our header
# Set up our request, execute, and read
data = urlencode(values) # encode for sending URL request
req = Request(url, data, headers) # make POST request to url with data and headers
response = urlopen(req) # get the response from the server
the_page = response.read() # read the response from the server
# Do other stuff with the response
I'm having problems with a Flask view that should return a response with content-type "application/json" in response to a POST request.
Specifically, if I do:
curl -v -d 'foo=bar' http://example.org/jsonpost
to this view:
#app.route('/jsonpost', methods=['GET', 'POST'])
def json_post():
resp = make_response('{"test": "ok"}')
resp.headers['Content-Type'] = "application/json"
return resp
I get some sort of connection reset:
* About to connect() to example.org port 80 (#0)
* Trying xxx.xxx.xxx.xxx... connected
* Connected to example.org (xxx.xxx.xxx.xxx) port 80 (#0)
> POST /routing/jsonpost HTTP/1.1
> User-Agent: curl/7.19.7 (i486-pc-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8k zlib/1.2.3.3 libidn/1.15
> Host: example.org
> Accept: */*
> Content-Length: 7
> Content-Type: application/x-www-form-urlencoded
>
< HTTP/1.1 200 OK
< Server: nginx/1.2.4
< Date: Thu, 27 Dec 2012 14:07:59 GMT
< Content-Type: application/json
< Content-Length: 14
< Connection: keep-alive
< Set-Cookie: session="..."; Path=/; HttpOnly
< Cache-Control: public
<
* transfer closed with 14 bytes remaining to read
* Closing connection #0
curl: (18) transfer closed with 14 bytes remaining to read
If instead I do:
curl -d 'foo=bar' http://example.org/htmlpost
to:
#app.route('/htmlpost', methods=['GET', 'POST'])
def html_post():
resp = make_response('{"test": "ok"}')
resp.headers['Content-Type'] = "text/html"
return resp
I get the expected the full response (200-ok)
{"test": "ok"}
By the way, if I send a GET request to the same JSON route:
curl http://example.org/jsonpost
I also get the expected response..
Any ideas?
Thanks to Audrius's comments I tracked a possible source of the problem to the interaction between uWSGI and nginx: apparently, if you receive POST data in a request you must read it before returning a response.
This, for example, fixes my issue.
#app.route('/jsonpost', methods=['GET', 'POST'])
def json_post():
if request.method == 'POST':
dummy = request.form
resp = make_response('{"test": "ok"}')
resp.headers['Content-Type'] = "application/json"
return resp
A different solution involves passing --post-buffering 1 to uWSGI as described by uWSGI's author Roberto De Ioris.
I still don't understand why the problem does not present itself with Content-Type set to "text/html"