I'm using httplib.HTTPConnection to submit an HTTP POST request. I get a 200 response status but the response data looks obfuscated or something.
When I submit the request in Firefox, the response is displayed fine.
conn = httplib.HTTPConnection("www.foo.com")
conn.request('POST', '/foo', postdata, headers)
resp = conn.getresponse()
conn.close()
print resp.read()
If the response is unexpectedly 'binary', look at the Content-Encoding header. Most likely you are being served a compressed response; it can be either gzip or deflate.
If you the encoding is gzip, decode it with:
import zlib
decompressor = zlib.decompressobj(16 + zlib.MAX_WBITS)
data = decompressor.decompress(response_body)
For deflate, you'd have to try both a default compressor and one with -zlib.MAX_WBITS:
try:
decompressor = zlib.decompressobj()
data = decompressor.decompress(response_body)
except zlib.error:
decompressor = zlib.decompressobj(-zlib.MAX_WBITS)
data = decompressor.decompress(response_body)
In addition to the other answer, you could probably disable encoding by setting the Accept-Encoding header to identity.
headers = {
# ...
"Accept-Encoding": "identity",
# ...
}
Related
I uploaded a file/csv to my Pivotal Cloud Foundry (PCF) ECS S3 storage. Which was created successfully.
import requests
url = "https://host/bucket/test.csv"
payload = {}
files = [
('file-name', open('test.csv','rb'))
]
headers = {
'X-Amz-Date': '20200622T112852Z',
'Authorization': 'AWS4-HMAC-SHA256 Credential=cred_endpoint, SignedHeaders=host;x-amz-date;x-amz-target, Signature=sigV4'
}
response = requests.put(url, headers=headers, data=payload, files=files)
When I try to access it I get a response like this
GET https://host/bucket/test.csv Status Code - 200
----------------------------017095580637364775189995
Content-Disposition: form-data; name="file-name"; filename="test.csv"
Content-Type: text/csv
rank,name
1,Michael
2,Jim
3,Dwight
----------------------------017095580637364775189995--
Which is the correct data but it's in this weird format with boundaries in the body, what would be the most robust way to read this kind of response body and generate the exact same or similar file/csv as the one upload?
Also am I even doing this correctly? Because all the docs imply there should be a [181(whatever the file size) bytes of object data] in the body instead of the actual content of the file.
I am trying to send through an image using the following code: This is just a part of my code, I didn't include the headers here but they are set up correctly, with content-type as content-type: multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe
img = "piano.jpg"
f = open(img,'rb')
out = f.read()
files = {'file':out}
p = requests.post("https://ecg-api.gumtree.com.au/api/pictures",headers=headers, data=files)
f.close()
I get a 400 error incorrect multipart/form-data format
How do I send the image properly?
Extra Details:
Network analysis shows the following request been sent:
POST https://ecg-api.gumtree.com.au/api/pictures HTTP/1.1
host: ecg-api.gumtree.com.au
content-type: multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe
authorization: Basic YXV5grehg534
accept: */*
x-ecg-ver: 1.49
x-ecg-ab-test-group: gblios_9069_b;gblios-8982-b
accept-encoding: gzip
x-ecg-udid: 73453-7578p-8657
x-ecg-authorization-user: id="1635662", token="ee56hgjfjdghgjhfj"
accept-language: en-AU
content-length: 219517
user-agent: Gumtree 12.6.0 (iPhone; iOS 13.3; en_AU)
x-ecg-original-machineid: Gk435454-hhttehr
Form data:
file: ����..JFIF.....H.H..��.LExif..MM.*..................�i.........&......�..
I cut off the the formdata part for file as its too long. My headers are written as follows (I have made up the actual auth values here):
idd = "1635662"
token = "ee56hgjfjdghgjhfj"
headers = {
"authority":"ecg-api.gumtree.com.au",
"content-type":"multipart/form-data; boundary=eBayClAsSiFiEdSpOsTiMaGe",
"authorization":"Basic YXV5grehg534",
"accept":"*/*",
"x-ecg-ver":"1.49",
"x-ecg-ab-test-group":"gblios_9069_b;gblios-8982-b",
"accept-encoding":"gzip",
"x-ecg-udid":"73453-7578p-8657",
"x-ecg-authorization-user":f"id={idd}, token={token}",
"accept-language":"en-AU",
"content-length":"219517",
"user-agent":"Gumtree 12.6.0 (iPhone; iOS 13.3; en_AU)",
"x-ecg-original-machineid":"Gk435454-hhttehr"
}
Maybe its the way I have written the headers? I suspect its the way I have written the x-ecg-authorization-user part in headers? Because I realise even putting random values for the token or id gives me the same 400 error incorrect multipart/form-data format
You can try the following code. Don't set content-type in the headers.Let Pyrequests do that for you
files = {'file': (os.path.basename(filename), open(filename, 'rb'), 'application/octet-stream')}
upload_files_url = "url"
headers = {'Authorization': access_token, 'JWTAUTH': jwt}
r2 = requests.post(parcels_files_url, files=files, headers=headers)
I am currently able to send OpenCV image frames to my Flask Server using the following code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
headers = {"Content-type": "text/plain"}
try:
conn.request("POST", "/", imencoded.tostring(), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response
But I want to send a unique_id along with the frame I tried combining the frame and the id using JSON but getting following error TypeError: Object of type 'bytes' is not JSON serializable does anybody have any idea how I can send some additional data along with the frame to the server.
UPDATED:
json format code
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
data = {"uid" : "23", "frame" : imencoded.tostring()}
headers = {"Content-type": "application/json"}
try:
conn.request("POST", "/", json.dumps(data), headers)
response = conn.getresponse()
except conn.timeout as e:
print("timeout")
return response
I have actually solved the query by using the Python requests module instead of the http.client module and have done the following changes to my above code.
import requests
def sendtoserver(frame):
imencoded = cv2.imencode(".jpg", frame)[1]
file = {'file': ('image.jpg', imencoded.tostring(), 'image/jpeg', {'Expires': '0'})}
data = {"id" : "2345AB"}
response = requests.post("http://127.0.0.1/my-script/", files=file, data=data, timeout=5)
return response
As I was trying to send a multipart/form-data and requests module has the ability to send both files and data in a single request.
You can try encoding your image in base64 string
import base64
with open("image.jpg", "rb") as image_file:
encoded_string = base64.b64encode(image_file.read())
And send it as a normal string.
As others suggested base64 encoding might be a good solution, however if you can't or don't want to, you could add a custom header to the request, such as
headers = {"X-my-custom-header": "uniquevalue"}
Then on the flask side:
unique_value = request.headers.get('X-my-custom-header')
or
unique_value = request.headers['X-my-custom-header']
That way you avoid the overhead of processing your image data again (if that matters) and you can generate a unique id for each frame with something like the python uuid module.
Hope that helps
I have a simple Python script which should read a file from HTTP source and make a PUT request to another HTTP source.
block_size = 4096
file = urllib2.urlopen('http://path/to/someting.file').read(block_size)
headers = {'X-Auth-Token': token_id, 'content-type': 'application/octet-stream'}
response = requests.put(url='http://server/path', data=file, headers=headers)
How can I make synchronous reading and putting this file by block_size (chunk) while the block is not empty?
What you want to do is called "streaming uploads". Try the following.
Get the file as a stream:
resp = requests.get(url, stream = True)
And then post the file like object:
requests.post(url, data= resp.iter_content(chunk_size= 4096))
I have searched for days and trying to solve the problem by myself, unsuccessfully.
I found that is possible to attach files to QC Run (using Python or Ruby) with something like this (send it in a Rest Request):
Content example:
headers = {'accept': 'application/xml', 'Content-Type': 'multipart/form-data; boundary=exampleboundary'}
--exampleboundary
Content-Disposition: form-data; name="filename"
example.txt
--exampleboundary
Content-Disposition: form-data; name="description"
Here is the text that describes example.txt
--exampleboundary
Content-Disposition: form-data; name="file"; filename="example.txt"
Content-Type: text/plain
ContentOfFile
--exampleboundary--
This really works but (apparently) only for text files (.txt). I really need to upload some images like test evidences/screenshots.
How can I achieve that? Can anyone help me to solve this problem?
I am sending the request content like this:
import requests
#login
response = requests.get("http://"+server+"/qcbin/authentication-point/authenticate", auth=(user,pwd))
# Get ALM token in dict format
token = response.cookies.get_dict()
requests.post(url, content, cookies=token, headers=headers_image)
Thank you.
Referring to Barney's comment I leave here the answer that solved the problem.
def upload_result_file(self, run_id, report_file, token):
url = "http://%s/qcbin/rest/domains/%s/projects/%s/runs/%s/attachments" % (server, domain, project, run_id)
payload = open(report_file, 'rb')
headers_file = {}
headers_file['Content-Type'] = "application/octet-stream"
headers_file['slug'] = "test-results." + report_file[report_file.rfind(".")+1: ]
response = requests.post(url, headers=headers_file, data=payload, cookies=token)
if not (response.status_code == 200 or response.status_code == 201):
print "Attachment step failed!", response.text, response.url, response.status_code
return
From:
https://github.com/macroking/ALM-Integration/blob/master/ALM_Integration_Util.py
In API:
if not request.form:
abort(405)
request.form.get('file', "")
file = file.read().encode("base64")
In POST call:
-F 'file=#/var/path/to/my/file/test.png' http://xxx.xx.xx.xxx:8080/todo/api/v1.0/tasks
Thank koxta for sharing this solution.
With this solution, I can upload a RobotFramework's log file as an attachment of a test run successfully.
Share my code:
def upload_log(self, entity_type, entity_id, file_name):
qurl = '%s/qcbin/rest/domains/%s/projects/%s/%s/%s/attachments' %(self._url, self._domain, self._project, entity_type, entity_id)
headers = self._headers
headers['Content-Type'] = 'application/octet-stream'
headers['slug'] = 'log.' +file_name[file_name.rfind(".")+1: ]
print (headers)
if os.path.isfile(file_name):
with open(file_name, 'rb') as log_file:
binary_data = log_file.read()
print (binary_data)
response = self.session.post(qurl, data=binary_data, headers=headers)
print (response.text)
if response.status_code != 201:
raise Exception('Failed to upload %s - code=%s message=%s' %(file_name, response.status_code, response.text))