I am developing an API that allows outside clients to send a binary file which will be processed. my web.data() is a string and the function I am calling requires a binary. How do I get it into the correct format? Maybe I have the incorrect Headers? how do I extract the binary data. I am using web.py.
-----------------POST request----------------------------------------------------
import json
import requests
files = {'file':('000038fe4b46c210c37bdde767835007', open('000038fe4b46c210c37bdde767835007', 'rb'))}
headers = {'content-type' : 'application/octet-stream', 'X-Auth-Token':'xxxf'}
r = requests.post('http://XXX:8080/v1/binaries', files = files, headers = header
-----------------------API function------------------------------
def POST(self):
a = web.ctx.env.get("HTTP_X_AUTH_TOKEN", None)
creds = authenticator(a)
postdata = web.data().read()
analysis = atklite.FileAnalysis(data=postdata)
metadata = analysis.return_analysis()
------------------------Traceback--------------------------------
File "/usr/lib/pymodules/python2.7/web/application.py", line 242, in process
return self.handle()
File "/usr/lib/pymodules/python2.7/web/application.py", line 233, in handle
return self._delegate(fn, self.fvars, args)
File "/usr/lib/pymodules/python2.7/web/application.py", line 415, in _delegate
return handle_class(cls)
File "/usr/lib/pymodules/python2.7/web/application.py", line 390, in handle_class
return tocall(*args)
File "/home/XXXXXX/ProcessingCode/bfsapi.py", line 75, in POST
postdata = web.data().read()
AttributeError: 'str' object has no attribute 'read'
Thanks
Sorry if the formatting got all messed up in the Post.
Even if it is a binary file, reading raw post data would get you a encoded string. You would need to decode to convert to binary data. You can write to a file as follows:
written = open('binary.file', 'wb')
for chunk in rawdata.chunks():
written.write(chunk)
written.close()
Related
I'm trying to import the Json data generated by an Impinj R420 reader.
The code i use is:
# import urllib library
import urllib.request
from urllib.request import urlopen
# import json
import json
# store the URL in url as
# parameter for urlopen
url = "http://10.234.92.19:14150"
# store the response of URL
response = urllib.request.urlopen(url)
# storing the JSON response
# from url in data
data_json = json.loads(response())
# print the json response
print(data_json)
When i execute the programm it gives the following error:
Traceback (most recent call last):
File "C:\Users\V\PycharmProjects\Stapelcontrole\main.py", line 13, in <module>
response = urllib.request.urlopen(url)
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 216, in urlopen
return opener.open(url, data, timeout)
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 519, in open
response = self._open(req, data)
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 536, in _open
result = self._call_chain(self.handle_open, protocol, protocol +
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 496, in _call_chain
result = func(*args)
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 1377, in http_open
return self.do_open(http.client.HTTPConnection, req)
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\urllib\request.py", line 1352, in do_open
r = h.getresponse()
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 1374, in getresponse
response.begin()
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 318, in begin
version, status, reason = self._read_status()
File "C:\Users\V\AppData\Local\Programs\Python\Python310\lib\http\client.py", line 300, in _read_status
raise BadStatusLine(line)
http.client.BadStatusLine: {"epc":"3035307B2831B383E019E8EA","firstSeenTimestamp":"2022-04-11T11:24:23.592434Z","isHeartBeat":false}
Process finished with exit code 1
I know this is an error in the response where it gets a faulty HTTP status code.
Yet i don't know how to fix the error.
Could you advice me how to fix this?
The {"epc":"3035307B2831B383E019E8EA","firstSeenTimestamp":"2022-04-11T11:24:23.592434Z","isHeartBeat":false} is an answer i expect.
Thanks in advance
Edit:
Even with
with urllib.request.urlopen(url) as f:
data_json = json.load(f)`
i get the same BadStatusLine error.
I can't setup the reader any different, it can only sent a JSON response trough the IP-adress of the device. Is there a way to import the data without the HTTP Protocol?
# store the response of URL
response = urllib.request.urlopen(url)
# storing the JSON response
# from url in data
data_json = json.loads(response())
Here you are actually calling response, I do not know what you want to achieve by that, but examples in urllib.request docs suggest that urllib.request.urlopen should be treated akin to local file handle, thus please replace above using
with urllib.request.urlopen(url) as f:
data_json = json.load(f)
Observe that I used json.load not json.loads
EDIT: After Is there a way to import the data without the HTTP Protocol? I conclude more low-level solution is needed, hopefully socket will you allow to get what you want, using echo client example as starting point I prepared following code
import socket
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.connect(("10.234.92.19",14150))
s.sendall(b'GET / HTTP/1.1\r\n')
data = s.recv(1024)
print(data)
If everything will work as intended you should get printed 1024 first bytes of answer. If it so change 1024 to value which will be always bigger than number of bytes of response and use json.dumps(data) to get parsed response
I need to load a file using a soap endpoint url...When I use the below code to load it the files are getting loaded but they are not in a readable format...When I load using SOAPUI tool it loads properly...
import requests
xml = '''<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/"
xmlns:v1="http://s.sa.com/services/Attachment/v1.0">
<soapenv:Header/>
<soapenv:Body>
<v1:attachment>
<filename>FUZZY.csv</filename>
<data>cid:138641430598</data>
</v1:attachment>
</soapenv:Body>
</soapenv:Envelope>'''
target_url = 'https://s.sa.com:443/soatest/FileAttachmentService'
headers = {'Content-Type': 'text/xml','charset':'utf-8'}
r = requests.post(target_url,data=xml,headers=headers,auth=('3user1',''))
print 'r.text = ', r.text
print 'r.content = ', r.content
print 'r.status_code = ', r.status_code
New changes:-
files = {'file':open('./FUZZY.csv','rb')}
print files
r = requests.post(target_url,files=files,data=xml,headers=headers,auth=('p3user1',''))
Error:
Traceback (most recent call last):
File "soapcall_python.py", line 18, in <module>
r = requests.post(target_url,files=files,data=xml,headers=headers,auth=('p3user1',''))
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 418, in request
prep = self.prepare_request(req)
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/sessions.py", line 356, in prepare_request
hooks=merge_hooks(request.hooks, self.hooks),
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 297, in prepare
self.prepare_body(data, files)
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 432, in prepare_body
(body, content_type) = self._encode_files(files, data)
File "/opt/python2.7/lib/python2.7/site-packages/requests-2.3.0-py2.7.egg/requests/models.py", line 109, in _encode_files
raise ValueError("Data must not be a string.")
ValueError: Data must not be a string.
You aren't sending the contents of the file anywhere. You're just sending a reference to a file that doesn't exist anywhere that the server can see.
As the docs for SOAP references to attachments explains, the way you do this is to send a MIME-multipart message. If you're using the CID reference mechanism, that cid isn't some arbitrary string, it has to match the Content-ID header of a message in the MIME envelope.
The requests docs for POST a Multipart-Encoded File explain how to send the contents of a file as a message within a MIME request; briefly:
with open('FUZZY.csv', 'rb') as f:
files = {'file': f}
r = requests.post(target_url,
data=xml, headers=headers, auth=('3user1',''),
files=files)
However, this simple method doesn't give you access to the Content-ID that will be generated under the covers for your message. So, if you want to use the CID reference mechanism, you will need to generate the MIME envelope manually (e.g., by using email.mail.MIMEMultipart) and sending the entire thing as a data string.
I'm trying to implement a simple Django view that accept a file attachment and some other parameters and proxy the request on a remote API call.
Please note: the proxy is not the point of my question :)
This is how I implemented the view:
def image_upload(request):
token = request.POST['token']
image_file = request.FILES.values()[0]
files = {'file': ('myupload.txt', image_file.read())}
client_id = request.POST['client_id']
folder_id = request.POST['folder_id']
advert_id = request.POST['advert_id']
image_type = request.POST['image_type']
crop_image = request.POST['crop_image']
api_base_url = settings.API_BASE_URL
file_post_data = {'client_id': client_id, 'folder_id': folder_id, 'advert_id': advert_id,
'image_type': image_type, 'crop_image': crop_image}
auth_header = {'Authorization': 'Token ' + token}
r = requests.post(api_base_url + 'assets/image/upload/',
data = json.dumps(file_post_data),
headers = auth_header,
files = files)
return r.json()
The problem is that when I test this view (I use Django Test Client to do it) I get an error on the line "files = files)" that says "ValueError: cannot encode objects that are not 2-tuples".
The complete trace log is this one:
======================================================================
ERROR: test_image_upload (fbx.tests.FbxTest)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/andrea/Documents/src/fbxapp/onboard/fbx/tests.py", line 18, in test_image_upload
'image_type': 'A', 'crop_image': False, 'attachment': fp})
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 449, in post
response = super(Client, self).post(path, data=data, content_type=content_type, **extra)
File "/usr/local/lib/python2.7/dist-packages/django/test/client.py", line 262, in post
return self.request(**r)
File "/usr/local/lib/python2.7/dist-packages/django/core/handlers/base.py", line 111, in get_response
response = callback(request, *callback_args, **callback_kwargs)
File "/home/andrea/Documents/src/fbxapp/onboard/fbx/views.py", line 42, in image_upload
files = files)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 88, in post
return request('post', url, data=data, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/api.py", line 44, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python2.7/dist-packages/requests/sessions.py", line 324, in request
prep = req.prepare()
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 225, in prepare
p.prepare_body(self.data, self.files)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 385, in prepare_body
(body, content_type) = self._encode_files(files, data)
File "/usr/local/lib/python2.7/dist-packages/requests/models.py", line 99, in _encode_files
fields = to_key_val_list(data or {})
File "/usr/local/lib/python2.7/dist-packages/requests/utils.py", line 136, in to_key_val_list
raise ValueError('cannot encode objects that are not 2-tuples')
ValueError: cannot encode objects that are not 2-tuples
I've also tried a quick test using this to read a file: files = {'file': ('myupload.txt', open('/tmp/mytmp.txt', 'rb'))}
but it doesn't work. Do tou have any idea about how to fix this?
Thanks!
You cannot simultaneously post JSON data and multipart/form-data which is in essence what you're trying to do here. json.dumps returns a string so you're sending a string that looks like
'{"client_id": 1, "folder_id": 2, "advert_id": 3, "image_type": "jpeg", "crop_image": true}'
And then telling requests you want to use that in combination with a multipart/form-data request by sending something in via the files parameter. That is impossible and could possibly raise a better exception.
Either add the file to the JSON data after reading it into memory and send that with the appropriate Content-Type header or send the entire thing as a multipart/form-data request without using json.dumps and simply passing the dictionary you're creating to data. Use one or the other but not both.
That aside, you say that your exception comes from the line that only has files=files) on it, but it is not that line alone that causes the exception. The exception is raised by the function which happens to end on that line. The fact that your exception arises from that too is mere coincidence. This is an wart in Python that may possibly be fixed in Python 3.4. You should upgrade, because 3.4 will be awesome and newer versions of Django support Python 3.x.
Let me start off by saying, I'm not new to programming but am very new to python.
I've written a program using urllib2 that requests a web page that I would then like to save to a file. The web page is about 300KB, which doesn't strike me as particularly large but seems to be enough to give me trouble, so I'm calling it 'large'.
I'm using a simple call to copy directly from the object returned from urlopen into the file:
file.write(webpage.read())
but it will just sit for minutes, trying to write into the file and I eventually receive the following:
Traceback (most recent call last):
File "program.py", line 51, in <module>
main()
File "program.py", line 43, in main
f.write(webpage.read())
File "/usr/lib/python2.7/socket.py", line 351, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/httplib.py", line 541, in read
return self._read_chunked(amt)
File "/usr/lib/python2.7/httplib.py", line 592, in _read_chunked
value.append(self._safe_read(amt))
File "/usr/lib/python2.7/httplib.py", line 649, in _safe_read
raise IncompleteRead(''.join(s), amt)
httplib.IncompleteRead: IncompleteRead(6384 bytes read, 1808 more expected)
I don't know why this should give the program so much grief?
EDIT |
here is how I'm retrieving the page
jar = cookielib.CookieJar()
cookie_processor = urllib2.HTTPCookieProcessor(jar);
opener = urllib2.build_opener(cookie_processor)
urllib2.install_opener(opener)
requ_login = urllib2.Request(LOGIN_PAGE,
data = urllib.urlencode( { 'destination' : "", 'username' : USERNAME, 'password' : PASSWORD } ))
requ_page = urllib2.Request(WEBPAGE)
try:
#login
urllib2.urlopen(requ_login)
#get desired page
portfolio = urllib2.urlopen(requ_page)
except urllib2.URLError as e:
print e.code, ": ", e.reason
I'd use a handy fileobject copier function provided by shutil module. It worked on my machine :)
>>> import urllib2
>>> import shutil
>>> remote_fo = urllib2.urlopen('http://docs.python.org/library/shutil.html')
>>> with open('bigfile', 'wb') as local_fo:
... shutil.copyfileobj(remote_fo, local_fo)
...
>>>
UPDATE: You may want to pass the 3rd argument to copyfileobj that controls the size of internal buffer used to transfer bytes.
UPDATE2: There's nothing fancy about shutil.copyfileobj. It simply reads a chunk of bytes from source file object and writes it the destination file object repeatedly until there's nothing more to read. Here's the actual source code of it that I grabbed from inside Python standard library:
def copyfileobj(fsrc, fdst, length=16*1024):
"""copy data from file-like object fsrc to file-like object fdst"""
while 1:
buf = fsrc.read(length)
if not buf:
break
fdst.write(buf)
I want to get elevation data from Google Earth according to latitude and longitude, but I am not able to do this. I'm not sure what I'm doing wrong but my code is shown below.
def getElevation(locations,sensor="true", **elvtn_args):
elvtn_args.update({
'locations': locations,
'sensor': sensor
})
url = ELEVATION_BASE_URL
params = urllib.parse.urlencode(elvtn_args)
baseurl = url +"?"+ params;
req = urllib.request.urlopen(str(baseurl));
response = simplejson.load(req);
And the error I get is :
Traceback (most recent call last):
File "D:\GIS\Arctools\ElevationChart - Copy.py", line 85, in <module>
getElevation(pathStr)
File "D:\GIS\Arctools\ElevationChart - Copy.py", line 45, in getElevation
response = simplejson.load(req);
File "C:\Python32\lib\json\__init__.py", line 262, in load
parse_constant=parse_constant, object_pairs_hook=object_pairs_hook, **kw)
File "C:\Python32\lib\json\__init__.py", line 307, in loads
return _default_decoder.decode(s)
File "C:\Python32\lib\json\decoder.py", line 351, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
TypeError: can't use a string pattern on a bytes-like object
Any help appreciated.
Post is a little late but recently ran into the same problem. The solution below worked for me. Basically what Lennart said.
from urllib import request
import json
req = request.urlopen('https://someurl.net/api')
encoding = req.headers.get_content_charset()
obj = json.loads(req.read().decode(encoding))
In Python 3, binary data, such as the raw response of a http request, is stored in bytes objects. json/simplejson expects strings. The solution is to decode the bytes data to string data with the appropriate encoding, which you can find in the header.
You find the encoding with:
encoding = req.headers.get_content_charset()
You then make the content a string by:
body = req.readall().decode(encoding)
This body you then can pass to the json loader.
(Also, please stop calling the response "req". It's confusing, and makes it sounds like it is a request, which it isn't.)