So trying I'm POSTing a compressed file via httplib2 in Python 3.2. I get the following error:
io.UnsupportedOperation: fileno
I used to post just an xml file but since those files are getting too big I want to compress them inside the memory first.
This is how I create the compressed file in memory:
contentfile = open(os.path.join(r'path', os.path.basename(fname)), 'rb')
tempfile = io.BytesIO()
compressedFile = gzip.GzipFile(fileobj=tempfile, mode='wb')
compressedFile.write(contentfile.read())
compressedFile.close()
tempfile.seek(0)
and this is how I'm trying to POST it.
http.request(self.URL,'POST', tempfile, headers={"Content-Type": "application/x-gzip", "Connection": "keep-alive"})
Any ideas ?
Like i said, it worked well when using the xml file i.e. contentfile
Solved by providing the "Content-Length" header which obviously removes the need for httplib2 to check the length.
Related
I am trying to setup a script where I upload a file (now using the python requests library) to a Flask environment that runs inside of a Docker (-compose) container. The python script is ran in the hypervisor, and I force it to use the same version of python (3.6) as well. I am able to get responses from the server, but the file I upload is 12Kb, and the file the Flask container receives is 2Kb, and I have no clue what is going wrong.
It seems like when I use WireShark to capture the tcp stream, I receive a 2Kb file as well, so my guess the requests library applies some compression, but I can not seem to find any documentation about this happening.
I have tried to replace the file tuple in the sending code with solely the file handle, but this seemed to have no effect. Sending files of a different size results in a different filesize in Flask / Docker.
Sending a string instead of the filehandler ("1234567890") results in the filesize being as big as the string length (10 bytes).
Replacing the file opening method from rb to r results in a UnicodeDecodeError: 'ascii' codec can't decode byte 0xdf in position 14: ordinal not in range(128) raise inside requests -> encodings.
Hypervisor: send.py
import requests
with open('file.docx', 'rb') as f:
url = 'http://localhost:8081/file'
r = requests.post(url, files={'file': ('file', f, 'multipart/form-data')})
print(r.text)
Flask: file.py
#app.route('/file', methods=['POST'])
def parse_from_post():
file = request.files['file']
fn = secure_filename("file.docx") # did this manually instead of getting it from the request for testing reasons
folder = "/app/files/"
fl = os.path.join(folder, fn)
# Removes old file
if os.path.exists(fl):
os.remove(fl)
file.save(fl)
return ""
The problem lies with the filesize, which python requests does not take care of directly. I used the MultipartEncoder from the requests_toolbelt package, to encapsulate the file instead of directly plugging in the file in the requests post call.
Hypervisor: send.py
import requests
from requests_toolbelt import MultipartEncoder
with open('file.docx', 'rb') as f:
url = 'http://localhost:8081/file'
m = MultipartEncoder(fields={
"file": ("file.docx", f)
})
r = requests.post(url, data=m, headers={'Content-Type': m.content_type})
print(r.text)
I actually found this result from another post on SO, see the first comment on the question linking to https://toolbelt.readthedocs.io/....
I have a public viewable JSON file that is hosted on s3. I can access the file directly by clicking on the public url to the s3 object and view the JSON in my browser fine. Anyone on the internet can view the file easily.
Yet with the below code is ran in Python (using Lambda connected to an API trigger) I get [Errno 2] No such file or directory: as the errorMessage, and FileNotFoundError as the errorType.
def readLeetDictionary(self):
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
with open(jsonfile, 'r') as data_file:
self.data = json.load(data_file)
What am I missing here? Since the file is a publicly viewable JSON file I would assume I wouldn't be forced to use boto3 library and formally handshake to the file in order to read the file (with object = s3.get_object(Bucket=bucket_names,Key=object_name) for example) - would I?
The conde you need should be something like:
import urllib, json
def readLeetDictionary():
jsonfile = 'https://s3.amazonaws.com/url/to/public/facing/file/data.json'
response = urllib.urlopen(jsonfile)
data = json.loads(response.read())
print data
Please feel free to ask further or explain if this does not suit you.
I am trying to upload a file using Rest to a DJango Python API. But I noticed the file gets modified. Specifically a content-disposition is added to it. I haven't found a good way to remove this. The problem is I am trying to upload a tar that needs to be unzipped, but the modified content prevents unzipping the file.
I’m using this file parser on a rest page:
from rest_framework.parsers import FileUploadParser
The following code seems to get the file for me in the post method of an APIView
file_obj = request.FILES['file']
scanfile.file.save(file_obj.name, file_obj)
Where scanfile is a model with a file field.
The file gets saved with contents like this:
--b3c91a6c13e34fd5a1e253b1a72d63b3
Content-Disposition: form-data; name="file"; filename="sometar.tgz"
My tar file contents here.....
--b3c91a6c13e34fd5a1e253b1a72d63b3
My client looks like this:
filename = "sometar.tgz"
exclusion = "../../exclusionlist.txt"
headers = {'Content-Type': 'multipart/form-data;’,
'Authorization': 'JWT %s' % token,
}
url = "http://localhost:%s/api/scan/Project/%s/" % (port, filename)
#files = {'file': open(filename, 'rb'), 'exclusion_file': open(exclusion, 'rb')} # also tried this way but it just put the info in the same file and I see the headers in the file
files = [('file', open(filename, 'rb')), ('file', open(exclusion, 'rb'))]
x = requests.post(url, files=files, headers=headers)
So my question is how do I remove that content-disposition info from the saved file so I can properly unzip the file?
request.FILES['file'] is an UploadedFile object. You can get its name with request.FILES['file'].name and get just the content with request.FILES['file'].read().
You should be careful with read() and big files:
Read the entire uploaded data from the file. Be careful with this
method: if the uploaded file is huge it can overwhelm your system if
you try to read it into memory. You’ll probably want to use chunks()
instead; see below.
https://docs.djangoproject.com/en/1.11/ref/request-response/#django.http.HttpRequest.FILES
https://docs.djangoproject.com/en/1.11/ref/files/uploads/#django.core.files.uploadedfile.UploadedFile
I am trying to upload a file using python requests module and i am not sure whether we can use both data and files in the post call.
fileobj= open(filename,'rb')
upload_data = {
'data':payload,
'file':fileobj
}
resp = s.post(upload_url,data=upload_data,headers=upload_headers)
and this is not working. So can anyone help me with this ?
I think you should be using the data and files keyword parameters in the post request to send the data and file respectively.
with open(filename,'rb') as fileobj:
files = {'file': fileobj}
resp = s.post(upload_url,data=payload,files=files,headers=upload_headers)
I've also use a context manager just because it closes the file for me and takes care of exceptions that happen either during file opening or during something that happens with the requests post.
I would like to know, how to upload a file from memory to Flickr.
I am using the Python Flickr API kit (http://stuvel.eu/flickrapi).
Does the file in memory have a path that can be passed as filename?
Code
response = flickr.upload(filename=f.read(), callback=None, **keywords)
Error
TypeError at /image/new/
must be encoded string without NULL bytes, not str
Thanks in advance
You can try using the tempfile module to write it to disk before uploading it
import tempfile
with tempfile.NamedTemporaryFile(delete=True) as tfile:
tfile.write(f.read())
tfile.flush()
response = flickr.upload(filename=tfile.name,callback=None,**keywords)