I'm using python's requests library to upload images into a server created by django / drf. It is a simple task and test script looks like this:
files = {
'client_files[0]': ('415.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\415.png', 'rb')),
'client_files[1]': ('422.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\422.png', 'rb')),
'client_files[2]': ('485.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\485.png', 'rb'))
}
payload = {
'image_quality': 50
}
response = requests.post(post_url, auth=auth, data=payload,files=files)
The problem is, I need to upload hundreds, if not thousands of images, and in a single request. So when I try to read all files into memory in order to post them, I run out of memory quite fast.
I know there is a possible solution, by putting a generator in data argument of post method, and streaming the data by chunks. That would solve my problem quite well, but requests library does not support that functionality in files argument of the post method too.
I will be grateful for any help or advice how to solve this problem. Thanks in advance
Related
Can someone explain to me what multi part form data is. I am trying to post something into an api using python but it seems that i need to use multi part form data and not just the json file. Am i able to post multiple json files with this form of data?
with a quick google search I was able to upload 1 file in this formate but i would like to understand the formate of it and how to use it/ if it can be used to upload multiple files.
My project goal is to automate this process where I place Json files into a folder on my computer and run a script where 1 by 1 it uploads it into API but not sure how to do it with the multipart form data
files = {
'file': open("filltest.json",'rb')
}
response = requests.post(url=api_endpoint, files=files)
this is what i was able to create after googling it but an explanation would be super helpful.
thanks!
I am trying to upload a file through python requests using the PUT method. But on the server side, the file is never received.
The code that I am using is:
files = {'test' : open(r"C:\Users\test.jar", 'rb')}
response = session.put(api_base + url.get('url').format(foo, bar),
headers=headers, data=data, files=files)
Does PUT ignore file uploads? Is it only valid for posts?
What am I doing wrong?
It seems that Python's request module does not work for file uploads if you use the PUT method as you did. The documentation only shows one way to use POST requests. I'm not sure if this is a bug or not working intentionally.
Others also have problems using PUT requests for file transfer, see e.g:
Cannot PUT file to Django REST API by python requests
Python requests PUT method creates a zero byte file
OP moved to PycURL, which seems to work and also seems to be a good alternative.
I connect via Python to a web interface, where I get back as response files in multipart/form-data format.
I know the format type only when a browser responds to a form, but here the server sends its response in this format.
How can I get the original file from this answer without the metadata of the interface?
I have here a few examples with one, two, three and four contents.
I have no idea how to solve it and ask for your help.
I uploaded the files to zippyshare, because is to big for text view.
http://www22.zippyshare.com/v/EEXIbj79/file.html
http://www22.zippyshare.com/v/eXF62wpq/file.html
http://www22.zippyshare.com/v/sSi9crCT/file.html
http://www22.zippyshare.com/v/RiXF57WD/file.html
Thank you in advance
for http/s this works for me.
r = requests.get(URL, headers={'Connection': 'close'})
boundary = r.headers["Content-Type"].split(";", 1)[1].strip().replace("boundary=", "", 1)
comps = Content.split(boundary.encode())
I'm developing my Django website since about 2 months and I begin to get a good global result with my own functions.
But, now I have to start a very hard part (to my mind) and I need some advices, ideas before to do that.
My Django website creates some PDF files from HTML templates with Django variables. Up to now, I'm saving PDF files directly on my Desktop (in a specific folder) but it's completely unsecured.
So, I installed another web application which is named LogicalDoc in order to save PDF file directly on this application. PDF files are created and sent to LogicalDoc.
LogicalDoc owns 2 API : SOAP and REST (http://wiki.logicaldoc.com/rest/#/) and I know that Django could communicate with REST method.
I'm reading this part of Django documentation too in order to understand How I can process : https://docs.djangoproject.com/en/dev/topics/http/file-uploads/
I made a scheme in order to understand what I'm exposing :
Then, I write a script which makes some things :
When the PDF file is created, I create a folder inside LogicalDoc which takes for example the following name : lastname_firstname_birthday
Two possibilities : If the folder exists,I don't create a new folder, else I create it.
Once it's done, I send the PDF file directly inside the folder by comparing PDF name with folder name to do that
I have some questions about this process :
Firstly, is it possible to make this kind of things ?
Is it hard to do that ?
What kind of advices could you give me ?
Thank you so much !
PS : If you need some part of my script, mainly PDF creating part, I can post it just after my question ;)
An idea is pretty simple, however it always requires some practice.
I strongly advice you to use REST api and forget about SOAP as the only thing it can bring to you - is 'pain' :)
If we check documentation, document/create it gives next information.
Endpoint we have to communicate with.
[protocol]://[server]:[port]/document/create
HTTP method to use - POST
List of parameters to provide with your request: body,
document, content
Even more, you can test API by clicking on "Try it out" button and check requests in "Network" tab of your browser (if you open Developer Tools)
I am not sure what kind of metadata do you have to provide in 'document' parameter but what I know you can easy get an idea of what should be done by testing it and putting XML or JSON data into 'document' parameter.
Content is an array of bytes transferred to the server (which would be your file).
To sum up, a request to 'document/create' uri will be simple
body = { 'headers': {},'object': {},}
document = "<note>data</note>"
content=open('report.xls', 'rb') #r - reading, b - binary
r = requests.post('http://logicaldoc/document/create', body=body, document=document, content=content)
Please keep in mind that file transferring requests take time and sometimes you may get timeout exception. Your code will stop and will be waiting for response, so it may be a good idea to get some practice with asyncio or celery. Just keep in mind those kind of possible issues.
I'm working on a project that involves uploading an image to tumblr from Python. I've had luck using Tumblr's API( http://www.tumblr.com/docs/en/api ) in doing regular text-posts, but image uploads have been giving me trouble. The error messages their server returns have been limited to just telling me that there was an "Error Uploading Photo", which has been less than helpful.
Since their API seems to be based in using standard HTTP POST operations, I know that there has to be a way to do this. Unfortunately, I haven't made any progress for a couple of days, and I've decided to resort to bothering you guys about it.
I have tried using curl, and python's libraries: httplib, urllib, urllib2, and a third party library called urllib2_file (http://fabien.seisen.org/python/urllib2_file/). I'm frustrated that I haven't gotten them to work-- but I'm willing to try other additional terminal apps you can come up with.
Each method works fine with simple text posts, but each one of them doesn't seem to get the photo uploading done properly.
Here's my syntax for doing it with urllib2_file. Since urllib2 doesn't support 'multipart/form-data' methods for uploading data, I'm using urllib2_file to add that functionality-- but I haven't been able to get it to work. The tumblr api says that their servers accept multipart/form-data as well as the 'normal post' method for uploading files. I'd be happy if either worked.
import urllib, urllib2, urllib2_file
url = "http://www.tumblr.com/api/write"
values1 = { 'email':'EMAIL',
'password':'PASSWORD',
'type':'regular',
'title':'Pythons urllib2',
'body':'its pretty nice. Not sure how to make it upload stuff yet, though. Still getting some "error uploading photo" errors... So unhelpful.'}
values2 = { 'email':'EMAIL',
'password':'PASSWORD',
'type':'photo',
'data': open('../data/media/pics/2009/05-14/100_1167.JPG'),
'caption':'Caption'}
data = urllib.urlencode(values2)
print "just before defining the request"
req = urllib2.Request(url,data)
print "just before doing the urlopen."
#response = urllib2.urlopen(req)
try:
response = urllib2.urlopen(req)
except urllib2.URLError, e:
print e.code
print e.read()
print "figure out how to handle .read() properly"
#the_page = response.read()
#print the_page
print "done"
This would be the ideal way if it worked since using dictionaries to define the fields is really easy and I could make it look much cleaner in the future.
Any advice on how to troubleshoot what could be going wrong would be appreciated. At this point I don't know how to learn what could be going wrong. I wish I had the attention span for the http RFC.
I've been considering sniffing the packets between my computer at the server-- but reverse-engineering HTTP might be overkill.
Thanks!
'data': open('../data/media/pics/2009/05-14/100_1167.JPG'),
Looks like you're just passing in a file object .. add a .read() there
Tumblr has API v2 defined for Python. You can find it at GitHub PyTumblr.
I have used it to create a terminal based tool for using tumblr which is called teblr. You can find the source code here: https://github.com/vijaykumarhackr/teblr/