We Keep Coding

multipart form data for post in python

Can someone explain to me what multi part form data is. I am trying to post something into an api using python but it seems that i need to use multi part form data and not just the json file. Am i able to post multiple json files with this form of data?
with a quick google search I was able to upload 1 file in this formate but i would like to understand the formate of it and how to use it/ if it can be used to upload multiple files.
My project goal is to automate this process where I place Json files into a folder on my computer and run a script where 1 by 1 it uploads it into API but not sure how to do it with the multipart form data
files = {
'file': open("filltest.json",'rb')
}
response = requests.post(url=api_endpoint, files=files)
this is what i was able to create after googling it but an explanation would be super helpful.
thanks!

Stream multiple files with a single request using python requests

I'm using python's requests library to upload images into a server created by django / drf. It is a simple task and test script looks like this:
files = {
'client_files[0]': ('415.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\415.png', 'rb')),
'client_files[1]': ('422.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\422.png', 'rb')),
'client_files[2]': ('485.png', open('E:\\Olaylar\\test_set\\cvat_test_small\\485.png', 'rb'))
}
payload = {
'image_quality': 50
}
response = requests.post(post_url, auth=auth, data=payload,files=files)
The problem is, I need to upload hundreds, if not thousands of images, and in a single request. So when I try to read all files into memory in order to post them, I run out of memory quite fast.
I know there is a possible solution, by putting a generator in data argument of post method, and streaming the data by chunks. That would solve my problem quite well, but requests library does not support that functionality in files argument of the post method too.
I will be grateful for any help or advice how to solve this problem. Thanks in advance

HPE_UNEXPECTED_CONTENT_LENGTH error when Content-Length to BytesIO length

I'm moving a python pyramid project from python 2 to 3. I was using ReportLab to generate PDF files and send it to the front end. According to their examples I need to use io.BytesIO(), when previously it was StringIO().
Now using the generated document length to set the Content-Length in my response, I get an HPE_UNEXPECTED_CONTENT_LENGTH error.
pdf = io.BytesIO()
doc = SimpleDocTemplate(pdf)
doc.build(story)
pdfcontent = pdf.getvalue()
pdf.close()
response = Response(content_type='application/pdf', body=pdfcontent)
response.headers.add("Content-Length", str(len(pdfcontent)))
If I don't set the Content-Length attribute the download works fine, but I would rather not leave it blank.

I'm not sure about you particular example and error, but I'm pretty sure that when you provide the response body bytes like this, Pyramid sends the Content-Length header. No need to set it manually, it already has the bytes and therefore knows its size.
You should check the response headers (using your browser developer tools or a command line tools like curl or httpie).

Send PDF file from Django website to LogicalDOC

I'm developing my Django website since about 2 months and I begin to get a good global result with my own functions.
But, now I have to start a very hard part (to my mind) and I need some advices, ideas before to do that.
My Django website creates some PDF files from HTML templates with Django variables. Up to now, I'm saving PDF files directly on my Desktop (in a specific folder) but it's completely unsecured.
So, I installed another web application which is named LogicalDoc in order to save PDF file directly on this application. PDF files are created and sent to LogicalDoc.
LogicalDoc owns 2 API : SOAP and REST (http://wiki.logicaldoc.com/rest/#/) and I know that Django could communicate with REST method.
I'm reading this part of Django documentation too in order to understand How I can process : https://docs.djangoproject.com/en/dev/topics/http/file-uploads/
I made a scheme in order to understand what I'm exposing :
Then, I write a script which makes some things :
When the PDF file is created, I create a folder inside LogicalDoc which takes for example the following name : lastname_firstname_birthday
Two possibilities : If the folder exists,I don't create a new folder, else I create it.
Once it's done, I send the PDF file directly inside the folder by comparing PDF name with folder name to do that
I have some questions about this process :
Firstly, is it possible to make this kind of things ?
Is it hard to do that ?
What kind of advices could you give me ?
Thank you so much !
PS : If you need some part of my script, mainly PDF creating part, I can post it just after my question ;)

An idea is pretty simple, however it always requires some practice.
I strongly advice you to use REST api and forget about SOAP as the only thing it can bring to you - is 'pain' :)
If we check documentation, document/create it gives next information.
Endpoint we have to communicate with.
[protocol]://[server]:[port]/document/create
HTTP method to use - POST
List of parameters to provide with your request: body,
document, content
Even more, you can test API by clicking on "Try it out" button and check requests in "Network" tab of your browser (if you open Developer Tools)
I am not sure what kind of metadata do you have to provide in 'document' parameter but what I know you can easy get an idea of what should be done by testing it and putting XML or JSON data into 'document' parameter.
Content is an array of bytes transferred to the server (which would be your file).
To sum up, a request to 'document/create' uri will be simple
body = { 'headers': {},'object': {},}
document = "<note>data</note>"
content=open('report.xls', 'rb') #r - reading, b - binary
r = requests.post('http://logicaldoc/document/create', body=body, document=document, content=content)
Please keep in mind that file transferring requests take time and sometimes you may get timeout exception. Your code will stop and will be waiting for response, so it may be a good idea to get some practice with asyncio or celery. Just keep in mind those kind of possible issues.

Given a URL, how to encode a file's contents as base64 with Python / Django?

I am building a Django-based website, and am having trouble figuring out a decent way to email some larger PDFs and such to my users.
The files in question never touch our servers; they're handled on a CDN. So, my starting point is with the unique URLs for the files, not with the files themselves. It would be nice to find a solution that doesn't involve saving the files locally.
In order for me to be able to send the email in the way I want (with the PDF/DOCX/whatever attached to it), I need to be able to encode the attachment as a base-64 string.
I would prefer not to save the file to our server; I would also prefer not to read a response object in chunks and write it plainly to a file on our server, then encode that file.
That said, given a direct url to a file is there a way to stream the response and encode it in base64 as it comes in?
I have been reading about Django's StreamingHttpResponse and FileWrapper and feel like I am close, but I'm not able to put it together just yet.
Edit: the snippet below is working for now, but I'm worried about memory usage - how well would something like this scale?
import base64
req = requests.get('url')
encoded = base64.b64encode(req.content)

Thanks to beetea I am comfortable implementing the simple:
import base64
req = requests.get('url')
encoded = base64.b64encode(req.content)
As the solution to this issue.