I am trying to upload a large file (≥3GB) to my FastAPI server, without loading the entire file into memory, as my server has only 2GB of free memory.
Server side:
async def uploadfiles(upload_file: UploadFile = File(...):
Client side:
m = MultipartEncoder(fields = {"upload_file":open(file_name,'rb')})
prefix = "http://xxx:5000"
url = "{}/v1/uploadfiles".format(prefix)
try:
req = requests.post(
url,
data=m,
verify=False,
)
which returns:
HTTP 422 {"detail":[{"loc":["body","upload_file"],"msg":"field required","type":"value_error.missing"}]}
I am not sure what MultipartEncoder actually sends to the server, so that the request does not match. Any ideas?
With requests-toolbelt library, you have to pass the filename as well, when declaring the field for upload_file, as well as set the Content-Type header—which is the main reason for the error you get, as you are sending the request without setting the Content-Type header to multipart/form-data, followed by the necessary boundary string—as shown in the documentation. Example:
filename = 'my_file.txt'
m = MultipartEncoder(fields={'upload_file': (filename, open(filename, 'rb'))})
r = requests.post(url, data=m, headers={'Content-Type': m.content_type})
print(r.request.headers) # confirm that the 'Content-Type' header has been set
However, I wouldn't recommend using a library (i.e., requests-toolbelt) that hasn't provided a new release for over three years now. I would suggest using Python requests instead, as demonstrated in this answer and that answer (also see Streaming Uploads and Chunk-Encoded Requests), or, preferably, use the HTTPX library, which supports async requests (if you had to send multiple requests simultaneously), as well as streaming File uploads by default, meaning that only one chunk at a time will be loaded into memory (see the documentation). Examples are given below.
Option 1 (Fast) - Upload File and Form data using .stream()
As previously explained in detail in this answer, when you declare an UploadFile object, FastAPI/Starlette, under the hood, uses a SpooledTemporaryFile with the max_size attribute set to 1MB, meaning that the file data is spooled in memory until the file size exceeds the max_size, at which point the contents are written to disk; more specifically, to a temporary file on your OS's temporary directory—see this answer on how to find/change the default temporary directory—that you later need to read the data from, using the .read() method. Hence, this whole process makes uploading file quite slow; especially, if it is a large file (as you'll see in Option 2 below later on).
To avoid that and speed up the process, as the linked answer above suggested, one can access the request body as a stream. As per Starlette documentation, if you use the .stream() method, the (request) byte chunks are provided without storing the entire body to memory (and later to a temporary file, if the body size exceeds 1MB). This method allows you to read and process the byte chunks as they arrive. The below takes the suggested solution a step further, by using the streaming-form-data library, which provides a Python parser for parsing streaming multipart/form-data input chunks. This means that not only you can upload Form data along with File(s), but you also don't have to wait for the entire request body to be received, in order to start parsing the data. The way it's done is that you initialise the main parser class (passing the HTTP request headers that help to determine the input Content-Type, and hence, the boundary string used to separate each body part in the multipart payload, etc.), and associate one of the Target classes to define what should be done with a field when it has been extracted out of the request body. For instance, FileTarget would stream the data to a file on disk, whereas ValueTarget would hold the data in memory (this class can be used for either Form or File data as well, if you don't need the file(s) saved to the disk). It is also possible to define your own custom Target classes. I have to mention that streaming-form-data library does not currently support async calls to I/O operations, meaning that the writing of chunks happens synchronously (within a def function). Though, as the endpoint below uses .stream() (which is an async function), it will give up control for other tasks/requests to run on the event loop, while waiting for data to become available from the stream. You could also run the function for parsing the received data in a separate thread and await it, using Starlette's run_in_threadpool()—e.g., await run_in_threadpool(parser.data_received, chunk)—which is used by FastAPI internally when you call the async methods of UploadFile, as shown here. For more details on def vs async def, please have a look at this answer.
You can also perform certain validation tasks, e.g., ensuring that the input size is not exceeding a certain value. This can be done using the MaxSizeValidator. However, as this would only be applied to the fields you defined—and hence, it wouldn't prevent a malicious user from sending extremely large request body, which could result in consuming server resources in a way that the application may end up crashing—the below incorporates a custom MaxBodySizeValidator class that is used to make sure that the request body size is not exceeding a pre-defined value. The both validators desribed above solve the problem of limiting upload file (as well as the entire request body) size in a likely better way than the one desribed here, which uses UploadFile, and hence, the file needs to be entirely received and saved to the temporary directory, before performing the check (not to mention that the approach does not take into account the request body size at all)—using as ASGI middleware such as this would be an alternative solution for limiting the request body. Also, in case you are using Gunicorn with Uvicorn, you can also define limits with regards to, for example, the number of HTTP header fields in a request, the size of an HTTP request header field, and so on (see the documentation). Similar limits can be applied when using reverse proxy servers, such as Nginx (which also allows you to set the maximum request body size using the client_max_body_size directive).
A few notes for the example below. Since it uses the Request object directly, and not UploadFile and Form objects, the endpoint won't be properly documented in the auto-generated docs at /docs (if that's important for your app at all). This also means that you have to perform some checks yourself, such as whether the required fields for the endpoint were received or not, and if they were in the expected format. For instance, for the data field, you could check whether data.value is empty or not (empty would mean that the user has either not included that field in the multipart/form-data, or sent an empty value), as well as if isinstance(data.value, str). As for the file(s), you can check whether file_.multipart_filename is not empty; however, since a filename could likely not be included in the Content-Disposition by some user, you may also want to check if the file exists in the filesystem, using os.path.isfile(filepath) (Note: you need to make sure there is no pre-existing file with the same name in that specified location; otherwise, the aforementioned function would always return True, even when the user did not send the file).
Regarding the applied size limits, the MAX_REQUEST_BODY_SIZE below must be larger than MAX_FILE_SIZE (plus all the Form values size) you expcect to receive, as the raw request body (that you get from using the .stream() method) includes a few more bytes for the --boundary and Content-Disposition header for each of the fields in the body. Hence, you should add a few more bytes, depending on the Form values and the number of files you expect to receive (hence the MAX_FILE_SIZE + 1024 below).
app.py
from fastapi import FastAPI, Request, HTTPException, status
from streaming_form_data import StreamingFormDataParser
from streaming_form_data.targets import FileTarget, ValueTarget
from streaming_form_data.validators import MaxSizeValidator
import streaming_form_data
from starlette.requests import ClientDisconnect
import os
MAX_FILE_SIZE = 1024 * 1024 * 1024 * 4 # = 4GB
MAX_REQUEST_BODY_SIZE = MAX_FILE_SIZE + 1024
app = FastAPI()
class MaxBodySizeException(Exception):
def __init__(self, body_len: str):
self.body_len = body_len
class MaxBodySizeValidator:
def __init__(self, max_size: int):
self.body_len = 0
self.max_size = max_size
def __call__(self, chunk: bytes):
self.body_len += len(chunk)
if self.body_len > self.max_size:
raise MaxBodySizeException(body_len=self.body_len)
#app.post('/upload')
async def upload(request: Request):
body_validator = MaxBodySizeValidator(MAX_REQUEST_BODY_SIZE)
filename = request.headers.get('Filename')
if not filename:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY,
detail='Filename header is missing')
try:
filepath = os.path.join('./', os.path.basename(filename))
file_ = FileTarget(filepath, validator=MaxSizeValidator(MAX_FILE_SIZE))
data = ValueTarget()
parser = StreamingFormDataParser(headers=request.headers)
parser.register('file', file_)
parser.register('data', data)
async for chunk in request.stream():
body_validator(chunk)
parser.data_received(chunk)
except ClientDisconnect:
print("Client Disconnected")
except MaxBodySizeException as e:
raise HTTPException(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
detail=f'Maximum request body size limit ({MAX_REQUEST_BODY_SIZE} bytes) exceeded ({e.body_len} bytes read)')
except streaming_form_data.validators.ValidationError:
raise HTTPException(status_code=status.HTTP_413_REQUEST_ENTITY_TOO_LARGE,
detail=f'Maximum file size limit ({MAX_FILE_SIZE} bytes) exceeded')
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='There was an error uploading the file')
if not file_.multipart_filename:
raise HTTPException(status_code=status.HTTP_422_UNPROCESSABLE_ENTITY, detail='File is missing')
print(data.value.decode())
print(file_.multipart_filename)
return {"message": f"Successfuly uploaded {filename}"}
As mentioned earlier, to upload the data (on client side), you can use the HTTPX library, which supports streaming file uploads by default, and thus allows you to send large streams/files without loading them entirely into memory. You can pass additional Form data as well, using the data argument. Below, a custom header, i.e., Filename, is used to pass the filename to the server, so that the server instantiates the FileTarget class with that name (you could use the X- prefix for custom headers, if you wish; however, it is not officially recommended anymore).
To upload multiple files, use a header for each file (or, use random names on server side, and once the file has been fully uploaded, you can optionally rename it using the file_.multipart_filename attribute), pass a list of files, as described in the documentation (Note: use a different field name for each file, so that they won't overlap when parsing them on server side, e.g., files = [('file', open('bigFile.zip', 'rb')),('file_2', open('bigFile2.zip', 'rb'))], and finally, define the Target classes on server side accordingly.
test.py
import httpx
import time
url ='http://127.0.0.1:8000/upload'
files = {'file': open('bigFile.zip', 'rb')}
headers={'Filename': 'bigFile.zip'}
data = {'data': 'Hello World!'}
with httpx.Client() as client:
start = time.time()
r = client.post(url, data=data, files=files, headers=headers)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.status_code, r.json(), sep=' ')
Upload both File and JSON body
In case you would like to upload both file(s) and JSON instead of Form data, you can use the approach described in Method 3 of this answer, thus also saving you from performing manual checks on the received Form fields, as explained earlier (see the linked answer for more details). To do that, make the following changes in the code above.
app.py
#...
from fastapi import Form
from pydantic import BaseModel, ValidationError
from typing import Optional
from fastapi.encoders import jsonable_encoder
class Base(BaseModel):
name: str
point: Optional[float] = None
is_accepted: Optional[bool] = False
def checker(data: str = Form(...)):
try:
model = Base.parse_raw(data)
except ValidationError as e:
raise HTTPException(detail=jsonable_encoder(e.errors()), status_code=status.HTTP_422_UNPROCESSABLE_ENTITY)
return model
#...
#app.post('/upload')
async def upload(request: Request):
#...
# place this after the try-except block
model = checker(data.value.decode())
print(model.dict())
test.py
#...
import json
data = {'data': json.dumps({"name": "foo", "point": 0.13, "is_accepted": False})}
#...
Option 2 (Slow) - Upload File and Form data using UploadFile and Form
If you would like to use a normal def endpoint instead, see this answer.
app.py
from fastapi import FastAPI, File, UploadFile, Form, HTTPException, status
import aiofiles
import os
CHUNK_SIZE = 1024 * 1024 # adjust the chunk size as desired
app = FastAPI()
#app.post("/upload")
async def upload(file: UploadFile = File(...), data: str = Form(...)):
try:
filepath = os.path.join('./', os.path.basename(file.filename))
async with aiofiles.open(filepath, 'wb') as f:
while chunk := await file.read(CHUNK_SIZE):
await f.write(chunk)
except Exception:
raise HTTPException(status_code=status.HTTP_500_INTERNAL_SERVER_ERROR,
detail='There was an error uploading the file')
finally:
await file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
As mentioned earlier, using this option would take longer for the file upload to complete, and as HTTPX uses a default timeout of 5 seconds, you will most likely get a ReadTimeout exception (as the server will need some time to read the SpooledTemporaryFile in chunks and write the contents to a permanent location on the disk). Thus, you can configure the timeout (see the Timeout class in the source code too), and more specifically, the read timeout, which "specifies the maximum duration to wait for a chunk of data to be received (for example, a chunk of the response body)". If set to None instead of some positive numerical value, there will be no timeout on read.
test.py
import httpx
import time
url ='http://127.0.0.1:8000/upload'
files = {'file': open('bigFile.zip', 'rb')}
headers={'Filename': 'bigFile.zip'}
data = {'data': 'Hello World!'}
timeout = httpx.Timeout(None, read=180.0)
with httpx.Client(timeout=timeout) as client:
start = time.time()
r = client.post(url, data=data, files=files, headers=headers)
end = time.time()
print(f'Time elapsed: {end - start}s')
print(r.status_code, r.json(), sep=' ')
Related
Is there any way to get the response content in a middleware?
The following code is a copy from here.
#app.middleware("http")
async def add_process_time_header(request: Request, call_next):
start_time = time.time()
response = await call_next(request)
process_time = time.time() - start_time
response.headers["X-Process-Time"] = str(process_time)
return response
The response body is an iterator, which once it has been iterated through, it cannot be re-iterated again. Thus, you either have to save all the iterated data to a list (or bytes variable) and use that to return a custom Response, or initiate the iterator again. The options below demonstrate both approaches. In case you would like to get the request body inside the middleware as well, please have a look at this answer.
Option 1
Save the data to a list and use iterate_in_threadpool to initiate the iterator again, as described here - which is what StreamingResponse uses, as shown here.
from starlette.concurrency import iterate_in_threadpool
#app.middleware("http")
async def some_middleware(request: Request, call_next):
response = await call_next(request)
response_body = [chunk async for chunk in response.body_iterator]
response.body_iterator = iterate_in_threadpool(iter(response_body))
print(f"response_body={response_body[0].decode()}")
return response
Note 1: If your code uses StreamingResponse, response_body[0] would return only the first chunk of the response. To get the entire response body, you should join that list of bytes (chunks), as shown below (.decode() returns a string representation of the bytes object):
print(f"response_body={(b''.join(response_body)).decode()}")
Note 2: If you have a StreamingResponse streaming a body that wouldn't fit into your server's RAM (for example, a response of 30GB), you may run into memory errors when iterating over the response.body_iterator (this applies to both options listed in this answer), unless you loop through response.body_iterator (as shown in Option 2), but instead of storing the chunks in an in-memory variable, you store it somewhere on the disk. However, you would then need to retrieve the entire response data from that disk location and load it into RAM, in order to send it back to the client (which could extend the delay in responding to the client even more)—in that case, you could load the contents into RAM in chunks and use StreamingResponse, similar to what has been demonstrated here, here, as well as here, here and here (in Option 1, you can just pass your iterator/generator function to iterate_in_threadpool). However, I would not suggest following that approach, but instead have such endpoints returning large streaming responses excluded from the middleware, as described in this answer.
Option 2
The below demosntrates another approach, where the response body is stored in a bytes object (instead of a list, as shown above), and is used to return a custom Response directly (along with the status_code, headers and media_type of the original response).
#app.middleware("http")
async def some_middleware(request: Request, call_next):
response = await call_next(request)
response_body = b""
async for chunk in response.body_iterator:
response_body += chunk
print(f"response_body={response_body.decode()}")
return Response(content=response_body, status_code=response.status_code,
headers=dict(response.headers), media_type=response.media_type)
I am using FastAPI to return a video response from googlevideo.com. This is the code I am using:
#app.get(params.api_video_route)
async def get_api_video(url=None):
def iter():
req = urllib.request.Request(url)
with urllib.request.urlopen(req) as resp:
yield from io.BytesIO(resp.read())
return StreamingResponse(iter(), media_type="video/mp4")
but this is not working
I want this Nodejs to be converted into Python FastAPI:
app.get("/download-video", function(req, res) {
http.get(decodeURIComponent(req.query.url), function(response) {
res.setHeader("Content-Length", response.headers["content-length"]);
if (response.statusCode >= 400)
res.status(500).send("Error");
response.on("data", function(chunk) { res.write(chunk); });
response.on("end", function() { res.end(); }); }); });
I encountered similar issues but solved all. The main idea is to create a session with requests.Session(), and yield a chunk one by one, instead of getting all the content and yield it at once. This works very nicely without making any memory issue at all.
#app.get(params.api_video_route)
async def get_api_video(url=None):
def iter():
session = requests.Session()
r = session.get(url, stream=True)
r.raise_for_status()
for chunk in r.iter_content(1024*1024):
yield chunk
return StreamingResponse(iter(), media_type="video/mp4")
The quick solution would be to replace yield from io.BytesIO(resp.read()) with the one below (see FastAPI documentation - StreamingResponse for more details).
yield from resp
However, instead of using urllib.request and resp.read() (which would read the entire file contents into memory, hence the reason for taking too long to respond), I would suggest you use the HTTPX library, which, among other things, provides async support as well. Also, it supports Streaming Responses (see async Streaming Responses too), and thus, you can avoid loading the entire response body into memory at once (especially, when dealing with large files). Below are provided examples in both synchronous and asynchronous ways on how to stream a video from a given URL.
Note: Both versions below would allow multiple clients to connect to the server and get the video stream without being blocked, as a normal def endpoint in FastAPI is run in an external threadpool that is then awaited, instead of being called directly (as it would block the server)—thus ensuring that FastAPI will still work asynchronously. Even if you defined the endpoint of the first example below with async def instead, it would still not block the server, as StreamingResponse will run the code (for sending the body chunks) in an external threadpool that is then awaited (have a look at this comment and the source code here), if the function for streaming the response body (i.e., iterfile() in the examples below) is a normal generator/iterator (as in the first example) and not an async one (as in the second example). However, if you had some other I/O or CPU blocking operations inside that endpoint, it would result in blocking the server, and hence, you should drop the async definition on that endpooint. The second example demonstrates how to implement the video streaming in an async def endpoint, which is useful when you have to call other async functions inside the endpoint that you have to await, as well as you thus save FastAPI from running the endpoint in an external threadpool. For more details on def vs async def, please have a look at this answer.
The below examples use iter_bytes() and aiter_bytes() methods, respectively, to get the response body in chunks. These functions, as described in the documentation links above and in the source code here, can handle gzip, deflate, and brotli encoded responses. One can alternatively use the iter_raw() method to get the raw response bytes, without applying content decoding (if is not needed). This method, in contrast to iter_bytes(), allows you to optionally define the chunk_size for streaming the response content, e.g., iter_raw(1024 * 1024). However, this doesn't mean that you read the body in chunks of that size from the server (that is serving the file) directly. If you had a closer look at the source code of iter_raw(), you would see that it just uses a ByteChunker that stores the byte contents into memory (using BytesIO() stream) and returns the content in fixed-size chunks, depending the chunk size you passed to the function (whereas raw_stream_bytes, as shown in the linked source code above, contains the actual byte chunk read from the stream).
Using HTTPX with def endpoint
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import httpx
app = FastAPI()
#app.get('/video')
def get_video(url: str):
def iterfile():
with httpx.stream("GET", url) as r:
for chunk in r.iter_bytes():
yield chunk
return StreamingResponse(iterfile(), media_type="video/mp4")
Using HTTPX with async def endpoint
from fastapi import FastAPI
from fastapi.responses import StreamingResponse
import httpx
app = FastAPI()
#app.get('/video')
async def get_video(url: str):
async def iterfile():
async with httpx.AsyncClient() as client:
async with client.stream("GET", url) as r:
async for chunk in r.aiter_bytes():
yield chunk
return StreamingResponse(iterfile(), media_type="video/mp4")
You can use public videos provided here to test the above. Example:
http://127.0.0.1:8000/video?url=http://commondatastorage.googleapis.com/gtv-videos-bucket/sample/BigBuckBunny.mp4
If you would like to return a custom Response or FileResponse instead—which I wouldn't really recommend in case you are dealing with large video files, as you should either read the entire contents into memory, or save the contents to a temporary file on disk that you later have to read again into memory, in order to send it back to the client—please have a look at this answer and this answer.
I have created an endpoint, as shown below:
#app.post("/report/upload")
def create_upload_files(files: UploadFile = File(...)):
try:
with open(files.filename,'wb+') as wf:
wf.write(file.file.read())
wf.close()
except Exception as e:
return {"error": e.__str__()}
It is launched with uvicorn:
../venv/bin/uvicorn test_upload:app --host=0.0.0.0 --port=5000 --reload
I am performing some tests of uploading a file of around 100 MB using Python requests, and takes around 128 seconds:
f = open(sys.argv[1],"rb").read()
hex_convert = binascii.hexlify(f)
items = {"files": hex_convert.decode()}
start = time.time()
r = requests.post("http://192.168.0.90:5000/report/upload",files=items)
end = time.time() - start
print(end)
I tested the same upload script with an API endpoint using Flask and takes around 0.5 seconds:
from flask import Flask, render_template, request
app = Flask(__name__)
#app.route('/uploader', methods = ['GET', 'POST'])
def upload_file():
if request.method == 'POST':
f = request.files['file']
f.save(f.filename)
return 'file uploaded successfully'
if __name__ == '__main__':
app.run(host="192.168.0.90",port=9000)
Is there anything I am doing wrong?
You can write the file(s) using synchronous writing , after defining the endpoint with def, as shown in this answer, or using asynchronous writing (utilising aiofiles), after defining the endpoint with async def; UploadFile methods are async methods, and thus, you need to await them. Example is given below. For more details on def vs async def and how they may affect your API's performance (depending on the tasks performed inside the endpoints), please have a look at this answer.
Upload Single File
app.py
from fastapi import File, UploadFile
import aiofiles
#app.post("/upload")
async def upload(file: UploadFile = File(...)):
try:
contents = await file.read()
async with aiofiles.open(file.filename, 'wb') as f:
await f.write(contents)
except Exception:
return {"message": "There was an error uploading the file"}
finally:
await file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
Read the File in chunks
Or, you can use async in the chunked manner, to avoid loading the entire file into memory. If, for example, you have 8GB of RAM, you can’t load a 50GB file (not to mention that the available RAM will always be less than the total amount that is installed, as the native OS and other applications running on your machine will use some of the RAM). Hence, in that case, you should rather load the file into memory in chunks and process the data one chunk at a time. This method, however, may take longer to complete, depending on the chunk size you choose; below, that is 1024 * 1024 bytes (= 1MB). You can adjust the chunk size as desired.
from fastapi import File, UploadFile
import aiofiles
#app.post("/upload")
async def upload(file: UploadFile = File(...)):
try:
async with aiofiles.open(file.filename, 'wb') as f:
while contents := await file.read(1024 * 1024):
await f.write(contents)
except Exception:
return {"message": "There was an error uploading the file"}
finally:
await file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
Alternatively, you could use shutil.copyfileobj(), which is used to copy the contents of a file-like object to another file-like object (see this answer as well). By default the data is read in chunks with the default buffer (chunk) size being 1MB (i.e., 1024 * 1024 bytes) for Windows and 64KB for other platforms (see source code here). You can specify the buffer size by passing the optional length parameter. Note: If negative length value is passed, the entire contents of the file will be read—see f.read() documentation as well, which .copyfileobj() uses under the hood. The source code of .copyfileobj() can be found here—there isn't really anything that different from the previous approach in reading/writing the file contents. However, .copyfileobj() uses blocking I/O operations behind the scenes, and this would result in blocking the entire server (if used inside an async def endpoint). Thus, to avoid that , you could use Starlette's run_in_threadpool() to run all the needed functions in a separate thread (that is then awaited) to ensure that the main thread (where coroutines are run) does not get blocked. The same exact function is used by FastAPI internally when you call the async methods of the UploadFile object, i.e., .write(), .read(), .close(), etc.—see source code here. Example:
from fastapi import File, UploadFile
from fastapi.concurrency import run_in_threadpool
import shutil
#app.post("/upload")
async def upload(file: UploadFile = File(...)):
try:
f = await run_in_threadpool(open, file.filename, 'wb')
await run_in_threadpool(shutil.copyfileobj, file.file, f)
except Exception:
return {"message": "There was an error uploading the file"}
finally:
if 'f' in locals(): await run_in_threadpool(f.close)
await file.close()
return {"message": f"Successfuly uploaded {file.filename}"}
test.py
import requests
url = 'http://127.0.0.1:8000/upload'
file = {'file': open('images/1.png', 'rb')}
resp = requests.post(url=url, files=file)
print(resp.json())
Upload Multiple Files
app.py
from fastapi import File, UploadFile
import aiofiles
#app.post("/upload")
async def upload(files: List[UploadFile] = File(...)):
for file in files:
try:
contents = await file.read()
async with aiofiles.open(file.filename, 'wb') as f:
await f.write(contents)
except Exception:
return {"message": "There was an error uploading the file(s)"}
finally:
await file.close()
return {"message": f"Successfuly uploaded {[file.filename for file in files]}"}
Read the Files in chunks
To read the file(s) in chunks instead, see the approaches described earlier in this answer.
test.py
import requests
url = 'http://127.0.0.1:8000/upload'
files = [('files', open('images/1.png', 'rb')), ('files', open('images/2.png', 'rb'))]
resp = requests.post(url=url, files=files)
print(resp.json())
Update
Digging into the source code, it seems that the latest versions of Starlette (which FastAPI uses underneath) use a SpooledTemporaryFile (for UploadFile data structure) with max_size attribute set to 1MB (1024 * 1024 bytes) - see here - in contrast to older versions where max_size was set to the default value, i.e., 0 bytes, such as the one here.
The above means, in the past, data used to be fully loaded into memory no matter the size of file (which could lead to issues when a file couldn't fit into RAM), whereas, in the latest version, data is spooled in memory until the file size exceeds max_size (i.e., 1MB), at which point the contents are written to disk; more specifically, to the OS's temporary directory (Note: this also means that the maximum size of file you can upload is bound by the storage available to the system's temporary directory.. If enough storage (for your needs) is available on your system, there's nothing to worry about; otherwise, please have a look at this answer on how to change the default temporary directory). Thus, the process of writing the file multiple times—that is, initially loading the data into RAM, then, if the data exceeds 1MB in size, writing the file to temporary directory, then reading the file from temporary directory (using file.read()) and finally, writing the file to a permanent directory—is what makes uploading file slow compared to using Flask framework, as OP noted in their question (though, the difference in time is not that big, but just a few seconds, depending on the size of file).
Solution
The solution (if one needs to upload files quite larger than 1MB and uploading time is important to them) would be to access the request body as a stream. As per Starlette documentation, if you access .stream(), then the byte chunks are provided without storing the entire body to memory (and later to temporary directory, if the body contains file data that exceeds 1MB). Example is given below, where time of uploading is recorded on client side, and which ends up being the same as when using Flask framework with the example given in OP's question.
app.py
from fastapi import Request
import aiofiles
#app.post('/upload')
async def upload(request: Request):
try:
filename = request.headers['filename']
async with aiofiles.open(filename, 'wb') as f:
async for chunk in request.stream():
await f.write(chunk)
except Exception:
return {"message": "There was an error uploading the file"}
return {"message": f"Successfuly uploaded {filename}"}
In case your application does not require saving the file to disk, and all you need is the file to be loaded directly into memory, you can just use the below (make sure your RAM has enough space available to accommodate the accumulated data):
from fastapi import Request
#app.post('/upload')
async def upload(request: Request):
body = b''
try:
filename = request.headers['filename']
async for chunk in request.stream():
body += chunk
except Exception:
return {"message": "There was an error uploading the file"}
#print(body.decode())
return {"message": f"Successfuly uploaded {filename}"}
test.py
import requests
import time
with open("images/1.png", "rb") as f:
data = f.read()
url = 'http://127.0.0.1:8000/upload'
headers = {'filename': '1.png'}
start = time.time()
resp = requests.post(url=url, data=data, headers=headers)
end = time.time() - start
print(f'Elapsed time is {end} seconds.', '\n')
print(resp.json())
For more details and code examples (on uploading multiple Files and Form/JSON data) using the approach above, please have a look at this answer.
I was able to upload large file to server using the below code -
#app.route("/upload", methods=["POST"])
def upload():
with open("/tmp/output_file", "bw") as f:
chunk_size = 4096
while True:
chunk = request.stream.read(chunk_size)
if len(chunk) == 0:
return
f.write(chunk)
But if I use request.form['userId'] or any parameter which is sent as form data in the above code it fails.
As per one of the blog post it says- Flask’s request has a stream, that will have the file data you are uploading. You can read from it treating it as a file-like object. The trick seems to be that you shouldn’t use other request attributes like request.form or request.file because this will materialize the stream into memory/file. Flask by default saves files to disk if they exceed 500Kb, so don’t touch file.
Is there a way where we can send additional parameters like userId along with the file being uploaded in flask?
use headers in requests.
if you want to send user name along with data
headers['username'] = 'name of the user'
r = requests.post(url, data=chunk, headers=headers)
I try to use falcon package in my project. Problem is I didn't find a way to get body data from the HTTP post request.
I used code from example, but req.stream.read() doesn't return JSON as expected.
The code is:
raw_json = req.stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
How to get the POST data?
Thanks for any help
in falcon 2, if you work with json type, use req.media
for example:
import falcon
from json import dumps
class Resource(object):
def on_post(self, req, resp, **kwargs):
result = req.media
# do your job
resp.body = dumps(result)
api = falcon.API()
api.add_route('/test', Resource())
Little digging into the problem led to the following linked issue on github. It states that falcon framework at least in its version 0.3 and working with Python 2 didn't parse data 'POSTed' as string if they are aptly escaped. We could use more information on what data you are trying to send over POST request and in what format is that being sent, as in if its being send as simple text, or with Header Information Content-Type:application/json, or if its coming through an HTML form.
While the exact issue is not clear from the question I could still suggest trying to use bounded_stream instead of stream as in:
raw_json = req.bounded_stream.read()
result.json(raw_json, encoding='utf-8')
resp.body = json.dumps(result_json, encoding='utf-8')
for the official documentation suggests use of bounded_stream where uncertain conditions such as Content-Length undefined or 0, or if header information is missing altogether.
bounded_stream is described as the following in the official falcon documentation.
File-like wrapper around stream to normalize certain differences between the native input objects employed by different WSGI servers. In particular, bounded_stream is aware of the expected Content-Length of the body, and will never block on out-of-bounds reads, assuming the client does not stall while transmitting the data to the server.
Falcon receives the HTTP requests data as buffer object as passed by WSGI wrapper which receives the data from client, and its possible it doesn't run proper parsing on top of the data to convert to a more usable data structure for performance reasons.
Big thanks to Ryan (and Prateek Jain) for the answer.
The solution is simply to put app.req_options.auto_parse_form_urlencoded=True. For example:
import falcon
class ThingsResource(object):
def on_post(self, req, resp):
value = req.get_param("value", required=True)
#do something with value
app = falcon.API()
app.req_options.auto_parse_form_urlencoded=True
things = ThingsResource()
app.add_route('/things', things)
The field you're looking for is somewhat confusingly named, but it's req.media:
Returns a deserialized form of the request stream. When called, it will attempt to deserialize the request stream using the Content-Type header as well as the media-type handlers configured via falcon.RequestOptions.
If the request is JSON, req.media already contains a python dict.
I have added changes in request.py in falcon framework to parse application/x-www-form-urlencoded and multipart/from-data.
I have raised pull request - https://github.com/falconry/falcon/pull/1236 but it is not yet merged in master.
Check this - https://github.com/branelmoro/falcon
I have added new code to parse POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data.
Text fields will be available in req.form_data dictionary and upload file buffer stream will be available in req.files dictionary.
I hope this will help to access POST and GET parameters separately and we will be able to upload files as well.
Good thing about the change is that it will not load entire uploaded file in memory.
Below is sample code to show how to use POST, PUT and DELETE application/x-www-form-urlencoded and multipart/form-data:
import falcon
class Resource(object):
def on_post(self, req, resp):
# req.form_data will return dictionary of text field names and their values
print(req.form_data)
# req.form_data will return dictionary of file field names and
# their buffer class FileStream objects as values
print(req.files)
# support we are uploading a image.jpg in `pancard` file field then
# req.files["pancard"] will be FileStream buffer object
# We can use set_max_upload_size method to set maximum allowed
# file size let say 1Mb = 1*1024*1024 bytes for this file
req.files["pancard"].set_max_upload_size(1*1024*1024)
# We can use uploadto method to upload file on required path (Note: absolute filepath is required)
# This method returns boolean - `True` on successful upload
# and if upload is unsuccessful then it returns `False` and sets error on failure.
path = "/tmp/" + req.files["pancard"].name
response = req.files["pancard"].uploadto("/tmp/" + path)
print(response)
# Once file is uploaded sucessfully, we can check it's size
print(req.files["pancard"].size)
# If file is not uploaded sucessfully, we can check it's error
print(req.files["pancard"].error)
resp.body = "Done file upload"
resp.status = falcon.HTTP_200
# falcon.API instances are callable WSGI apps
app = falcon.API()
things = Resource()
# things will handle post requests to the '/post_path' URL path
app.add_route('/post_path', things)
Do let me know if you have any doubts.
So far... for me bounded_stream.read() and stream.read() both get the posted data as type str. I have only found one way around the issue so far:
def on_post(self, req, resp):
posted_data = json.loads(req.stream.read())
print(str(type(posted_data)))
print(posted_data)
Loading the string into a json dict once the posted data is received is my only solution that I can come up with
Here's something I used while designing an API.
import falcon
import json
class VerifierResource():
def on_post(self, req, resp):
credentials = json.loads(req.stream.read())
if credentials['username'] == USER \
and credentials['passwd'] == PASSWORD:
resp.body = json.dumps({"status": "verified"})
else:
resp.body = json.dumps({"status": "invalid"})
api = falcon.API()
api.add_route('/verify', VerifierResource())
This returns a serialized JSON with corresponding response body.
there is a sample way to get media from body. I use to get the body in the post method:
def on_post(req,resp)
arguments = {}
# get body media on post method
body = req.get_media()
if 'something' in body:
arguments['something'] = body['something']
send body content type Media-Type and print resp or use in code, but if want to send JSON body your code should cover give JSON parameters.
Do let me know if you have any doubts.