I am trying to write endpoint for image uploading, it will accept just image in base64 format. I have read how to get file from request in FastAPI and implement that in this way:
server side
#router.post("/v1/installation/{some_identifier}/image")
#db.create_connection
async def upload_installation_image(some_identifier: int, data: UploadFile = File(...)):
...
content = await data.read()
image_uuid = await save_image(installation.uuid, content, data.content_type)
return {"image_uuid": image_uuid}
And its works fine, but in this case I need to send data in this way:
client side
def test_upload_image(...):
...
response = session.post(
f"/v1/installation/{relink_serial}/image", files={"file": ("test_file_name.png", data, "image/png")}
)
...
But I want to be able to upload image like this:
client side
def test_upload_image(...):
...
response = session.post(
f"/v1/installation/{installation_uuid}/image", data=data, headers={"content-type": "image/png"}
)
I have read a lot of articles and other questions, but all of them suggest to use UploadFile and send data as json with file or body parameter.
Is this even possible, to load just 1 file like in second test I show?
I have find solution. Instead of using predefined FastAPI types, I can directly get all data from request object, so my code looks like this:
#router.post("/v1/installation/{some_identifier}/image")
#db.create_connection
async def upload_installation_image(some_identifier: int, request: Request):
...
content = await request.body()
image_uuid = await save_image(installation.uuid, content, content_type)
return {"image_uuid": image_uuid}
Thats exactly what I am looking for, because in my case client app have send to me whole file in body.
Related
Is there any way to get the response content in a middleware?
The following code is a copy from here.
#app.middleware("http")
async def add_process_time_header(request: Request, call_next):
start_time = time.time()
response = await call_next(request)
process_time = time.time() - start_time
response.headers["X-Process-Time"] = str(process_time)
return response
The response body is an iterator, which once it has been iterated through, it cannot be re-iterated again. Thus, you either have to save all the iterated data to a list (or bytes variable) and use that to return a custom Response, or initiate the iterator again. The options below demonstrate both approaches. In case you would like to get the request body inside the middleware as well, please have a look at this answer.
Option 1
Save the data to a list and use iterate_in_threadpool to initiate the iterator again, as described here - which is what StreamingResponse uses, as shown here.
from starlette.concurrency import iterate_in_threadpool
#app.middleware("http")
async def some_middleware(request: Request, call_next):
response = await call_next(request)
response_body = [chunk async for chunk in response.body_iterator]
response.body_iterator = iterate_in_threadpool(iter(response_body))
print(f"response_body={response_body[0].decode()}")
return response
Note 1: If your code uses StreamingResponse, response_body[0] would return only the first chunk of the response. To get the entire response body, you should join that list of bytes (chunks), as shown below (.decode() returns a string representation of the bytes object):
print(f"response_body={(b''.join(response_body)).decode()}")
Note 2: If you have a StreamingResponse streaming a body that wouldn't fit into your server's RAM (for example, a response of 30GB), you may run into memory errors when iterating over the response.body_iterator (this applies to both options listed in this answer), unless you loop through response.body_iterator (as shown in Option 2), but instead of storing the chunks in an in-memory variable, you store it somewhere on the disk. However, you would then need to retrieve the entire response data from that disk location and load it into RAM, in order to send it back to the client (which could extend the delay in responding to the client even more)—in that case, you could load the contents into RAM in chunks and use StreamingResponse, similar to what has been demonstrated here, here, as well as here, here and here (in Option 1, you can just pass your iterator/generator function to iterate_in_threadpool). However, I would not suggest following that approach, but instead have such endpoints returning large streaming responses excluded from the middleware, as described in this answer.
Option 2
The below demosntrates another approach, where the response body is stored in a bytes object (instead of a list, as shown above), and is used to return a custom Response directly (along with the status_code, headers and media_type of the original response).
#app.middleware("http")
async def some_middleware(request: Request, call_next):
response = await call_next(request)
response_body = b""
async for chunk in response.body_iterator:
response_body += chunk
print(f"response_body={response_body.decode()}")
return Response(content=response_body, status_code=response.status_code,
headers=dict(response.headers), media_type=response.media_type)
I'm trying to communicate with a fingerprint device. Actaully it sends data through a websocket connection. So, I think I can communicate with the device using webscokets. Here I'm using FastAPI, but it only accepts JSON data. The problem is that I need to handle XML data, however, I do not know how to send and accept data in XML format.
FastAPI can accept and validate other types of data as well, not only JSON as you stated. Have a look at the documentation. Regarding XML, as FastAPI is actually Starlette underneath, you can use Starlette's Request object directly to read the request body as bytes, and return a custom Response with the XML data (if required). You can check if the incoming request is of the required Content-Type, and if so, let it through; otherwise, you could raise an HTTPException. Below is a working example using Python requests on client side and a normal HTTP endpoint on server side.
Using HTTP Protocol
app.py
from fastapi import FastAPI, Response, Request, HTTPException
app = FastAPI()
#app.post("/submit")
async def submit(request: Request):
content_type = request.headers['Content-Type']
if content_type == 'application/xml':
body = await request.body()
return Response(content=body, media_type="application/xml")
else:
raise HTTPException(status_code=400, detail=f'Content type {content_type} not supported')
test.py
import requests
body = """<?xml version='1.0' encoding='utf-8'?><a>б</a>"""
headers = {'Content-Type': 'application/xml'}
url = 'http://127.0.0.1:8000/submit'
r = requests.post(url, data=body.encode('utf-8'), headers=headers)
print(r.content)
In websockets, you can use send_bytes() and receive_bytes() for the communication, as described in Starlette's documentation, allowing you to send and receive (byte encoded) XML data as well. If you would like to perform validation on the received XML data, have a look at this answer.
Using WebSocket Protocol
app.py
from fastapi import FastAPI, Request, WebSocket, WebSocketDisconnect
import uvicorn
app = FastAPI()
#app.websocket("/ws")
async def get_stream(websocket: WebSocket):
await websocket.accept()
try:
while True:
contents = await websocket.receive_bytes()
print(str(contents, 'utf-8'))
except WebSocketDisconnect:
print("Client disconnected")
if __name__ == '__main__':
uvicorn.run(app, host='127.0.0.1', port=8000)
test.py
import websockets
import asyncio
async def main():
url = 'ws://127.0.0.1:8000/ws'
async with websockets.connect(url) as ws:
while True:
b = bytes("<?xml version='1.0' encoding='utf-8'?><a>б</a>", 'utf-8')
await ws.send(b)
await asyncio.sleep(1)
asyncio.run(main())
To start off, I am writing an async wrapper for the WordPress REST API. I have a Wordpress site hosted on Bluehost. I am working with the endpoint for media (image) uploads. I have successfully managed to upload an image but there are 2 changes I would like to make. The second change is what I really want, but out of curiosity, I would like to know how to implement change 1 too. I'll provide the code first and then some details.
Working code
async def upload_local_pic2(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title': title, 'status':'publish'}
mpwriter.append_json(json)
with open(local_url, 'rb') as f:
print(f)
payload = mpwriter.append(f)
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
Change 1
The first change is uploading using aiofiles.open() instead of just using open() as I expect to be processing lots of files. The following code does not work.
async def upload_local_pic(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title': title, 'status':'publish'}
mpwriter.append_json(json)
async with aiofiles.open(local_url, 'rb') as f:
print(f)
payload = mpwriter.append(f)
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
Change 2
My other change is that I would like to have another function that can upload the files directly to the WordPress server without downloading them locally. So instead of getting a local picture, I want to pass in the url of an image online. The following code also does not work.
async def upload_pic(self, image_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'title':title, 'status':'publish'}
mpwriter.append_json(json)
async with self.session.get(image_url) as image_response:
image_content = image_response.content
print(image_content)
payload = mpwriter.append(image_content)
async with self.session.post(url, data = payload) as response:
x = await response.read()
print(x)
Details/Debugging
I'm trying to figure out why each one won't work. I think the key is the calls to print(image_content)
and print(f) that show what exactly I am inputting to mpwriter.append
In the example that works where I just use the standard Python open() function, I am apparently passing in <_io.BufferedReader name='/redactedfilepath/index.jpeg'>
In the change 1 example with aiofile, I am passing in <aiofiles.threadpool.binary.AsyncBufferedReader object at 0x7fb803122250>
Wordpress will return this html:
b'<head><title>Not Acceptable!</title></head><body><h1>Not Acceptable!</h1><p>An appropriate representation of the requested resource could not be found on this server. This error was generated by Mod_Security.</p></body></html>'
And finally, in change 2 where I try to pass in what the get request to the url gives me I get
<StreamReader 292 bytes>. The response returned by WordPress is the same as above with Mod Security.
Any idea how I can make these examples work? It seems like they are all some type of io reader but I guess the underlying aiohttp code treats them differently.
Also this shouldn't really matter, but this is the url I am passing into the change 2 example.
Ok, so I figured out both changes.
For the first change when trying to read a file with aiofiles, I need to just read the whole file instead of passing in the file handler. Also, I need to set the content disposition manually.
async def upload_local_pic(self, local_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'status':'publish'}
mpwriter.append_json(json)
async with aiofiles.open(local_url, mode='rb') as f:
contents = await f.read()
payload = mpwriter.append(contents)
payload.set_content_disposition('attachment', filename= title+'.jpg')
async with self.session.post(url, data=payload) as response:
x = await response.read()
print(x)
For the second change, it's a similar concept with just uploading a file directly from the URL. Instead of passing in the handler that will read the content, I need to read the entire content first. I also need to set the content-disposition manually.
async def upload_pic(self, image_url, date, title):
url = f'{self.base_url}/wp-json/wp/v2/media'
with aiohttp.MultipartWriter() as mpwriter:
json = {'status':'publish'}
mpwriter.append_json(json)
async with self.session.get(image_url) as image_response:
image_content = await image_response.read()
payload = mpwriter.append(image_content)
payload.set_content_disposition('attachment', filename=title+'.jpg')
async with self.session.post(url, data = payload) as response:
x = await response.read()
print(x)
I will answer only to the title of the post (and not the questions that are in between).
The following code should give a short example of how to upload a file from URL#1 to URL#2 (without the need to download the file to the local machine and only then do the upload).
I will give two examples here:
Read all the content of the file into the memory (not downloading). This is of-course not so good when working with huge files...
Read and send the file in chunks (so we won't read all the file content at once).
Example #1: Reading all file content AT ONCE and uploading
import asyncio
import aiohttp
async def http_upload_from_url(src, dst):
async with aiohttp.ClientSession() as session:
src_resp = await session.get(src)
#print(src_resp)
dst_resp = await session.post(dst, data=src_resp.content)
#print(dst_resp)
try:
asyncio.run(http_upload_from_url(SRC_URL, DST_URL))
except Exception as e:
print(e)
Example #2: Reading file content IN CHUNKS and uploading
import asyncio
import aiohttp
async def url_sender(url=None, chunk_size=65536):
async with aiohttp.ClientSession() as session:
resp = await session.get(url)
#print(resp)
async for chunk in resp.content.iter_chunked(chunk_size):
#print(f"send chunk with size {len(chunk)}")
yield chunk
async def chunked_http_upload_from_url(src, dst):
async with aiohttp.ClientSession() as session:
resp = await session.post(dst, data=url_sender(src))
#print(resp)
#print(await resp.text())
try:
asyncio.run(chunked_http_upload_from_url(SRC_URL, DST_URL))
except Exception as e:
print(e)
Some notes:
You need to define SRC_URL and DST_URL.
I've only added the prints for debug (in-case you don't get a [200 OK] response).
I'm working on a kind of personal cloud project which allows to upload and download files (with a google like search feature). The backend is written in python (using aiohttp) while I'm using a react.js website as a client. When a file is uploaded, the backend stores it on the filesystem and renames it with its sha256 hash. The original name and a few other metadata are stored next to it (description, etc). When the user downloads the file, I'm serving it using multipart and I want the user to get it with the original name and not the hash, indeed my-cool-image.png is more user friendly than a14e0414-b84c-4d7b-b0d4-49619b9edd8a. But I'm not able to do it (whatever I try, the download file is called with the hash).
Here is my code:
async def download(self, request):
if not request.username:
raise exceptions.Unauthorized("A valid token is needed")
data = await request.post()
hash = data["hash"]
file_path = storage.get_file(hash)
dotfile_path = storage.get_file("." + hash)
if not os.path.exists(file_path) or not os.path.exists(dotfile_path):
raise exceptions.NotFound("file <{}> does not exist".format(hash))
with open(dotfile_path) as dotfile:
dotfile_content = json.load(dotfile)
name = dotfile_content["name"]
headers = {
"Content-Type": "application/octet-stream; charset=binary",
"Content-Disposition": "attachment; filename*=UTF-8''{}".format(
urllib.parse.quote(name, safe="")
),
}
return web.Response(body=self._file_sender(file_path), headers=headers)
Here is what it looks like (according to the browser):
Seems right, yet it's not working.
One thing I want to make clear though: sometimes I get a warning (on the client side) saying Resource interpreted as Document but transferred with MIME type application/octet-stream. I don't know the MIME type of the files since they are provided by the users, but I tried to use image/png (I did my test with a png image stored on the server). The file didn't get downloaded (it was displayed in the browser, which is not something that I want) and the file name was still its hash so it didn't help with my issue.
Here is the entire source codes of the backend:
https://git.io/nexmind-node
And of the frontend:
https://git.io/nexmind-client
EDIT:
I received a first answer by Julien Castiaux so I tried to implement it, even though it looks better, it doesn't solve my issue (I still have the exact same behaviour):
async def download(self, request):
if not request.username:
raise exceptions.Unauthorized("A valid token is needed")
data = await request.post()
hash = data["hash"]
file_path = storage.get_file(hash)
dotfile_path = storage.get_file("." + hash)
if not os.path.exists(file_path) or not os.path.exists(dotfile_path):
raise exceptions.NotFound("file <{}> does not exist".format(hash))
with open(dotfile_path) as dotfile:
dotfile_content = json.load(dotfile)
name = dotfile_content["name"]
response = web.StreamResponse()
response.headers['Content-Type'] = 'application/octet-stream'
response.headers['Content-Disposition'] = "attachment; filename*=UTF-8''{}".format(
urllib.parse.quote(name, safe="") # replace with the filename
)
response.enable_chunked_encoding()
await response.prepare(request)
with open(file_path, 'rb') as fd: # replace with the path
for chunk in iter(lambda: fd.read(1024), b""):
await response.write(chunk)
await response.write_eof()
return response
From aiohttp3 documentation
StreamResponse is intended for streaming data, while Response contains HTTP BODY as an attribute and sends own content as single piece with the correct Content-Length HTTP header.
You rather use aiohttp.web.StreamResponse as you are sending (potentially very large) files. Using StreamResponse, you have full control of the outgoing http response stream : headers manipulation (including the filename) and chunked encoding.
from aiohttp import web
import urllib.parse
async def download(req):
resp = web.StreamResponse()
resp.headers['Content-Type'] = 'application/octet-stream'
resp.headers['Content-Disposition'] = "attachment; filename*=UTF-8''{}".format(
urllib.parse.quote(filename, safe="") # replace with the filename
)
resp.enable_chunked_encoding()
await resp.prepare(req)
with open(path_to_the_file, 'rb') as fd: # replace with the path
for chunk in iter(lambda: fd.read(1024), b""):
await resp.write(chunk)
await resp.write_eof()
return resp
app = web.Application()
app.add_routes([web.get('/', download)])
web.run_app(app)
Hope it helps !
I'm using FastAPI and currently I return a csv which I read from SQL server with pandas. (pd.read_sql())
However the csv is quite big for the browser and I want to return it with a File response:
https://fastapi.tiangolo.com/advanced/custom-response/ (end of the page).
I cannot seem to do this without first writing it to a csv file which seems slow and will clutter the filesystem with csv's on every request.
So my questions way, is there way to return a FileResponse from a sql database or pandas dataframe.
And if not, is there a way to delete the generated csv files, after it has all been read by the client?
Thanks for your help!
Kind regards,
Stephan
Based HEAVILY off this https://github.com/tiangolo/fastapi/issues/1277
Turn your dataframe into a stream
use a streaming response
Modify headers so it's a download (optional)
from fastapi.responses import StreamingResponse
import io
#app.get("/get_csv")
async def get_csv():
df = pandas.DataFrame(dict(col1 = 1, col2 = 2))
stream = io.StringIO()
df.to_csv(stream, index = False)
response = StreamingResponse(iter([stream.getvalue()]),
media_type="text/csv"
)
response.headers["Content-Disposition"] = "attachment; filename=export.csv"
return response
Adding to the code that was previously mentioned, I found it useful to place another response header, in order for the client to be able to see the "Content-Disposition". This is due to the fact, that only CORS-safelisted response headers can be seen by default by the client. "Content-Disposition" is not part of this list, so it must be added explicitly https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Expose-Headers.
I don't know if there is another way to specify this, for the client or server in a more general way so that it applies to all the necessary endpoints, but this is the way I applied it.
#router.post("/files", response_class = StreamingResponse)
async def anonymization(file: bytes = File(...), config: str = Form(...)):
# file as str
inputFileAsStr = StringIO(str(file,'utf-8'))
# dataframe
df = pd.read_csv(inputFileAsStr)
# send to function to handle anonymization
results_df = anonymize(df, config)
# output file
outFileAsStr = StringIO()
results_df.to_csv(outFileAsStr, index = False)
response = StreamingResponse(
iter([outFileAsStr.getvalue()]),
media_type='text/csv',
headers={
'Content-Disposition': 'attachment;filename=dataset.csv',
'Access-Control-Expose-Headers': 'Content-Disposition'
}
)
# return
return response
I was beating my head against the wall on this one as well. My use case is slightly different as I am storing images, pdfs, etc. as blobs in my maria database. I found that the trick was to pass the blob contents to BytesIO and the rest was simple.
from fastapi.responses import StreamingResponse
from io import BytesIO
#router.get('/attachment/{id}')
async def get_attachment(id: int):
mdb = messages(s.MARIADB)
attachment = mdb.getAttachment(id)
memfile = BytesIO(attachment['content'])
response = StreamingResponse(memfile, media_type=attachment['contentType'])
response.headers["Content-Disposition"] = f"inline; filename={attachment['name']}"
return response