We Keep Coding

Http Post data from Python to net core app

I am trying to send a file from a Python script to my .net core webserver.
In Python I am doing this using the requests library, and my code looks like so.
filePath = "run-1.csv"
with open(filePath, "rb") as postFile:
file_dict = {filePath: postFile}
response = requests.post(server_path + "/batchUpload/create", files=file_dict, verify=validate_sql)
print (response.text)
This code executes fine, and I can see the request fine in my webserver code which looks like so:
[HttpPost]
[Microsoft.AspNetCore.Authorization.AllowAnonymous]
public string Create(IFormFile file) //Dictionary<string, IFormFile>
{
var ms = new MemoryStream();
file.CopyTo(ms);
var text = Encoding.ASCII.GetString(ms.ToArray());
Debug.Print(text);
return "s";
}
However, the file parameter always returns as null.
Also, I can see the file parameter fine when getting data posted from postMan
I suspect that this problem has to do with how .net core model binding works, but not sure...
Any suggestions here on how to get my file displaying on the server?

Solved my issue - the problem was that in Python I was assigning my file to my upload dictionary with the actual file name "./run1.csv" rather than a literal string "file"
Updating this fixed my issue.
file_dict = {"file": postFile}
This is what I believe #nalnpir mentioned above.
I figured this out by posting from postman and also from my python code to http://httpbin.org/post and comparing the respoinse

The example from the requests docs is mostly correct, except that the key has to match the parameter of the controller method signature.
url = 'https://www.url.com/api/post'
files = {'parameterName': open('filename.extension', 'rb')}
r = requests.post(url, files=files)
So in this case the controller action should be
[HttpPost]
public string Post(IFormFile parameterName)

how do i use FileResponse.set_headers() in my django application to enable .mp3 file download

I am trying to use FileResponse.set_header() to set Content-Disposition to attachment, so that Audio file can be downloaded rather than playing on the browser, in my python/django powered website.
is there any way i can implement this code bellow to make it work?
song = models.song.objects.filter(id__exact=song_id).first()
file_name = ntpath.basename(song.songfile.url)
content_type = 'audio/mpeg'
with open(identify_song.songfile.path, 'rb') as my_f:
file_like = my_f
response = FileResponse(my_f, content_type=content_type, as_attachment=True, filename="{}".format(file_name))
response['Content-Disposition'] = 'attachment; filename="{}"'.format(file_name)
size = response['Content-Length'] = os.path.getsize(identify_song.songfile.path)
#print(file_name)
return(response)
this code does not give me any error but it's not working.
So i discovered about FileResponse.set_header() in the django doc, so i tried using it like so.
`song = models.song.objects.filter(id__exact=song_id).first()
file_name = ntpath.basename(song.songfile.url)
FileResponse.set_headers(file_name, filelike='audio/mpeg', as_attachment=True)`
Then i got an error AttributeError: 'str' object has no attribute 'filename'.
please can anyone help me out, or if there is another way to do that in django i will so much appreciate someone's help. Or any other possible way i can set my Content-Disposition, in django, Nginx or in Javascript.

I have been working all day with this functionality in order to download a generated file, lemme share you how I did it, it worked like a charm.
documentation : https://docs.djangoproject.com/en/3.0/ref/request-response/
http header 'Content-Disposition': https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Disposition
try:
wdata = request.GET
download = wdata.get("download") or ''
allowed_params = [
'template', 'bases',
]
if download in allowed_params:
out, err = utldol.download_file(download, request)
if err:
raise ValueError(str(err))
else:
raise ValueError(str('Parameter not recognized'))
file = FileResponse(
out.get("file"), filename=out.get("filename"),
)
file['Content-Disposition'] = 'attachment; filename="{}"'.format(
out.get("filename")
)
return file
except Exception as ex:
return HttpResponseBadRequest(str(ex))
The parameter "file" contains the file instance : open('file.txt','rb') :
out.get("file")
The parameter "filename" contains the name of the file
out.get("filename")
finally, when the file is throw by the browser :
Hope my experience would be helpful, any views, please just lemme know.
greetings,

There is no answer to this question as the method 'FileResponse.set_header()' can be called with a file like object only and it makes some guesses about the header which are not reliable if you use file types that are rather exotic. At least you can overwrite this function that returns nothing but only sets the header information or you can do this little task in your code yourself. Here is my example using an in-memory string buffer.
filename = "sample.creole"
str_stream = StringIO()
str_stream.write("= Some Creole-formated text\nThis is creole content.")
str_stream.seek(0)
response = FileResponse(str_stream.read())
response.headers['Content-Disposition'] = 'attachment; filename={}'.format(filename)
response.headers['Content-Type'] = 'application/creole'
return response
With that approach yo can still use the default 'FileResponse.set_header()' functionality to set the 'Content-Length' which works properly. As the other 'FileResponse' parameters such as 'as_attachment' and 'filename' don't work as reliable. Probably nobody noticed yet as 'FileResponse' is rarely used, and as a matter of fact the same functionality can be implemented using 'HttpResponse'.
response = HttpResponse(content_type='application/creole')
response.headers['Content-Disposition'] = 'attachment; filename={}'.format(filename)
response.write(str_stream.read())
return response

The utldol it is a python file that contains a function, that generates a xls file. The function returns smth like these :
outcome, error = None, None
try:
........
........
file = open(output.get("filedir"), 'rb')
outcome = {
"file": file, "filename": output.get("filename"),
}
except Exception as ex:
error = str(ex)
return [outcome, error]
output contains the fullpath of the xls file generated then it is read by "open" function.

Python requests_toolbelt MultipartEncoder filename

Using requests_toolbelt to upload large files in a Multipart form, I have constructed a method below which succeeds in uploading the file, however I cannot access the posted filename. How do I access the filename on the server?
# client-side
file = open('/Volumes/Extra/test/my_video.mpg', 'rb')
payload = MultipartEncoder({file.name: file})
r = requests.post(url, data=payload, headers={'Content-Type': 'application/octet-stream'})
# server-side
#view_config(route_name='remote.agent_upload', renderer='json')
def remote_agent_upload(request):
r = request.response
fs = request.body_file
f = open('/Volumes/Extra/tests2/bar.mpg', 'wb') # wish to use filename here
f.write(fs.read())
fs.close()
f.close()
return r

OK, it looks like you are using the name of the file as the field name. Also, the way that you are doing it, seems like the entire post content is being written to file... Is this the desired outcome? Have you tried to actually play your mpg files after you write them on the server side?
I don't have an HTTP server readily available to test at the moment which automagically gives me a request object, but I am assuming that the request object is a webob.Request object (at least it seems like that is the case, please correct me if I'm wrong)
OK, let me show you my test. (This works on python3.4, not sure what version of Python you are using, but I think it should also work on Python 2.7 - not tested though)
The code in this test is a bit long, but it is heavily commented to help you understand what I did every step of the way. Hopefully, it will give you a better understanding of how HTTP requests and responses work in python with the tools you are using
# My Imports
from requests_toolbelt import MultipartEncoder
from webob import Request
import io
# Create a buffer object that can be read by the MultipartEncoder class
# This works just like an open file object
file = io.BytesIO()
# The file content will be simple for my test.
# But you could just as easily have a multi-megabyte mpg file
# Write the contents to the file
file.write(b'test mpg content')
# Then seek to the beginning of the file so that the
# MultipartEncoder can read it from the beginning
file.seek(0)
# Create the payload
payload = MultipartEncoder(
{
# The name of the file upload field... Not the file name
'uploadedFile': (
# This would be the name of the file
'This is my file.mpg',
# The file handle that is ready to be read from
file,
# The content type of the file
'application/octet-stream'
)
}
)
# To send the file, you would use the requests.post method
# But the content type is not application-octet-stream
# The content type is multipart/form-data; with a boundary string
# Without the proper header type, your server would not be able to
# figure out where the file begins and ends and would think the
# entire post content is the file, which it is not. The post content
# might even contain multiple files
# So, to send your file, you would use:
#
# response = requests.post(url, data=payload, headers={'Content-Type': payload.content_type})
# Instead of sending the payload to the server,
# I am just going to grab the output as it would be sent
# This is because I don't have a server, but I can easily
# re-create the object using this output
postData = payload.to_string()
# Create an input buffer object
# This will be read by our server (our webob.Request object)
inputBuffer = io.BytesIO()
# Write the post data to the input buffer so that the webob.Request object can read it
inputBuffer.write(postData)
# And, once again, seek to 0
inputBuffer.seek(0)
# Create an error buffer so that errors can be written to it if there are any
errorBuffer = io.BytesIO()
# Setup our wsgi environment just like the server would give us
environment = {
'HTTP_HOST': 'localhost:80',
'PATH_INFO': '/index.py',
'QUERY_STRING': '',
'REQUEST_METHOD': 'POST',
'SCRIPT_NAME': '',
'SERVER_NAME': 'localhost',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.0',
'CONTENT_TYPE': payload.content_type,
'wsgi.errors': errorBuffer,
'wsgi.input': inputBuffer,
'wsgi.multiprocess': False,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)
}
# Create our request object
# This is the same as your request object and should have all our info for reading
# the file content as well as the file name
request = Request(environment)
# At this point, the request object is the same as what you get on your server
# So, from this point on, you can use the following code to get
# your actual file content as well as your file name from the object
# Our uploaded file is in the POST. And the POST field name is 'uploadedFile'
# Grab our file so that it can be read
uploadedFile = request.POST['uploadedFile']
# To read our content, you can use uploadedFile.file.read()
print(uploadedFile.file.read())
# And to get the file name, you can use uploadedFile.filename
print(uploadedFile.filename)
So, I think this modified code will work for you. (Hopefully)
Again, not tested because I don't actually have a server to test with. And also, I don't know what kind of object your "request" object is on the server side.... OK, here goes:
# client-side
import requests
file = open('/Volumes/Extra/test/my_video.mpg', 'rb')
payload = MultipartEncoder({'uploadedFile': (file.name, file, 'application/octet-stream')})
r = requests.post('http://somewhere/somefile.py', data=payload, headers={'Content-Type': payload.content_type})
# server-side
#view_config(route_name='remote.agent_upload', renderer='json')
def remote_agent_upload(request):
# Write your actual file contents, not the post data which contains multi part boundary
uploadedFile = request.POST['uploadedFile']
fs = uploadedFile.file
# The file name is insecure. What if the file name comes through as '../../../etc/passwd'
# If you don't secure this, you've just wiped your /etc/passwd file and your server is toast
# (assuming the web user has write permission to the /etc/passwd file
# which it shouldn't, but just giving you a worst case scenario)
fileName = uploadedFile.filename
# Secure the fileName here...
# Make sure it doesn't have any slashes or double dots, or illegal characters, etc.
# I'll leave that up to you
# Write the file
f = open('/Volumes/Extra/tests2/' + fileName, 'wb')
f.write(fs.read())

Probably too late for the original OP but may help someone else. This is how I upload a file with accompanying json in a multipart/form-data upload using the MultipartEncoder. When I require a file to uploaded as binary with a single json string as part of a multipart request (so there are just two parts, the file and the json). Note that in creating my request header (it's a custom header as designated by the receiving server) I get the content_type from the encoded object (it usually comes through as multipart/form-data). I'm using simplejson.dumps but you can just use json.dumps I believe.
m = MultipartEncoder([
('json', (None, simplejson.dumps(datapayload), 'text/plain')),
('file', (os.path.basename(file_path), open(file_path, 'rb'), 'text/plain'))],
None, encoding='utf-8')
headers = {'Authorization': 'JwToken' + ' ' + jwt_str, 'content-type': m.content_type}
response = requests.post(uri, headers=headers, data=m, timeout=45, verify=True )
In the file part, the field is called "file", but I use os.path.basename(file_path) to get just the filename from a full file path e.g. c:\temp\mytestfile.txt . It's possible I could just as easily call the file something else (that's not the original name) in this field if I wanted to.

How do I handle file upload via PUT request in Django?

I'm implementing a REST-style interface and would like to be able to create (via upload) files via a HTTP PUT request. I would like to create either a TemporaryUploadedFile or a InMemoryUploadedFile which I can then pass to my existing FileField and .save() on the object that is part of the model, thereby storing the file.
I'm not quite sure about how to handle the file upload part. Specifically, this being a put request, I do not have access to request.FILES since it does not exist in a PUT request.
So, some questions:
Can I leverage existing functionality in the HttpRequest class, specifically the part that handles file uploads? I know a direct PUT is not a multipart MIME request, so I don't think so, but it is worth asking.
How can I deduce the mime type of what is being sent? If I've got it right, a PUT body is simply the file without prelude. Do I therefore require that the user specify the mime type in their headers?
How do I extend this to large amounts of data? I don't want to read it all into memory since that is highly inefficient. Ideally I'd do what TemporaryUploadFile and related code does - write it part at a time?
I've taken a look at this code sample which tricks Django into handling PUT as a POST request. If I've got it right though, it'll only handle form encoded data. This is REST, so the best solution would be to not assume form encoded data will exist. However, I'm happy to hear appropriate advice on using mime (not multipart) somehow (but the upload should only contain a single file).
Django 1.3 is acceptable. So I can either do something with request.raw_post_data or request.read() (or alternatively some other better method of access). Any ideas?

Django 1.3 is acceptable. So I can
either do something with
request.raw_post_data or
request.read() (or alternatively some
other better method of access). Any
ideas?
You don't want to be touching request.raw_post_data - that implies reading the entire request body into memory, which if you're talking about file uploads might be a very large amount, so request.read() is the way to go. You can do this with Django <= 1.2 as well, but it means digging around in HttpRequest to figure out the the right way to use the private interfaces, and it's a real drag to then ensure your code will also be compatible with Django >= 1.3.
I'd suggest that what you want to do is to replicate the existing file upload behaviour parts of the MultiPartParser class:
Retrieve the upload handers from request.upload_handlers (Which by default will be MemoryFileUploadHandler & TemporaryFileUploadHandler)
Determine the request's content length (Search of Content-Length in HttpRequest or MultiPartParser to see the right way to do this.)
Determine the uploaded file's filename, either by letting the client specify this using the last path part of the url, or by letting the client specify it in the "filename=" part of the Content-Disposition header.
For each handler, call handler.new_file with the relevant args (mocking up a field name)
Read the request body in chunks using request.read() and calling handler.receive_data_chunk() for each chunk.
For each handler call handler.file_complete(), and if it returns a value, that's the uploaded file.
How can I deduce the mime type of what
is being sent? If I've got it right, a
PUT body is simply the file without
prelude. Do I therefore require that
the user specify the mime type in
their headers?
Either let the client specify it in the Content-Type header, or use python's mimetype module to guess the media type.
I'd be interested to find out how you get on with this - it's something I've been meaning to look into myself, be great if you could comment to let me know how it goes!
Edit by Ninefingers as requested, this is what I did and is based entirely on the above and the django source.
upload_handlers = request.upload_handlers
content_type = str(request.META.get('CONTENT_TYPE', ""))
content_length = int(request.META.get('CONTENT_LENGTH', 0))
if content_type == "":
return HttpResponse(status=400)
if content_length == 0:
# both returned 0
return HttpResponse(status=400)
content_type = content_type.split(";")[0].strip()
try:
charset = content_type.split(";")[1].strip()
except IndexError:
charset = ""
# we can get the file name via the path, we don't actually
file_name = path.split("/")[-1:][0]
field_name = file_name
Since I'm defining the API here, cross browser support isn't a concern. As far as my protocol is concerned, not supplying the correct information is a broken request. I'm in two minds as to whether I want say image/jpeg; charset=binary or if I'm going to allow non-existent charsets. In any case, I'm putting setting Content-Type validly as a client-side responsibility.
Similarly, for my protocol, the file name is passed in. I'm not sure what the field_name parameter is for and the source didn't give many clues.
What happens below is actually much simpler than it looks. You ask each handler if it will handle the raw input. As the author of the above states, you've got MemoryFileUploadHandler & TemporaryFileUploadHandler by default. Well, it turns out MemoryFileUploadHandler will when asked to create a new_file decide whether it will or not handle the file (based on various settings). If it decides it's going to, it throws an exception, otherwise it won't create the file and lets another handler take over.
I'm not sure what the purpose of counters was, but I've kept it from the source. The rest should be straightforward.
counters = [0]*len(upload_handlers)
for handler in upload_handlers:
result = handler.handle_raw_input("",request.META,content_length,"","")
for handler in upload_handlers:
try:
handler.new_file(field_name, file_name,
content_type, content_length, charset)
except StopFutureHandlers:
break
for i, handler in enumerate(upload_handlers):
while True:
chunk = request.read(handler.chunk_size)
if chunk:
handler.receive_data_chunk(chunk, counters[i])
counters[i] += len(chunk)
else:
# no chunk
break
for i, handler in enumerate(upload_handlers):
file_obj = handler.file_complete(counters[i])
if not file_obj:
# some indication this didn't work?
return HttpResponse(status=500)
else:
# handle file obj!

Newer Django versions allow for handling this a lot easier thanks to https://gist.github.com/g00fy-/1161423
I modified the given solution like this:
if request.content_type.startswith('multipart'):
put, files = request.parse_file_upload(request.META, request)
request.FILES.update(files)
request.PUT = put.dict()
else:
request.PUT = QueryDict(request.body).dict()
to be able to access files and other data like in POST. You can remove the calls to .dict() if you want your data to be read-only.

I hit this problem while working with Django 2.2, and was looking for something that just worked for uploading a file via PUT request.
from django.http import QueryDict
from django.http.multipartparser import MultiValueDict
from django.core.files.uploadhandler import (
SkipFile,
StopFutureHandlers,
StopUpload,
)
class PutUploadMiddleware(object):
def __init__(self, get_response):
self.get_response = get_response
def __call__(self, request):
method = request.META.get("REQUEST_METHOD", "").upper()
if method == "PUT":
self.handle_PUT(request)
return self.get_response(request)
def handle_PUT(self, request):
content_type = str(request.META.get("CONTENT_TYPE", ""))
content_length = int(request.META.get("CONTENT_LENGTH", 0))
file_name = request.path.split("/")[-1:][0]
field_name = file_name
content_type_extra = None
if content_type == "":
return HttpResponse(status=400)
if content_length == 0:
# both returned 0
return HttpResponse(status=400)
content_type = content_type.split(";")[0].strip()
try:
charset = content_type.split(";")[1].strip()
except IndexError:
charset = ""
upload_handlers = request.upload_handlers
for handler in upload_handlers:
result = handler.handle_raw_input(
request.body,
request.META,
content_length,
boundary=None,
encoding=None,
)
counters = [0] * len(upload_handlers)
for handler in upload_handlers:
try:
handler.new_file(
field_name,
file_name,
content_type,
content_length,
charset,
content_type_extra,
)
except StopFutureHandlers:
break
for chunk in request:
for i, handler in enumerate(upload_handlers):
chunk_length = len(chunk)
chunk = handler.receive_data_chunk(chunk, counters[i])
counters[i] += chunk_length
if chunk is None:
# Don't continue if the chunk received by
# the handler is None.
break
for i, handler in enumerate(upload_handlers):
file_obj = handler.file_complete(counters[i])
if file_obj:
# If it returns a file object, then set the files dict.
request.FILES.appendlist(file_name, file_obj)
break
any(handler.upload_complete() for handler in upload_handlers)

Am I parsing this HTTP POST request properly?

Let me start off by saying, I'm using the twisted.web framework. Twisted.web's file uploading didn't work like I wanted it to (it only included the file data, and not any other information), cgi.parse_multipart doesn't work like I want it to (same thing, twisted.web uses this function), cgi.FieldStorage didn't work ('cause I'm getting the POST data through twisted, not a CGI interface -- so far as I can tell, FieldStorage tries to get the request via stdin), and twisted.web2 didn't work for me because the use of Deferred confused and infuriated me (too complicated for what I want).
That being said, I decided to try and just parse the HTTP request myself.
Using Chrome, the HTTP request is formed like this:
------WebKitFormBoundary7fouZ8mEjlCe92pq
Content-Disposition: form-data; name="upload_file_nonce"
11b03b61-9252-11df-a357-00266c608adb
------WebKitFormBoundary7fouZ8mEjlCe92pq
Content-Disposition: form-data; name="file"; filename="login.html"
Content-Type: text/html
<!DOCTYPE html>
<html>
<head>
...
------WebKitFormBoundary7fouZ8mEjlCe92pq
Content-Disposition: form-data; name="file"; filename=""
------WebKitFormBoundary7fouZ8mEjlCe92pq--
Is this always how it will be formed? I'm parsing it with regular expressions, like so (pardon the wall of code):
(note, I snipped out most of the code to show only what I thought was relevant (the regular expressions (yeah, nested parentheses), this is an __init__ method (the only method so far) in an Uploads class I built. The full code can be seen in the revision history (I hope I didn't mismatch any parentheses)
if line == "--{0}--".format(boundary):
finished = True
if in_header == True and not line:
in_header = False
if 'type' not in current_file:
ignore_current_file = True
if in_header == True:
m = re.match(
"Content-Disposition: form-data; name=\"(.*?)\"; filename=\"(.*?)\"$", line)
if m:
input_name, current_file['filename'] = m.group(1), m.group(2)
m = re.match("Content-Type: (.*)$", line)
if m:
current_file['type'] = m.group(1)
else:
if 'data' not in current_file:
current_file['data'] = line
else:
current_file['data'] += line
you can see that I start a new "file" dict whenever a boundary is reached. I set in_header to True to say that I'm parsing headers. When I reach a blank line, I switch it to False -- but not before checking if a Content-Type was set for that form value -- if not, I set ignore_current_file since I'm only looking for file uploads.
I know I should be using a library, but I'm sick to death of reading documentation, trying to get different solutions to work in my project, and still having the code look reasonable. I just want to get past this part -- and if parsing an HTTP POST with file uploads is this simple, then I shall stick with that.
Note: this code works perfectly for now, I'm just wondering if it will choke on/spit out requests from certain browsers.

My solution to this Problem was parsing the content with cgi.FieldStorage like:
class Root(Resource):
def render_POST(self, request):
self.headers = request.getAllHeaders()
# For the parsing part look at [PyMOTW by Doug Hellmann][1]
img = cgi.FieldStorage(
fp = request.content,
headers = self.headers,
environ = {'REQUEST_METHOD':'POST',
'CONTENT_TYPE': self.headers['content-type'],
}
)
print img["upl_file"].name, img["upl_file"].filename,
print img["upl_file"].type, img["upl_file"].type
out = open(img["upl_file"].filename, 'wb')
out.write(img["upl_file"].value)
out.close()
request.redirect('/tests')
return ''

You're trying to avoid reading documentation, but I think the best advice is to actually read:
rfc 2388 Returning Values from Forms: multipart/form-data
rfc 1867 Form-based File Upload in HTML
to make sure you don't miss any cases. An easier route might be to use the poster library.

The content-disposition header has no defined order for fields, plus it may contain more fields than just the filename. So your match for filename may fail - there may not even be a filename!
See rfc2183 (edit that's for mail, see rfc1806, rfc2616 and maybe more for http)
Also I would suggest in these kind of regexps to replace every space by \s*, and not to rely on character case.