there is my code:
client:
def upload(url,file):
file_name=file.split('\\')[-1]
with open(file,'rb') as f:
r=requests.post(url,data=f)
server side:
$datastr = fopen('php://input',"rb");
$filename='upload/111.bin';
if ($fp = fopen($filename,"wb")){
while(!feof($datastr)){
fwrite($fp,fread($datastr,4096)) ;
}
}
it can work, but I don't know how to post the file stream with file name.
if I use r=requests.post(url,data={ 'name':file_name, 'file':open(file,'rb') }), $_POST is null
According to the documentation you can POST a multipart file with requests, but to POST it as a stream you should use requests-toolbelt.
If no streaming is actually required, you could modify your code to be:
def upload(url,file):
file_name=file.split('\\')[-1]
with open(file,'rb') as f:
r=requests.post(url,files={"file": (file_name, f)})
Using files in this way let’s you include file name and other header info (see docs for more information)
Related
I am trying to send a file from a Python script to my .net core webserver.
In Python I am doing this using the requests library, and my code looks like so.
filePath = "run-1.csv"
with open(filePath, "rb") as postFile:
file_dict = {filePath: postFile}
response = requests.post(server_path + "/batchUpload/create", files=file_dict, verify=validate_sql)
print (response.text)
This code executes fine, and I can see the request fine in my webserver code which looks like so:
[HttpPost]
[Microsoft.AspNetCore.Authorization.AllowAnonymous]
public string Create(IFormFile file) //Dictionary<string, IFormFile>
{
var ms = new MemoryStream();
file.CopyTo(ms);
var text = Encoding.ASCII.GetString(ms.ToArray());
Debug.Print(text);
return "s";
}
However, the file parameter always returns as null.
Also, I can see the file parameter fine when getting data posted from postMan
I suspect that this problem has to do with how .net core model binding works, but not sure...
Any suggestions here on how to get my file displaying on the server?
Solved my issue - the problem was that in Python I was assigning my file to my upload dictionary with the actual file name "./run1.csv" rather than a literal string "file"
Updating this fixed my issue.
file_dict = {"file": postFile}
This is what I believe #nalnpir mentioned above.
I figured this out by posting from postman and also from my python code to http://httpbin.org/post and comparing the respoinse
The example from the requests docs is mostly correct, except that the key has to match the parameter of the controller method signature.
url = 'https://www.url.com/api/post'
files = {'parameterName': open('filename.extension', 'rb')}
r = requests.post(url, files=files)
So in this case the controller action should be
[HttpPost]
public string Post(IFormFile parameterName)
I want to get the content of an uploaded file on S3 using botocore and aiohttp service. As the files may have a huge size:
I don't want to store the whole file content in memory,
I want to be able to handle other requests while downloading files from S3 (aiobotocore, aiohttp),
I want to be able to apply modifications on the files I download, so I want to treat it line by line and stream the response to the client
For now, I have the following code in my aiohttp handler:
import asyncio
import aiobotocore
from aiohttp import web
#asyncio.coroutine
def handle_get_file(loop):
session = aiobotocore.get_session(loop=loop)
client = session.create_client(
service_name="s3",
region_name="",
aws_secret_access_key="",
aws_access_key_id="",
endpoint_url="http://s3:5000"
)
response = yield from client.get_object(
Bucket="mybucket",
Key="key",
)
Each time I read one line from the given file, I want to send the response. Actually, get_object() returns a dict with a Body (ClientResponseContentProxy object) inside. Using the method read(), how can I get a chunk of the expected response and stream it to the client ?
When I do :
for content in response['Body'].read(10):
print("----")
print(content)
The code inside the loop is never executed.
But when I do :
result = yield from response['Body'].read(10)
I get the content of the file in result. I am a little bit confused about how to use read() here.
Thanks
it's because the aiobotocore api is different than the one of botocore , here read() returns a FlowControlStreamReader.read generator for which you need to yield from
it looks something like that (taken from https://github.com/aio-libs/aiobotocore/pull/19)
resp = yield from s3.get_object(Bucket='mybucket', Key='k')
stream = resp['Body']
try:
chunk = yield from stream.read(10)
while len(chunk) > 0:
...
chunk = yield from stream.read(10)
finally:
stream.close()
and actually in your case you can even use readline()
https://github.com/KeepSafe/aiohttp/blob/c39355bef6c08ded5c80e4b1887e9b922bdda6ef/aiohttp/streams.py#L587
I'm using Django 1.8.1 with Python 3.4 and i'm trying to use requests to download a processed file. The following code works perfect for a normal request.get command to download the exact file at the server location, or unprocessed file.
The file needs to get processed based on the passed data (shown below as "data"). This data will need to get passed into the Django backend, and based off the text pass variables to run an internal program from the server and output .gcode instead .stl filetype.
python file.
import requests, os, json
SERVER='http://localhost:8000'
authuser = 'admin#google.com'
authpass = 'passwords'
#data not implimented
##############################################
data = {FirstName:Steve,Lastname:Escovar}
############################################
category = requests.get(SERVER + '/media/uploads/9128342/141303729.stl', auth=(authuser, authpass))
#download to path file
path = "/home/bradman/Downloads/requestdata/newfile.stl"
if category.status_code == 200:
with open(path, 'wb') as f:
for chunk in category:
f.write(chunk)
I'm very confused about this, but I think the best course of action is to pass the data along with request.get, and somehow make some function to grab them inside my views.py for Django. Anyone have any ideas?
To use data in request you can do
get( ... , params=data)
(and you get data as parameters in url)
or
post( ... , data=data).
(and you send data in body - like HTML form)
BTW. some APIs need params= and data= in one request of GET or POST to send all needed information.
Read requests documentation
Using requests_toolbelt to upload large files in a Multipart form, I have constructed a method below which succeeds in uploading the file, however I cannot access the posted filename. How do I access the filename on the server?
# client-side
file = open('/Volumes/Extra/test/my_video.mpg', 'rb')
payload = MultipartEncoder({file.name: file})
r = requests.post(url, data=payload, headers={'Content-Type': 'application/octet-stream'})
# server-side
#view_config(route_name='remote.agent_upload', renderer='json')
def remote_agent_upload(request):
r = request.response
fs = request.body_file
f = open('/Volumes/Extra/tests2/bar.mpg', 'wb') # wish to use filename here
f.write(fs.read())
fs.close()
f.close()
return r
OK, it looks like you are using the name of the file as the field name. Also, the way that you are doing it, seems like the entire post content is being written to file... Is this the desired outcome? Have you tried to actually play your mpg files after you write them on the server side?
I don't have an HTTP server readily available to test at the moment which automagically gives me a request object, but I am assuming that the request object is a webob.Request object (at least it seems like that is the case, please correct me if I'm wrong)
OK, let me show you my test. (This works on python3.4, not sure what version of Python you are using, but I think it should also work on Python 2.7 - not tested though)
The code in this test is a bit long, but it is heavily commented to help you understand what I did every step of the way. Hopefully, it will give you a better understanding of how HTTP requests and responses work in python with the tools you are using
# My Imports
from requests_toolbelt import MultipartEncoder
from webob import Request
import io
# Create a buffer object that can be read by the MultipartEncoder class
# This works just like an open file object
file = io.BytesIO()
# The file content will be simple for my test.
# But you could just as easily have a multi-megabyte mpg file
# Write the contents to the file
file.write(b'test mpg content')
# Then seek to the beginning of the file so that the
# MultipartEncoder can read it from the beginning
file.seek(0)
# Create the payload
payload = MultipartEncoder(
{
# The name of the file upload field... Not the file name
'uploadedFile': (
# This would be the name of the file
'This is my file.mpg',
# The file handle that is ready to be read from
file,
# The content type of the file
'application/octet-stream'
)
}
)
# To send the file, you would use the requests.post method
# But the content type is not application-octet-stream
# The content type is multipart/form-data; with a boundary string
# Without the proper header type, your server would not be able to
# figure out where the file begins and ends and would think the
# entire post content is the file, which it is not. The post content
# might even contain multiple files
# So, to send your file, you would use:
#
# response = requests.post(url, data=payload, headers={'Content-Type': payload.content_type})
# Instead of sending the payload to the server,
# I am just going to grab the output as it would be sent
# This is because I don't have a server, but I can easily
# re-create the object using this output
postData = payload.to_string()
# Create an input buffer object
# This will be read by our server (our webob.Request object)
inputBuffer = io.BytesIO()
# Write the post data to the input buffer so that the webob.Request object can read it
inputBuffer.write(postData)
# And, once again, seek to 0
inputBuffer.seek(0)
# Create an error buffer so that errors can be written to it if there are any
errorBuffer = io.BytesIO()
# Setup our wsgi environment just like the server would give us
environment = {
'HTTP_HOST': 'localhost:80',
'PATH_INFO': '/index.py',
'QUERY_STRING': '',
'REQUEST_METHOD': 'POST',
'SCRIPT_NAME': '',
'SERVER_NAME': 'localhost',
'SERVER_PORT': '80',
'SERVER_PROTOCOL': 'HTTP/1.0',
'CONTENT_TYPE': payload.content_type,
'wsgi.errors': errorBuffer,
'wsgi.input': inputBuffer,
'wsgi.multiprocess': False,
'wsgi.multithread': False,
'wsgi.run_once': False,
'wsgi.url_scheme': 'http',
'wsgi.version': (1, 0)
}
# Create our request object
# This is the same as your request object and should have all our info for reading
# the file content as well as the file name
request = Request(environment)
# At this point, the request object is the same as what you get on your server
# So, from this point on, you can use the following code to get
# your actual file content as well as your file name from the object
# Our uploaded file is in the POST. And the POST field name is 'uploadedFile'
# Grab our file so that it can be read
uploadedFile = request.POST['uploadedFile']
# To read our content, you can use uploadedFile.file.read()
print(uploadedFile.file.read())
# And to get the file name, you can use uploadedFile.filename
print(uploadedFile.filename)
So, I think this modified code will work for you. (Hopefully)
Again, not tested because I don't actually have a server to test with. And also, I don't know what kind of object your "request" object is on the server side.... OK, here goes:
# client-side
import requests
file = open('/Volumes/Extra/test/my_video.mpg', 'rb')
payload = MultipartEncoder({'uploadedFile': (file.name, file, 'application/octet-stream')})
r = requests.post('http://somewhere/somefile.py', data=payload, headers={'Content-Type': payload.content_type})
# server-side
#view_config(route_name='remote.agent_upload', renderer='json')
def remote_agent_upload(request):
# Write your actual file contents, not the post data which contains multi part boundary
uploadedFile = request.POST['uploadedFile']
fs = uploadedFile.file
# The file name is insecure. What if the file name comes through as '../../../etc/passwd'
# If you don't secure this, you've just wiped your /etc/passwd file and your server is toast
# (assuming the web user has write permission to the /etc/passwd file
# which it shouldn't, but just giving you a worst case scenario)
fileName = uploadedFile.filename
# Secure the fileName here...
# Make sure it doesn't have any slashes or double dots, or illegal characters, etc.
# I'll leave that up to you
# Write the file
f = open('/Volumes/Extra/tests2/' + fileName, 'wb')
f.write(fs.read())
Probably too late for the original OP but may help someone else. This is how I upload a file with accompanying json in a multipart/form-data upload using the MultipartEncoder. When I require a file to uploaded as binary with a single json string as part of a multipart request (so there are just two parts, the file and the json). Note that in creating my request header (it's a custom header as designated by the receiving server) I get the content_type from the encoded object (it usually comes through as multipart/form-data). I'm using simplejson.dumps but you can just use json.dumps I believe.
m = MultipartEncoder([
('json', (None, simplejson.dumps(datapayload), 'text/plain')),
('file', (os.path.basename(file_path), open(file_path, 'rb'), 'text/plain'))],
None, encoding='utf-8')
headers = {'Authorization': 'JwToken' + ' ' + jwt_str, 'content-type': m.content_type}
response = requests.post(uri, headers=headers, data=m, timeout=45, verify=True )
In the file part, the field is called "file", but I use os.path.basename(file_path) to get just the filename from a full file path e.g. c:\temp\mytestfile.txt . It's possible I could just as easily call the file something else (that's not the original name) in this field if I wanted to.
I have seen the receipes for uploading files via multipartform-data and pycurl. Both methods seem to require a file on disk. Can these recipes be modified to supply binary data from memory instead of from disk ? I guess I could just use a xmlrpc server instead.I wanted to get around having to encode and decode the binary data and send it raw... Do pycurl and mutlipartform-data work with raw data ?
This (small) library will take a file descriptor, and will do the HTTP POST operation: https://github.com/seisen/urllib2_file/
You can pass it a StringIO object (containing your in-memory data) as the file descriptor.
Find something that cas work with a file handle. Then simply pass a StringIO object instead of a real file descriptor.
I met similar issue today, after tried both and pycurl and multipart/form-data, I decide to read python httplib/urllib2 source code to find out, I did get one comparably good solution:
set Content-Length header(of the file) before doing post
pass a opened file when doing post
Here is the code:
import urllib2, os
image_path = "png\\01.png"
url = 'http://xx.oo.com/webserviceapi/postfile/'
length = os.path.getsize(image_path)
png_data = open(image_path, "rb")
request = urllib2.Request(url, data=png_data)
request.add_header('Cache-Control', 'no-cache')
request.add_header('Content-Length', '%d' % length)
request.add_header('Content-Type', 'image/png')
res = urllib2.urlopen(request).read().strip()
return res
see my blog post: http://www.2maomao.com/blog/python-http-post-a-binary-file-using-urllib2/
Following python code works reliable on 2.6.x. The input data is of type str.
Note that the server that receives the data has to loop to read all the data as the large
data sizes will be chunked. Also attached java code snippet to read the chunked data.
def post(self, url, data):
self.curl = pycurl.Curl()
self.response_headers = StringIO.StringIO()
self.response_body = io.BytesIO()
self.curl.setopt(pycurl.WRITEFUNCTION, self.response_body.write)
self.curl.setopt(pycurl.HEADERFUNCTION, self.response_headers.write)
self.curl.setopt(pycurl.FOLLOWLOCATION, 1)
self.curl.setopt(pycurl.MAXREDIRS, 5)
self.curl.setopt(pycurl.TIMEOUT, 60)
self.curl.setopt(pycurl.ENCODING,"deflate, gzip")
self.curl.setopt(pycurl.URL, url)
self.curl.setopt(pycurl.VERBOSE, 1)
self.curl.setopt(pycurl.POST,1)
self.curl.setopt(pycurl.POSTFIELDS,data)
self.curl.setopt(pycurl.HTTPHEADER, [ "Content-Type: octate-stream" ])
self.curl.setopt(pycurl.POSTFIELDSIZE, len(data))
self.curl.perform()
return url, self.curl.getinfo(pycurl.RESPONSE_CODE),self.response_headers.getvalue(), self.response_body.getvalue()
Java code for the servlet engine:
int postSize = Integer.parseInt(req.getHeader("Content-Length"));
results = new byte[postSize];
int read = 0;
while(read < postSize) {
int n = req.getInputStream().read(results);
if (n < 0) break;
read += n;
}
found a solution that works with the cherrypy file upload example: urllib2-binary-upload.py
import io # Part of core Python
import requests # Install via: 'pip install requests'
# Get the data in bytes. I got it via:
# with open("smile.gif", "rb") as fp: data = fp.read()
data = b"GIF89a\x12\x00\x12\x00\x80\x01\x00\x00\x00\x00\xff\x00\x00!\xf9\x04\x01\n\x00\x01\x00,\x00\x00\x00\x00\x12\x00\x12\x00\x00\x024\x84\x8f\x10\xcba\x8b\xd8ko6\xa8\xa0\xb3Wo\xde9X\x18*\x15x\x99\xd9'\xad\x1b\xe5r(9\xd2\x9d\xe9\xdd\xde\xea\xe6<,\xa3\xd8r\xb5[\xaf\x05\x1b~\x8e\x81\x02\x00;"
# Hookbin is similar to requestbin - just a neat little service
# which helps you to understand which queries are sent
url = 'https://hookb.in/je2rEl733Yt9dlMMdodB'
in_memory_file = io.BytesIO(data)
response = requests.post(url, files=(("smile.gif", in_memory_file),))