We Keep Coding

How to make Bottle refuse too big uploaded files?

Here is a way to accept upload with Bottle:
<form action="/upload" method="post" enctype="multipart/form-data">
<input type="file" name="file" />
</form>
and
from bottle import route, request
#route('/upload', method='POST')
def do_upload():
myfile = request.files.get('file')
size = len(myfile.read()) # oops the file is already read anyway!
if size > 1024*1024: # 1 MB
return "File too big"
However, with this technique a 500 MB file would be read anyway, before noticing it's a "too big file".
Question: how to prevent a Bottle server to even accept a too big uploaded file, without having to read it first (and waste bandwidth/memory!)?
If not possible with Bottle only, how to do it with Apache + mod_wsgi (I currently use this)?

Because you are using Apache, you can add to the Apache configuration the LimitRequestBody directive and specify the limit. The request will be rejected before it even gets to your Python code.
https://httpd.apache.org/docs/2.4/mod/core.html#limitrequestbody

Uploading Files in webapp2/GAE

I need to upload and process a CSV file from a form in a Google App Engine application based on Webapp2 (Python) I understand I could use blobstore to temporary store the file but I am curious to know if there is a way to process the file without having to store it at all.

If you need to upload a file via webapp2 with an HTML form, the first thing you need to do is change HTML form enctype attribute to multipart/form-data, so the code snippet seems like:
<form action="/emails" class="form-horizontal" enctype="multipart/form-data" method="post">
<input multiple id="file" name="attachments" type="file">
</form>
In python code, you can read the file directly via request.POST, here's a sample code snippet:
class UploadHandler(BaseHandler):
def post(self):
attachments = self.request.POST.getall('attachments')
_attachments = [{'content': f.file.read(),
'filename': f.filename} for f in attachments]

The content of uploaded files is in self.request.POST in your handler, so you can get that content (assuming e.g the field for the uploaded file is named 'foo') with e.g
content = self.request.POST.multi['foo'].file.read()
So now you have the content as a string -- process it as you wish. This does of course assume the thing will fit in memory (no multi-megabyte uploads!-)...

Uploading file to Google App Engine using new dev_appserver

Google App Engine documentation makes it appear very simple to get the contents of an uploaded file (self.request.get('user_file')), but while I can do this with old_dev_appserver.py, I cannot get it with the current dev_appserver.py (v1.9.2).
Here's a simple example that generates the incoming data:
<form name="myform" action="http://localhost:8080/sync?cmd=tester" enctype="multipart/form-data" method="post">
Username: <input type="file" name="user_file" />
<input type="submit" value="Submit" />
</form>
In old_dev_appserver.py, one could get the file in GAE via self.request.get('user_file'), but not with the latest (1.9.2) dev_appserver.py
WebApp2 says "Uploaded files are available as cgi.FieldStorage (see the cgi module) instances directly in request.POST." But, request.POST is empty, and cgi.FieldStorage() does not contain 'user_file' either.
Strangely, if I print out self.request.params, I do see an element in the UnicodeMultiDict that is (u'user_file', FieldStorage(u'user_file', u'myfile.ico')). But when I try to get that element, either via named access or just iterating over params, I cannot get it. Even if I do a Len(self.request.params) I get one less than what I see, and the 'user_file' element is missing. If I do this with old_dev_appserver, the Len(self.request.params) is correct.
How do I get user_file?

Figured it out (finally!). I had been logging elements of the request (params, body, etc.) as I sorted through other issues. But you can only instantiate a cgi.FieldStorage element once, so by the time I got to the actual self.request.get('user_file'), it was already gone.
Classic case in which removing the debugging actually makes it work.

Uploading CSV to Flask for background processing

I'm looking to use Flask to host a single-page website that would allow users to upload a CSV that would be parsed and put into a database. All of the database shenanigans are complete (through SQLalchemy in another Python script) and I've got everything worked out once a script has access to the CSV, I just need help getting it there.
Here's the scenario:
1. User directs browser at URL (probably something like
http://xxx.xxx.xxx.xxx/upload/)
2. User chooses CSV to upload
3. User presses upload
4. File is uploaded and processed, but user is sent to a thank you page while our
script is still working on the CSV (so that their disconnect doesn't cause the
script to abort).
It's totally cool if the CSV is left on the server (in fact, it's probably preferred since we'd have a backup in case processing went awry)
I think what I want is a daemon that listens on a socket, but I'm not really experienced with this and don't know where to start getting it configured or setting up Flask.
If you think some framework other than Flask would be easier, definitely let me know, I'm not tied to Flask, I've just read that it's pretty easy to set up!
Thank you very much!!

Here is a (very slightly simplified) example of handling file uploading in web.py based on a cook book example (the Flash example, which I have less experience with, looks even easier):
import web
urls = ('/', 'Upload')
class Upload:
def GET(self):
web.header("Content-Type","text/html; charset=utf-8")
return """
<form method="POST" enctype="multipart/form-data" action="">
<input type="file" name="myfile" />
<br/>
<input type="submit" />
"""
def POST(self):
x = web.input(myfile={})
filedir = '/uploads' # change this to the directory you want to store the file in.
if 'myfile' in x: # to check if the file-object is created
filepath=x.myfile.filename.replace('\\','/') # replaces the windows-style slashes with linux ones.
filename=filepath.split('/')[-1] # splits the and chooses the last part (the filename with extension)
fout = open(filedir +'/'+ filename,'wb') # creates the file where the uploaded file should be stored
fout.write(x.myfile.file.read()) # writes the uploaded file to the newly created file.
fout.close() # closes the file, upload complete.
raise web.seeother('/')
if __name__ == "__main__":
app = web.application(urls, globals())
app.run()
This renders a upload form, and then (on POST) reads the uploaded file and saves it to a designated path.

Python / CGI - Upload file attempt returns an empty page

I really searched about 50 related pages but never seen a problem similar to my problem. When I press the submit button, it calls the script but the script returns an empty page and I see no file was uploaded. There is no typing error in my codes, I checked it several times and I really need this code running for my project. What might be the problem? I am running apache under ubuntu and my codes are:
html code:
<html><body>
<form enctype="multipart/form-data" action="save_file.py" method="post">
<p>File: <input type="file" name="file"></p>
<p><input type="submit" value="Upload"></p>
</form>
</body></html>
python code:
#!/usr/bin/env python
import cgi, os
import cgitb; cgitb.enable()
try: #windows needs stdio set for binary mode
import msvcrt
msvcrt.setmode (0, os.O_BINARY)
msvcrt.setmode (1, os.O_BINARY)
except ImportError:
pass
form = cgi.FieldStorage()
#nested FieldStorage instance holds the file
fileitem = form['file']
#if file is uploaded
if fileitem.filename:
#strip leading path from filename to avoid directory based attacks
fn = os.path.basename(fileitem.filename)
open('/files' + fn, 'wb').write(fileitem.file.read())
message = 'The file "' + fn + '" was uploaded successfully'
else:
message = 'No file was uploaded'
print """\
Content-Type: text/html\n
<html><body>
<p>%s</p>
</body></html>
""" % (message,)

I just tested your script, with a few small corrections to the paths to make it work for me locally. With the paths set correctly, and permissions set properly, this code does work fine.
Here are the things to make sure of:
In your html file's form properties, make sure you are pointing to the python script that lives in a cgi-bin: action="/cgi-bin/save_file.py". For me, I have a cgi-bin at the root of my web server, and I placed the python script there. It will not work if you are running the script from a standard document location on the web server
Make sure your save_file.py has executable permissions: chmod 755 save_file.py
In your save_file.py, ensure that you are building a valid path to open the file for saving. I made mine absolute just for testing purposes, but something like this: open(os.path.join('/path/to/upload/files', fn)
With those points set correctly, you should not have any problems.