I'm running python 2.4 from cgi and I'm trying to upload to a cloud service using a python api. In php, the $_FILE array contains a "tmp" element which is where the file lives until you place it where you want it. What's the equivalent in python?
if I do this
fileitem = form['file']
fileitem.filename is the name of the file
if i print fileitem, the array simply contains the file name and what looks to be the file itself.
I am trying to stream things and it requires the tmp location when using the php api.
The file is a real file, but the cgi.FieldStorage unlinked it as soon as it was created so that it would exist only as long as you keep it open, and no longer has a real path on the file system.
You can, however, change this...
You can extend the cgi.FieldStorage and replace the make_file method to place the file wherever you want:
import os
import cgi
class MyFieldStorage(cgi.FieldStorage):
def make_file(self, binary=None):
return open(os.path.join('/tmp', self.filename), 'wb')
You must also keep in mind that the FieldStorage object only creates a real file if it recieves more than 1000B (otherwise it is a cStringIO.StringIO)
EDIT: The cgi module actually makes the file with the tempfile module, so check that out if you want lots of gooey details.
Here's a code snippet taken from my site:
h = open("user_uploaded_file", "wb")
while 1:
data = form["file"].file.read(4096)
if not data:
break
h.write(data)
h.close()
Hope this helps.
Related
I am currently developing a little backend project for myself with the Python Framework FastAPI. I made an endpoint, where the user should be able to upload 2 files, while the first one is a zip-file (which contains X .xmls) and the latter a normal .xml file.
The code is as follows:
#router.post("/sendxmlinzip/")
def create_upload_files_with_zip(files: List[UploadFile] = File(...)):
if not len(files) == 2:
raise Httpex.EXPECTEDTWOFILES
my_file = files[0].file
zfile = zipfile.ZipFile(my_file, 'r')
filelist = []
for finfo in zfile.infolist():
print(finfo)
ifile = zfile.open(finfo)
line_list = ifile.readlines()
print(line_list)
This should print the content of the files, that are in the .zip file, but it raises the Exception
AttributeError: 'SpooledTemporaryFile' object has no attribute 'seekable'
In the row ifile = zfile.open(finfo)
Upon approximately 3 days research with a lot of trial and error involved, trying to use different functions such as .read() or .extract(), I gave up. Because the python docs literally state, that this should be possible in this way...
For you, who do not know about FastAPI, it's a backend fw for Restful Webservices and is using the starlette datastructure for UploadFile. Please forgive me, if I have overseen something VERY obvious, but I literally tried to check every corner, that may have been the possible cause of the error such as:
Check, whether another implementation is possible
Check, that the .zip file is correct
Check, that I attach the correct file (lol)
Debug to see, whether the actual data, that comes to the backend is indeed the .zip file
This is a known Python bug:
SpooledTemporaryFile does not fully satisfy the abstract for IOBase.
Namely, seekable, readable, and writable are missing.
This was discovered when seeking a SpooledTemporaryFile-backed lzma
file.
As #larsks suggested in his comment, I would try writing the contents of the spooled file to a new TemporaryFile, and then operate on that. As long as your files aren't too large, that should work just as well.
This is my workaround
with zipfile.ZipFile(io.BytesIO(file.read()), 'r') as zip:
I use multiple python scripts that collect data and write it into one single json data file.
It is not possible to combine the scripts.
The writing process is fast and it happens often that errors occur (e.g. some chars at the end duplicate), which is fatal, especially since I am using json format.
Is there a way to prevent a python script to write into a file if there are other script currently trying to write into the file? (It would be absolutely ok, if the data that the python script tries to write into the file gets lost, but it is important that the file syntax does not get somehow 'injured'.)
Code Snipped:
This opens the file and retrieves the data:
data = json.loads(open("data.json").read())
This appends a new dictionary:
data.append(new_dict)
And the old file is overwritten:
open("data.json","w").write( json.dumps(data) )
Info: data is a list which contains dicts.
Operating System: The hole process takes place on linux server.
On Windows, you could try to create the file, and bail out if an exception occurs (because file is locked by another script). But on Linux, your approach is bound to fail.
Instead, I would
write one file per new dictionary, suffixing filename by process ID and a counter
consuming process(es) don't read a single file, but the sorted files (according to modification time) and build the data from it
So in each script:
filename = "data_{}_{}.json".format(os.getpid(),counter)
counter+=1
open(filename ,"w").write( json.dumps(new_dict) )
and in the consumers (reading each dict of sorted files in a protected loop):
files = sorted(glob.glob("*.json"),key=os.path.getmtime())
data = []
for f in files:
try:
with open(f) as fh:
data.append(json.load(fh))
except Exception:
# IO error, malformed json file: ignore
pass
I will post my own solution, since it works for me:
Every single python script checks (before opening and writing the data file) whether a file called data_check exists. If so, the pyhthon script does not try to read and write the file and dismisses the data, that was supposed to be written into the file. If not, the python script creates the file data_check and then starts to read and wirte the file. After the writing process is done the file data_check is removed.
I am trying to use "requests" package and retrieve info from Github, like the Requests doc page explains:
import requests
r = requests.get('https://api.github.com/events')
And this:
with open(filename, 'wb') as fd:
for chunk in r.iter_content(chunk_size):
fd.write(chunk)
I have to say I don't understand the second code block.
filename - in what form do I provide the path to the file if created? where will it be saved if not?
'wb' - what is this variable? (shouldn't second parameter be 'mode'?)
following two lines probably iterate over data retrieved with request and write to the file
Python docs explanation also not helping much.
EDIT: What I am trying to do:
use Requests to connect to an API (Github and later Facebook GraphAPI)
retrieve data into a variable
write this into a file (later, as I get more familiar with Python, into my local MySQL database)
Filename
When using open the path is relative to your current directory. So if you said open('file.txt','w') it would create a new file named file.txt in whatever folder your python script is in. You can also specify an absolute path, for example /home/user/file.txt in linux. If a file by the name 'file.txt' already exists, the contents will be completely overwritten.
Mode
The 'wb' option is indeed the mode. The 'w' means write and the 'b' means bytes. You use 'w' when you want to write (rather than read) froma file, and you use 'b' for binary files (rather than text files). It is actually a little odd to use 'b' in this case, as the content you are writing is a text file. Specifying 'w' would work just as well here. Read more on the modes in the docs for open.
The Loop
This part is using the iter_content method from requests, which is intended for use with large files that you may not want in memory all at once. This is unnecessary in this case, since the page in question is only 89 KB. See the requests library docs for more info.
Conclusion
The example you are looking at is meant to handle the most general case, in which the remote file might be binary and too big to be in memory. However, we can make your code more readable and easy to understand if you are only accessing small webpages containing text:
import requests
r = requests.get('https://api.github.com/events')
with open('events.txt','w') as fd:
fd.write(r.text)
filename is a string of the path you want to save it at. It accepts either local or absolute path, so you can just have filename = 'example.html'
wb stands for WRITE & BYTES, learn more here
The for loop goes over the entire returned content (in chunks incase it is too large for proper memory handling), and then writes them until there are no more. Useful for large files, but for a single webpage you could just do:
# just W becase we are not writing as bytes anymore, just text.
with open(filename, 'w') as fd:
fd.write(r.content)
I've got a bunch of Python webdriver runs that I've converted from Selenium IDE. These runs have 1 or 2 settings that I would like to be able to change using a configuration file that would be created by running a script that collects user input that would set these 2 variables.
Here is my attempt at using the ConfigParser module:
import ConfigParser
file_path_input = raw_input("Enter path to 'webdriver' directory ex: '/home/user/': ")
print "you entered", file_path_input
url_input = raw_input("Enter url that the application will point to, ex: 172.31.13.56 or vtm55.example.com: ")
print "you entered", url_input
def createConfig(file_path_input):
"""
Create a config file
"""
config = ConfigParser.ConfigParser()
config.add_section("application_settings")
config.set("application_settings", "file_path_to_use", "file_path_input")
config.set("application_settings", "url_to_use", "url_input")
config.set("application_settings", "settings_info",
"Your application directory is in %(file_path_to_use)s and your application url is %(url_to_use)s")
with open(file_path_input, "wb") as config_file:
config.write(config_file)
The raw_input() and print() lines work, but the configuration file doesn't appear to be generated at all. Once the file has been created, I'd like to be able to insert the variables file_path_to_use and url_to_use in my various Python webdriver runs.
Your indentation is problematic, particularly in the createConfig function.
"file_path_input" is different from file_path_input.
Use 'w' (a normal write) instead of 'wb' (write bytes).
You have config and config_file backwards - call write on config_file, then pass it the content you want to write. If you have a list of strings, simply loop through them and write each one with basic file I/O:
configs = ['file_path_to_use:' + file_path_input,
'url_to_use:' + url_input]
with open(file_path_input, 'w') as config_file:
for line in configs:
config_file.write(line + '\n')
Sample result:
file_path_to_use:/home/user/
url_input:vtm.example.com
I didn't include the "settings_info" portion because it's just a recap.
You can then split or partition on ':' when reading the file back in.
I would recommend writing your output as text, not using ConfigParser. I've had odd issues with using to create config files in the past (though better luck using it to read them). If you convert to write just a text file with standard file-i/o, you have more specificity in how it's written, while still writing it in a format that ConfigParser can later read in.
You could scrap using ConfigParser and instead create a .py file with the variables saved within in the format KEY=VALUE in the same way you are creating a config file.
Then within your code, write your own function that opens the .py file (using with syntax, so that it automatically closes the file without .close()) and reads it, saving it as a string.
You can then use string manipulation to get the string you want and assign it to a new variable (remember to convert it to the data type you want to use, i.e. int() for an integer). Use type() to check the type of the variable if you aren't sure.
It actually starts with lowercase configparser and configparser.ConfigParser().
Please update.
I'm trying to use AjaxUpload with Python:
http://valums.com/ajax-upload/
I would like to know how to access the uploaded file with Python. On the web site, it says:
* PHP: $_FILES['userfile']
* Rails: params[:userfile]
What is the Syntax for Python?
request.params['userfile'] doesn't seem to work.
Thanks in advance! Here is my current code (using PIL imported as Image)
im = Image.open(request.params['myFile'].file)
import cgi
#This will give you the data of the file,
# but won't give you the filename, unfortunately.
# For that you have to do some other trick.
file_data = cgi.FieldStorage.getfirst('file')
#<IGNORE if you're not using mod_python>
#(If you're using mod_python you can also get the Request object
# by passing 'req' to the relevant function in 'index.py', like "def func(req):"
# Then you access it with req.form.getfirst('file') instead. NOTE that the
# first method will work even when using mod_python, but the first FieldStorage
# object called is the only one with relevant data, so if you pass 'req' to the
# function you have to use the method that uses 'req'.)
#</IGNORE>
#Then you can write it to a file like so...
file = open('example_filename.wtvr','w')#'w' is for 'write'
file.write(file_data)
file.close()
#Then access it like so...
file = open('example_filename.wtvr','r')#'r' is for 'read'
#And use file.read() or whatever else to do what you want.
I'm working with Pyramid, and I was trying to do the same thing. After some time I came up with this solution.
from cStringIO import StringIO
from cgi import FieldStorage
fs = FieldStorage(fp=request['wsgi.input'], environ=request)
f = StringIO(fs.value)
im = Image.open(f)
I'm not sure if it's the "right" one, but it seems to work.
in django, you can use:
request.FILES['file']
instead of:
request.POST['file']
i did not know how to do in pylons...maybe it is the same concept..