Is this usage of Python tempfile.NamedTemporaryFile secure (i.e. devoid security issues of deprecated tempfile.mktemp)?
def mktemp2():
"""Create and close an empty temporary file.
Return the temporary filename"""
tf = tempfile.NamedTemporaryFile(delete=False)
tfilename = tf.name
tf.close()
return tfilename
outfilename = mktemp2()
subprocess.call(['program_name','-o',outfilename])
What I need to run external command that requires output file name as one of the arguments. It overwrites the outfilename if that exists without warnings. I want to use temporary file as I just need to read its content, I don't need it later.
Totally unsafe. There is an opportunity for an attacker to create the file with whatever permissions they like (or a symlink) with that name between when it is deleted and opened by the subprocess
If you can instead create the file in a directory other than /tmp that is owned and onnly read/writeable by your process, you don't need to concern yourself with the security of the file as anything in the directory is protected
Related
As part of a Python project on Windows, I need to use small files as a means to communicate different processes. Since the external process must be called with subprocess.run(program, args...), I can't simply obtain a file descriptor for my file and pass it as a parameter to the external process. Instead, I need a file with a name which can be accessed from the normal filesystem. Thus, I would like a simple way to create a temporary file which is stored in memory (RAM) instead of disk, and which has a name other external processes can use to access it. In Linux, this can be achieved with the function os.mkfifo(). However, this function is not available in Windows.
At the moment, I am simply using the tempfile module to create a temporary file which is stored in disk and deleted once it is no longer needed. Here is a small reproducible example for my code:
import tempfile
import subprocess
import os
fd = tempfile.NamedTemporaryFile(mode="w+t", delete=False) # Obtain file descriptor
file_path = fd.name # Obtain file path
# Write data (encoded as str) to the file and close it
fd.write(data)
fd.close()
# Access this same file from an external process
output = subprocess.run( [program, file_path], stdout=subprocess.PIPE).stdout.decode('utf-8')
# Delete the file
os.remove(file_path)
print("External process output:", output)
So, my question is the following: How can I change this code so that in line fd.write(data) the data is written to RAM instead of disk?
I am writing a command line diary application in Python, and I want to be able to store images in an encrypted SQLite database as a byte array (using solution here).
Then to show images I was going to load the byte array using PIL.Image.fromarray(...) and display it with PIL.Image.show(...), but I am concerned after reading the documation as it says it temporarily stores the image:
Obviously this is not desirable as the point of this application is to store elements in an encrypted form and I am worried it will leave these images stored on the disk after the application has exited. But to resolve this worry I can not seem to find where it temporarily stores the image and if this is deleted after. I have checked the /tmp/ directory.
So my question is where does PIL.Image.show(...) temporarily store images and will they be left around on my computer in an unencrypted format after?
After looking at the source of the PIL.Image.show(...) method, the method seems to write the image temporarily to disk (as the documentation stated) and then is immediately erased after it has been opened with an image viewer.
For example, the method for macOS is:
def show_file(self, file, **options):
"""Display given file"""
fd, path = tempfile.mkstemp()
with os.fdopen(fd, "w") as f:
f.write(file)
with open(path, "r") as f:
subprocess.Popen(
["im=$(cat); open -a Preview.app $im; sleep 20; rm -f $im"],
shell=True,
stdin=f,
)
os.remove(path)
return 1
This uses the tempfile.mkstemp([suffix=''[, prefix='tmp'[, dir=None[, text=False]]]]) method to create a temporary file. The documentation states:
If dir is specified, the file will be created in that directory; otherwise, a default directory is used. The default directory is chosen from a platform-dependent list, but the user of the application can control the directory location by setting the TMPDIR, TEMP or TMP environment variables.
As no dir is specified, the default directory used, which is selected like so: (stated here):
When set to a value other than None, this variable defines the default value for the dir argument to all the functions defined in this module.
If tempdir is unset or None at any call to any of the above functions, Python searches a standard list of directories and sets tempdir to the first one which the calling user can create files in. The list is:
The directory named by the TMPDIR environment variable.
The directory named by the TEMP environment variable.
The directory named by the TMP environment variable.
A platform-specific location:
On RiscOS, the directory named by the Wimp$ScrapDir environment variable.
On Windows, the directories C:\TEMP, C:\TMP, \TEMP, and \TMP, in that order.
On all other platforms, the directories /tmp, /var/tmp, and /usr/tmp, in that order.
As a last resort, the current working directory.
I'm currently working on an export feature for a web application using Pyramid on Python and running on Ubuntu 14.04. It zips the files into a NamedTemporaryFile and sends it back through a FileResponse:
# Create the temporary file to store the zip
with NamedTemporaryFile(delete=True) as output:
map_zip = zipfile.ZipFile(output, 'w', zipfile.ZIP_DEFLATED)
length_mapdir = len(map_directory)
for root, dirs, files in os.walk(map_directory, followlinks=True):
for file in files:
file_path = os.path.join(root, file)
map_zip.write(file_path, file_path[length_mapdir:])
map_zip.close()
#Send the response as an attachement to let the user download the file
response = FileResponse(os.path.abspath(output.name))
response.headers['Content-Type'] = 'application/download'
response.headers['Content-Disposition'] = 'attachement; filename="'+filename+'"'
return response
On the client's side, the export takes some time then the file download popup appears, nothing goes wrong and everything is in the zip as planned.
While the file is zipping, I can see a file taking up more and more size in /tmp/, and before the download popup appears, the file disappears. I assume this is the NamedTemporaryFile.
While the file is being zipped or downloaded, there isn't any significant change in the amount of RAM being used, it stays around 40mb while the actual zip is over 800mb.
Where is pyramid downloading the file from? From what I understand of tempfile, it is unlinked when it is closed. If that's true, is it possible another process could write on the memory where the file was stored, corrupting whatever pyramid is downloading?
In Unix environments something called reference counting is used when a file is created, and opened. For each open() call on a file, the reference number is increased, for each close() it is decreased. unlink() is special in that when that is called the file is unlinked from the directory tree, but will remain on disk so long as the reference count stays above 0.
In your case NamedTemporaryFile() creates a file on disk named /tmp/somefile
/tmp/somefile now has a link count of 1
/tmp/somefile then has open() called on it, so that it can return the file to you, this increases the reference count to 1
/tmp/somefile is then written to by your code, in this case a zip file
/tmp/somefile is then passed to FileResponse() which then has open() called on it, increasing the reference count to 2
You exit the scope of the with statement, and NamedTemporaryFile() calls close() followed by unlink(). Your file now has 1 reference to it, and a link count of 0. Due to the reference still existing, the file still exists on disk, but can no longer be seen when searching for it.
FileResponse() is iterated over by your WSGI server, and eventually once the file has been fully read, your WSGI server calls close() on it, dropping the reference count to 0, at which point the file system will clean the file up entirely
It is at that last point that the file is no longer accessible. In the mean time your file is completely safe and there is no way for it to be overwritten in memory or otherwise.
That being said, if FileResponse() was lazy loaded for example (i.e. it wouldn't open() the file until the WSGI server started sending the response), it would be entirely possible that it would attempt to open() the temporary file too late, and NamedTemporaryFile() would have already deleted the file. Just something to keep in mind.
I have the following method in Python.
def get_rum_data(file_path, query):
if file_path is not None and query is not None:
command = FETCH_RUM_COMMAND_DATA % (constants.RUM_JAR_PATH,
constants.RUM_SERVER, file_path,
query)
print command
execute_command(command).communicate()
Now inside get_rum_data I need to create the file if it does not exists, if it exists I need to append the data. How to do that in python.
I tried, open(file_path, 'w') , which gave me an exception.
Traceback (most recent call last):
File "utils.py", line 180, in <module>
get_rum_data('/User/rokumar/Desktop/sample.csv', '\'show tables\'')
File "utils.py", line 173, in get_rum_data
open(file_path, 'w')
IOError: [Errno 2] No such file or directory: '/User/rokumar/Desktop/sample.csv'
I though open would create file in write mode.
It shall be as simple as:
fname = "/User/rokumar/Desktop/sample.csv"
with open(fname, "a") as f:
# do here what you want
# it will get closed at this point by context manager
But I would suspect, that you are trying to use not existing directory. Usually, "a" mode creates the file if it can be created.
Make sure, the directory exists.
You can check if all directories in file_path exist prior to trying to write a file.
import os
file_path = '/Users/Foo/Desktop/bar.txt'
print os.path.dirname(file_path)
# /Users/Foo/Desktop
if not os.path.exists(os.path.dirname(file_path)):
os.mkdirs(os.path.dirname(file_path))
# recursively create directories if necessary
with open(file_path, "a") as my_file:
# mode a will either create the file if it does not exist
# or append the content to its end if it exists.
my_file.write(your_text_to_append)
-- Edit: Small and probably unnecessary extension --
expanduser:
In your case, as it turned out that the initial problem was a missing s in the path to the user directory, there is a useful feature for resolving the current users base directory (works for unix, linux and windows): see expanduser from the os.path module. With that you could write your path as path = '~/Desktop/bar.txt' and the tilde (~) will be expanded just like on your shell. (with the additional benefit that if you started your script from another user it would then expand to her home directory.
App config directory:
Since in most cases it is not desirable to write a file to the desktop (*nix systems for instance might not have a desktop installed), there is a great utility function in the click package. If you look at the get_app_dir() function on Github, you can see how they provide expanding to an appropriate app dir and supporting multiple operating systems (the function has no dependencies besides the WIN variable that is defined in the _compat.py module as WIN = sys.platform.startswith('win') and the _posixify() function defined on line 17. Often that's a good starting point for defining an app directory to store certain data in.
As tempfile.mktemp is depreciated in Python 2.7 I generate a unique path to a temporary file as follows:
temp = tempfile.NamedTemporaryFile(suffix=".py")
path_to_generated_py = temp.name
temp.close()
# now I use path_to_gerated_py to create a python file
Is this the recommended way in Python 2.7? As I close the temp file immediately it looks like misusing NamedTemporaryFile....
The direct replacement for tempfile.mktemp() is tempfile.mkstemp(). The latter creates the file, like NamedTemporaryFile, so you must close it (as in your code snippet). The difference with NamedTemporaryFile is that the file is not deleted when closed. This is actually required: your version has a theoretical race condition where two processes might end up with the same temporary file name. If you use mkstemp() instead, the file is never deleted, and will likely be overwritten by the 3rd-party library you use --- but at any point in time, the file exists, and so there is no risk that another process would create a temporary file of the same name.