We Keep Coding

Why a new NamedTemporaryFile object has a path, but a file is not available? [duplicate]

I am attempting to create and write to a temporary file on Windows OS using Python. I have used the Python module tempfile to create a temporary file.
But when I go to write that temporary file I get an error Permission Denied. Am I not allowed to write to temporary files?! Am I doing something wrong? If I want to create and write to a temporary file how should should I do it in Python? I want to create a temporary file in the temp directory for security purposes and not locally (in the dir the .exe is executing).
IOError: [Errno 13] Permission denied: 'c:\\users\\blah~1\\appdata\\local\\temp\\tmpiwz8qw'
temp = tempfile.NamedTemporaryFile().name
f = open(temp, 'w') # error occurs on this line

NamedTemporaryFile actually creates and opens the file for you, there's no need for you to open it again for writing.
In fact, the Python docs state:
Whether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
That's why you're getting your permission error. What you're probably after is something like:
f = tempfile.NamedTemporaryFile(mode='w') # open file
temp = f.name # get name (if needed)

Use the delete parameter as below:
tmpf = NamedTemporaryFile(delete=False)
But then you need to manually delete the temporary file once you are done with it.
tmpf.close()
os.unlink(tmpf.name)
Reference for bug: https://github.com/bravoserver/bravo/issues/111
regards,
Vidyesh

Consider using os.path.join(tempfile.gettempdir(), os.urandom(24).hex()) instead. It's reliable, cross-platform, and the only caveat is that it doesn't work on FAT partitions.
NamedTemporaryFile has a number of issues, not the least of which is that it can fail to create files because of a permission error, fail to detect the permission error, and then loop millions of times, hanging your program and your filesystem.

The following custom implementation of named temporary file is expanded on the original answer by Erik Aronesty:
import os
import tempfile
class CustomNamedTemporaryFile:
"""
This custom implementation is needed because of the following limitation of tempfile.NamedTemporaryFile:
> Whether the name can be used to open the file a second time, while the named temporary file is still open,
> varies across platforms (it can be so used on Unix; it cannot on Windows NT or later).
"""
def __init__(self, mode='wb', delete=True):
self._mode = mode
self._delete = delete
def __enter__(self):
# Generate a random temporary file name
file_name = os.path.join(tempfile.gettempdir(), os.urandom(24).hex())
# Ensure the file is created
open(file_name, "x").close()
# Open the file in the given mode
self._tempFile = open(file_name, self._mode)
return self._tempFile
def __exit__(self, exc_type, exc_val, exc_tb):
self._tempFile.close()
if self._delete:
os.remove(self._tempFile.name)

This issue might be more complex than many of you think. Anyway this was my solution:
Make use of atexit module
def delete_files(files):
for file in files:
file.close()
os.unlink(file.name)
Make NamedTemporaryFile delete=False
temp_files = []
result_file = NamedTemporaryFile(dir=tmp_path(), suffix=".xlsx", delete=False)
self.temp_files.append(result_file)
Register delete_files as a clean up function
atexit.register(delete_files, temp_files)

tempfile.NamedTemporaryFile() :
It creates and opens a temporary file for you.
f = open(temp, 'w') :
You are again going to open the file which is already open and that's why you are getting Permission Denied error.
If you really wants to open the file again then you first need to close it which will look something like this-
temp= tempfile.NamedTemporaryFile()
temp.close()
f = open(temp.name, 'w')

Permission was denied because the file is Open during line 2 of your code.
close it with f.close() first then you can start writing on your tempfile

File lives on windows vs linux in Python 3.4

I have a script that I need to run on ubuntu and windows each using Python 3.4 and when I run on windows I get an exception, "PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\Users\me\Desktop\tmp9uvk57b4.txt'" while on linux, it works error free.
I've boiled down my problem to this example snippet. I'm not sure where the problem lies, but the snippet takes some text and writes it to a temporary file. After a while, it removes the temp file and that is where the error comes in.
#!/usr/bin/env python3
import os
import tempfile
msg = "THIS IS A HORRIBLE MESSAGE"
txt = None
try:
txt = tempfile.mkstemp(dir='.', suffix='.txt')[1]
with open(txt, "w") as f:
f.write(msg)
except Exception as exp:
raise exp
finally:
if txt:
os.remove(txt)
I assume there is some issue where windows doesn't close the file while linux does. Can I just explicitly close it again? Will that mess up anything on linux? Is there a good windows/linux gotcha resource?

tempfile.mkstemp has two return values, an open filehandle and the filename. You don't use the open filehandle, such that it is never closed. Therefore the error message.
import os
import tempfile
msg = "THIS IS A HORRIBLE MESSAGE"
fd, filename = tempfile.mkstemp(dir='.', suffix='.txt')
try:
with os.fdopen(fd, "w") as f:
f.write(msg)
finally:
os.remove(filename)

Subprocess error file

I'm using the python module subprocess to call a program and redirect the possible std error to a specific file with the following command:
with open("std.err","w") as err:
subprocess.call(["exec"],stderr=err)
I want that the "std.err" file is created only if there are errors, but using the command above if there are no errors the code will create an empty file.
How i can make python create a file only if it's not empty?
I can check after execution if the file is empty and in case remove it, but i was looking for a "cleaner" way.

You could use Popen, checking stderr:
from subprocess import Popen,PIPE
proc = Popen(["EXEC"], stderr=PIPE,stdout=PIPE,universal_newlines=True)
out, err = proc.communicate()
if err:
with open("std.err","w") as f:
f.write(err)
On a side note, if you care about the return code you should use check_call, you could combine it with a NamedTemporaryFile:
from tempfile import NamedTemporaryFile
from os import stat,remove
from shutil import move
try:
with NamedTemporaryFile(dir=".", delete=False) as err:
subprocess.check_call(["exec"], stderr=err)
except (subprocess.CalledProcessError,OSError) as e:
print(e)
if stat(err.name).st_size != 0:
move(err.name,"std.err")
else:
remove(err.name)

You can create your own context manager to handle the cleanup for you -- you can't really do what you're describing here, which boils down to asking how you can see into the future. Something like this (with better error handling, etc.):
import os
from contextlib import contextmanager
#contextmanager
def maybeFile(fileName):
# open the file
f = open(fileName, "w")
# yield the file to be used by the block of code inside the with statement
yield f
# the block is over, do our cleanup.
f.flush()
# if nothing was written, remember that we need to delete the file.
needsCleanup = f.tell() == 0
f.close()
if needsCleanup:
os.remove(fileName)
...and then something like:
with maybeFile("myFileName.txt") as f:
import random
if random.random() < 0.5:
f.write("There should be a file left behind!\n")
will either leave behind a file with a single line of text in it, or will leave nothing behind.

Create a temporary FIFO (named pipe) in Python?

How can you create a temporary FIFO (named pipe) in Python? This should work:
import tempfile
temp_file_name = mktemp()
os.mkfifo(temp_file_name)
open(temp_file_name, os.O_WRONLY)
# ... some process, somewhere, will read it ...
However, I'm hesitant because of the big warning in Python Docs 11.6 and potential removal because it's deprecated.
EDIT: It's noteworthy that I've tried tempfile.NamedTemporaryFile (and by extension tempfile.mkstemp), but os.mkfifo throws:
OSError -17: File already exists
when you run it on the files that mkstemp/NamedTemporaryFile have created.

os.mkfifo() will fail with exception OSError: [Errno 17] File exists if the file already exists, so there is no security issue here. The security issue with using tempfile.mktemp() is the race condition where it is possible for an attacker to create a file with the same name before you open it yourself, but since os.mkfifo() fails if the file already exists this is not a problem.
However, since mktemp() is deprecated you shouldn't use it. You can use tempfile.mkdtemp() instead:
import os, tempfile
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'myfifo')
print filename
try:
os.mkfifo(filename)
except OSError, e:
print "Failed to create FIFO: %s" % e
else:
fifo = open(filename, 'w')
# write stuff to fifo
print >> fifo, "hello"
fifo.close()
os.remove(filename)
os.rmdir(tmpdir)
EDIT: I should make it clear that, just because the mktemp() vulnerability is averted by this, there are still the other usual security issues that need to be considered; e.g. an attacker could create the fifo (if they had suitable permissions) before your program did which could cause your program to crash if errors/exceptions are not properly handled.

You may find it handy to use the following context manager, which creates and removes the temporary file for you:
import os
import tempfile
from contextlib import contextmanager
#contextmanager
def temp_fifo():
"""Context Manager for creating named pipes with temporary names."""
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'fifo') # Temporary filename
os.mkfifo(filename) # Create FIFO
try:
yield filename
finally:
os.unlink(filename) # Remove file
os.rmdir(tmpdir) # Remove directory
You can use it, for example, like this:
with temp_fifo() as fifo_file:
# Pass the fifo_file filename e.g. to some other process to read from.
# Write something to the pipe
with open(fifo_file, 'w') as f:
f.write("Hello\n")

How about using
d = mkdtemp()
t = os.path.join(d, 'fifo')

If it's for use within your program, and not with any externals, have a look at the Queue module. As an added benefit, python queues are thread-safe.

Effectively, all that mkstemp does is run mktemp in a loop and keeps attempting to exclusively create until it succeeds (see stdlib source code here). You can do the same with os.mkfifo:
import os, errno, tempfile
def mkftemp(*args, **kwargs):
for attempt in xrange(1024):
tpath = tempfile.mktemp(*args, **kwargs)
try:
os.mkfifo(tpath, 0600)
except OSError as e:
if e.errno == errno.EEXIST:
# lets try again
continue
else:
raise
else:
# NOTE: we only return the path because opening with
# os.open here would block indefinitely since there
# isn't anyone on the other end of the fifo.
return tpath
else:
raise IOError(errno.EEXIST, "No usable temporary file name found")

Why not just use mkstemp()?
For example:
import tempfile
import os
handle, filename = tempfile.mkstemp()
os.mkfifo(filename)
writer = open(filename, os.O_WRONLY)
reader = open(filename, os.O_RDONLY)
os.close(handle)

Implement touch using Python?

touch is a Unix utility that sets the modification and access times of files to the current time of day. If the file doesn't exist, it is created with default permissions.
How would you implement it as a Python function? Try to be cross platform and complete.
(Current Google results for "python touch file" are not that great, but point to os.utime.)

Looks like this is new as of Python 3.4 - pathlib.
from pathlib import Path
Path('path/to/file.txt').touch()
This will create a file.txt at the path.
--
Path.touch(mode=0o777, exist_ok=True)
Create a file at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the file already exists, the function succeeds if exist_ok is true (and its modification time is updated to the current time), otherwise FileExistsError is raised.

This tries to be a little more race-free than the other solutions. (The with keyword is new in Python 2.5.)
import os
def touch(fname, times=None):
with open(fname, 'a'):
os.utime(fname, times)
Roughly equivalent to this.
import os
def touch(fname, times=None):
fhandle = open(fname, 'a')
try:
os.utime(fname, times)
finally:
fhandle.close()
Now, to really make it race-free, you need to use futimes and change the timestamp of the open filehandle, instead of opening the file and then changing the timestamp on the filename (which may have been renamed). Unfortunately, Python doesn't seem to provide a way to call futimes without going through ctypes or similar...
EDIT
As noted by Nate Parsons, Python 3.3 will add specifying a file descriptor (when os.supports_fd) to functions such as os.utime, which will use the futimes syscall instead of the utimes syscall under the hood. In other words:
import os
def touch(fname, mode=0o666, dir_fd=None, **kwargs):
flags = os.O_CREAT | os.O_APPEND
with os.fdopen(os.open(fname, flags=flags, mode=mode, dir_fd=dir_fd)) as f:
os.utime(f.fileno() if os.utime in os.supports_fd else fname,
dir_fd=None if os.supports_fd else dir_fd, **kwargs)

def touch(fname):
if os.path.exists(fname):
os.utime(fname, None)
else:
open(fname, 'a').close()

Why not try this?:
import os
def touch(fname):
try:
os.utime(fname, None)
except OSError:
open(fname, 'a').close()
I believe this eliminates any race condition that matters. If the file does not exist then an exception will be thrown.
The only possible race condition here is if the file is created before open() is called but after os.utime(). But this does not matter because in this case the modification time will be as expected since it must have happened during the call to touch().

For a more low-level solution one can use
os.close(os.open("file.txt", os.O_CREAT))

This answer is compatible with all versions since Python-2.5 when keyword with has been released.
1. Create file if does not exist + Set current time
(exactly same as command touch)
import os
fname = 'directory/filename.txt'
with open(fname, 'a'): # Create file if does not exist
os.utime(fname, None) # Set access/modified times to now
# May raise OSError if file does not exist
A more robust version:
import os
with open(fname, 'a'):
try: # Whatever if file was already existing
os.utime(fname, None) # => Set current time anyway
except OSError:
pass # File deleted between open() and os.utime() calls
2. Just create the file if does not exist
(does not update time)
with open(fname, 'a'): # Create file if does not exist
pass
3. Just update file access/modified times
(does not create file if not existing)
import os
try:
os.utime(fname, None) # Set access/modified times to now
except OSError:
pass # File does not exist (or no permission)
Using os.path.exists() does not simplify the code:
from __future__ import (absolute_import, division, print_function)
import os
if os.path.exists(fname):
try:
os.utime(fname, None) # Set access/modified times to now
except OSError:
pass # File deleted between exists() and utime() calls
# (or no permission)
Bonus: Update time of all files in a directory
from __future__ import (absolute_import, division, print_function)
import os
number_of_files = 0
# Current directory which is "walked through"
# | Directories in root
# | | Files in root Working directory
# | | | |
for root, _, filenames in os.walk('.'):
for fname in filenames:
pathname = os.path.join(root, fname)
try:
os.utime(pathname, None) # Set access/modified times to now
number_of_files += 1
except OSError as why:
print('Cannot change time of %r because %r', pathname, why)
print('Changed time of %i files', number_of_files)

Here's some code that uses ctypes (only tested on Linux):
from ctypes import *
libc = CDLL("libc.so.6")
# struct timespec {
# time_t tv_sec; /* seconds */
# long tv_nsec; /* nanoseconds */
# };
# int futimens(int fd, const struct timespec times[2]);
class c_timespec(Structure):
_fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)]
class c_utimbuf(Structure):
_fields_ = [('atime', c_timespec), ('mtime', c_timespec)]
utimens = CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))
futimens = CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))
# from /usr/include/i386-linux-gnu/bits/stat.h
UTIME_NOW = ((1l << 30) - 1l)
UTIME_OMIT = ((1l << 30) - 2l)
now = c_timespec(0,UTIME_NOW)
omit = c_timespec(0,UTIME_OMIT)
# wrappers
def update_atime(fileno):
assert(isinstance(fileno, int))
libc.futimens(fileno, byref(c_utimbuf(now, omit)))
def update_mtime(fileno):
assert(isinstance(fileno, int))
libc.futimens(fileno, byref(c_utimbuf(omit, now)))
# usage example:
#
# f = open("/tmp/test")
# update_mtime(f.fileno())

Simplistic:
def touch(fname):
open(fname, 'a').close()
os.utime(fname, None)
The open ensures there is a file there
the utime ensures that the timestamps are updated
Theoretically, it's possible someone will delete the file after the open, causing utime to raise an exception. But arguably that's OK, since something bad did happen.

with open(file_name,'a') as f:
pass

The following is sufficient:
import os
def func(filename):
if os.path.exists(filename):
os.utime(filename)
else:
with open(filename,'a') as f:
pass
If you want to set a specific time for touch, use os.utime as follows:
os.utime(filename,(atime,mtime))
Here, atime and mtime both should be int/float and should be equal to epoch time in seconds to the time which you want to set.

Complex (possibly buggy):
def utime(fname, atime=None, mtime=None)
if type(atime) is tuple:
atime, mtime = atime
if atime is None or mtime is None:
statinfo = os.stat(fname)
if atime is None:
atime = statinfo.st_atime
if mtime is None:
mtime = statinfo.st_mtime
os.utime(fname, (atime, mtime))
def touch(fname, atime=None, mtime=None):
if type(atime) is tuple:
atime, mtime = atime
open(fname, 'a').close()
utime(fname, atime, mtime)
This tries to also allow setting the access or modification time, like GNU touch.

write_text() from pathlib.Path can be used.
>>> from pathlib import Path
>>> Path('aa.txt').write_text("")
0

It might seem logical to create a string with the desired variables, and pass it to os.system:
touch = 'touch ' + dir + '/' + fileName
os.system(touch)
This is inadequate in a number of ways (e.g.,it doesn't handle whitespace), so don't do it.
A more robust method is to use subprocess :
subprocess.call(['touch', os.path.join(dirname, fileName)])
While this is much better than using a subshell (with os.system), it is still only suitable for quick-and-dirty scripts; use the accepted answer for cross-platform programs.

There is also a python module for touch
>>> from touch import touch
>>> touch(file_name)
You can install it with pip install touch

Why don't you try:
newfile.py
#!/usr/bin/env python
import sys
inputfile = sys.argv[1]
with open(inputfile, 'r+') as file:
pass
python newfile.py foobar.txt
or
use subprocess:
import subprocess
subprocess.call(["touch", "barfoo.txt"])

I have a program that I use for backups: https://stromberg.dnsalias.org/~strombrg/backshift/
I profiled it using vmprof, and identified that touch was by far the most time-consuming part of it.
So I looked into ways of touching files quickly.
I found that on CPython 3.11, this was the fastest:
def touch3(filename, flags=os.O_CREAT | os.O_RDWR):
"""Touch a file using os.open+os.close - fastest on CPython 3.11."""
os.close(os.open(filename, flags, 0o644))
And on Pypy3 7.3.9, this was the fastest:
def touch1(filename):
"""Touch a file using pathlib - fastest on pypy3, and fastest overall."""
Path(filename).touch()
Of the two, pypy3's best was only slightly faster cpython's best.
I may create a web page about this someday, but for now all I have is a Subversion repo:
https://stromberg.dnsalias.org/svn/touch/trunk
It includes the 4 ways of doing touches I tried.