How can you create a temporary FIFO (named pipe) in Python? This should work:
import tempfile
temp_file_name = mktemp()
os.mkfifo(temp_file_name)
open(temp_file_name, os.O_WRONLY)
# ... some process, somewhere, will read it ...
However, I'm hesitant because of the big warning in Python Docs 11.6 and potential removal because it's deprecated.
EDIT: It's noteworthy that I've tried tempfile.NamedTemporaryFile (and by extension tempfile.mkstemp), but os.mkfifo throws:
OSError -17: File already exists
when you run it on the files that mkstemp/NamedTemporaryFile have created.
os.mkfifo() will fail with exception OSError: [Errno 17] File exists if the file already exists, so there is no security issue here. The security issue with using tempfile.mktemp() is the race condition where it is possible for an attacker to create a file with the same name before you open it yourself, but since os.mkfifo() fails if the file already exists this is not a problem.
However, since mktemp() is deprecated you shouldn't use it. You can use tempfile.mkdtemp() instead:
import os, tempfile
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'myfifo')
print filename
try:
os.mkfifo(filename)
except OSError, e:
print "Failed to create FIFO: %s" % e
else:
fifo = open(filename, 'w')
# write stuff to fifo
print >> fifo, "hello"
fifo.close()
os.remove(filename)
os.rmdir(tmpdir)
EDIT: I should make it clear that, just because the mktemp() vulnerability is averted by this, there are still the other usual security issues that need to be considered; e.g. an attacker could create the fifo (if they had suitable permissions) before your program did which could cause your program to crash if errors/exceptions are not properly handled.
You may find it handy to use the following context manager, which creates and removes the temporary file for you:
import os
import tempfile
from contextlib import contextmanager
#contextmanager
def temp_fifo():
"""Context Manager for creating named pipes with temporary names."""
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'fifo') # Temporary filename
os.mkfifo(filename) # Create FIFO
try:
yield filename
finally:
os.unlink(filename) # Remove file
os.rmdir(tmpdir) # Remove directory
You can use it, for example, like this:
with temp_fifo() as fifo_file:
# Pass the fifo_file filename e.g. to some other process to read from.
# Write something to the pipe
with open(fifo_file, 'w') as f:
f.write("Hello\n")
How about using
d = mkdtemp()
t = os.path.join(d, 'fifo')
If it's for use within your program, and not with any externals, have a look at the Queue module. As an added benefit, python queues are thread-safe.
Effectively, all that mkstemp does is run mktemp in a loop and keeps attempting to exclusively create until it succeeds (see stdlib source code here). You can do the same with os.mkfifo:
import os, errno, tempfile
def mkftemp(*args, **kwargs):
for attempt in xrange(1024):
tpath = tempfile.mktemp(*args, **kwargs)
try:
os.mkfifo(tpath, 0600)
except OSError as e:
if e.errno == errno.EEXIST:
# lets try again
continue
else:
raise
else:
# NOTE: we only return the path because opening with
# os.open here would block indefinitely since there
# isn't anyone on the other end of the fifo.
return tpath
else:
raise IOError(errno.EEXIST, "No usable temporary file name found")
Why not just use mkstemp()?
For example:
import tempfile
import os
handle, filename = tempfile.mkstemp()
os.mkfifo(filename)
writer = open(filename, os.O_WRONLY)
reader = open(filename, os.O_RDONLY)
os.close(handle)
Related
I'm using the python module subprocess to call a program and redirect the possible std error to a specific file with the following command:
with open("std.err","w") as err:
subprocess.call(["exec"],stderr=err)
I want that the "std.err" file is created only if there are errors, but using the command above if there are no errors the code will create an empty file.
How i can make python create a file only if it's not empty?
I can check after execution if the file is empty and in case remove it, but i was looking for a "cleaner" way.
You could use Popen, checking stderr:
from subprocess import Popen,PIPE
proc = Popen(["EXEC"], stderr=PIPE,stdout=PIPE,universal_newlines=True)
out, err = proc.communicate()
if err:
with open("std.err","w") as f:
f.write(err)
On a side note, if you care about the return code you should use check_call, you could combine it with a NamedTemporaryFile:
from tempfile import NamedTemporaryFile
from os import stat,remove
from shutil import move
try:
with NamedTemporaryFile(dir=".", delete=False) as err:
subprocess.check_call(["exec"], stderr=err)
except (subprocess.CalledProcessError,OSError) as e:
print(e)
if stat(err.name).st_size != 0:
move(err.name,"std.err")
else:
remove(err.name)
You can create your own context manager to handle the cleanup for you -- you can't really do what you're describing here, which boils down to asking how you can see into the future. Something like this (with better error handling, etc.):
import os
from contextlib import contextmanager
#contextmanager
def maybeFile(fileName):
# open the file
f = open(fileName, "w")
# yield the file to be used by the block of code inside the with statement
yield f
# the block is over, do our cleanup.
f.flush()
# if nothing was written, remember that we need to delete the file.
needsCleanup = f.tell() == 0
f.close()
if needsCleanup:
os.remove(fileName)
...and then something like:
with maybeFile("myFileName.txt") as f:
import random
if random.random() < 0.5:
f.write("There should be a file left behind!\n")
will either leave behind a file with a single line of text in it, or will leave nothing behind.
I am trying to run a diff on 2 named temporary files, I did not use difflib because its output was different from the linux diff.
When I run this code, It does not output anything. I tried a diff on regular files and that works just fine.
#using python 2.6
temp_stage = tempfile.NamedTemporaryFile(delete = False)
temp_prod = tempfile.NamedTemporaryFile(delete = False)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)
#this does not work, shows no output, tried both call and popen
subprocess.Popen(["diff", temp_stage.name, temp_prod.name])
#subprocess.call(["diff", temp_stage.name, temp_prod.name])
You need to force the files to be written out to disk by calling flush(); or else the data you were writing to the file may only exist in a buffer.
In fact, if you do this, you can even use delete = True, assuming there's no other reason to keep the files around. This keeps the benefit of using tempfile.
#!/usr/bin/python2
temp_stage = tempfile.NamedTemporaryFile(delete = True)
temp_prod = tempfile.NamedTemporaryFile(delete = True)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)
temp_stage.flush()
temp_prod.flush()
subprocess.Popen(["diff", temp_stage.name, temp_prod.name])
Unrelated to your .flush() issue, you could pass one file via stdin instead of writing data to disk:
from tempfile import NamedTemporaryFile
from subprocess import Popen, PIPE
with NamedTemporaryFile() as file:
file.write(prod_notes)
file.flush()
p = Popen(['diff', '-', file.name], stdin=PIPE)
p.communicate(stage_notes) # diff reads the first file from stdin
if p.returncode == 0:
print('the same')
elif p.returncode == 1:
print('different')
else:
print('error %s' % p.returncode)
diff reads from stdin if input filename is -.
If you use a named pipe then you don't need to write data to disk at all:
from subprocess import Popen, PIPE
from threading import Thread
with named_pipe() as path:
p = Popen(['diff', '-', path], stdin=PIPE)
# use thread, to support content larger than the pipe buffer
Thread(target=p.communicate, args=[stage_notes]).start()
with open(path, 'wb') as pipe:
pipe.write(prod_notes)
if p.wait() == 0:
print('the same')
elif p.returncode == 1:
print('different')
else:
print('error %s' % p.returncode)
where named_pipe() context manager is defined as:
import os
import tempfile
from contextlib import contextmanager
from shutil import rmtree
#contextmanager
def named_pipe(name='named_pipe'):
dirname = tempfile.mkdtemp()
try:
path = os.path.join(dirname, name)
os.mkfifo(path)
yield path
finally:
rmtree(dirname)
The content of a named pipe doesn't touch the disk.
I would suggest bypassing the tempfile handling since with a NTF you're going to have to handle cleanup anyway. Create a new file and write your data then close it. Flush the buffer and then call the subprocess commands. See if that gets it to run.
f=open('file1.blah','w')
f2=open('file2.blah','w')
f.write(stage_notes)
f.flush()
f.close()
f2.write(prod_notes)
f2.flush()
f2.close()
then run your subprocess calls
I know that it is possible to create a temporary file, and write the data of the file I wish to copy to it. I was just wondering if there was a function like:
create_temporary_copy(file_path)
There isn't one directly, but you can use a combination of tempfile and shutil.copy2 to achieve the same result:
import tempfile, shutil, os
def create_temporary_copy(path):
temp_dir = tempfile.gettempdir()
temp_path = os.path.join(temp_dir, 'temp_file_name')
shutil.copy2(path, temp_path)
return temp_path
You'll need to deal with removing the temporary file in the caller, though.
This isn't quite as concise, and I imagine there may be issues with exception safety, (e.g. what happens if 'original_path' doesn't exist, or the temporary_copy object goes out of scope while you have the file open) but this code adds a little RAII to the clean up. The difference here to using NamedTemporaryFile directly is that rather than ending up with a file object, you end up with a file, which is occasionally desirable (e.g. if you plan to call out to other code to read it, or some such.)
import os,shutil,tempfile
class temporary_copy(object):
def __init__(self,original_path):
self.original_path = original_path
def __enter__(self):
temp_dir = tempfile.gettempdir()
base_path = os.path.basename(self.original_path)
self.path = os.path.join(temp_dir,base_path)
shutil.copy2(self.original_path, self.path)
return self.path
def __exit__(self,exc_type, exc_val, exc_tb):
os.remove(self.path)
in your code you'd write:
with temporary_copy(path) as temporary_path_to_copy:
... do stuff with temporary_path_to_copy ...
# Here in the code, the copy should now have been deleted.
The following is more concise (OP's ask) than the selected answer. Enjoy!
import tempfile, shutil, os
def create_temporary_copy(path):
tmp = tempfile.NamedTemporaryFile(delete=True)
shutil.copy2(path, tmp.name)
return tmp.name
A variation on #tramdas's answer, accounting for the fact that the file cannot be opened twice on windows. This version ignores the preservation of the file extension.
import os, shutil, tempfile
def create_temporary_copy(src):
# create the temporary file in read/write mode (r+)
tf = tempfile.TemporaryFile(mode='r+b', prefix='__', suffix='.tmp')
# on windows, we can't open the the file again, either manually
# or indirectly via shutil.copy2, but we *can* copy
# the file directly using file-like objects, which is what
# TemporaryFile returns to us.
# Use `with open` here to automatically close the source file
with open(src,'r+b') as f:
shutil.copyfileobj(f,tf)
# display the name of the temporary file for diagnostic purposes
print 'temp file:',tf.name
# rewind the temporary file, otherwise things will go
# tragically wrong on Windows
tf.seek(0)
return tf
# make a temporary copy of the file 'foo.txt'
name = None
with create_temporary_copy('foo.txt') as temp:
name = temp.name
# prove that it exists
print 'exists', os.path.isfile(name) # prints True
# read all lines from the file
i = 0
for line in temp:
print i,line.strip()
i += 1
# temp.close() is implicit using `with`
# prove that it has been deleted
print 'exists', os.path.isfile(name) # prints False
A slight variation (in particular I needed the preserve_extension feature for my use case, and I like the "self-cleanup" feature):
import os, shutil, tempfile
def create_temporary_copy(src_file_name, preserve_extension=False):
'''
Copies the source file into a temporary file.
Returns a _TemporaryFileWrapper, whose destructor deletes the temp file
(i.e. the temp file is deleted when the object goes out of scope).
'''
tf_suffix=''
if preserve_extension:
_, tf_suffix = os.path.splitext(src_file_name)
tf = tempfile.NamedTemporaryFile(suffix=tf_suffix)
shutil.copy2(src_file_name, tf.name)
return tf
I want to use tempfile.NamedTemporaryFile() to write some contents into it and then open that file. I have written following code:
tf = tempfile.NamedTemporaryFile()
tfName = tf.name
tf.seek(0)
tf.write(contents)
tf.flush()
but I am unable to open this file and see its contents in Notepad or similar application. Is there any way to achieve this? Why can't I do something like:
os.system('start notepad.exe ' + tfName)
at the end.
I don't want to save the file permanently on my system. I just want the contents to be opened as a text in Notepad or similar application and delete the file when I close that application.
This could be one of two reasons:
Firstly, by default the temporary file is deleted as soon as it is closed. To fix this use:
tf = tempfile.NamedTemporaryFile(delete=False)
and then delete the file manually once you've finished viewing it in the other application.
Alternatively, it could be that because the file is still open in Python Windows won't let you open it using another application.
Edit: to answer some questions from the comments:
As of the docs from 2 when using delete=False the file can be removed by using:
tf.close()
os.unlink(tf.name)
You can also use it with a context manager so that the file will be closed/deleted when it goes out of scope. It will also be cleaned up if the code in the context manager raises.
import tempfile
with tempfile.NamedTemporaryFile() as temp:
temp.write('Some data')
temp.flush()
# do something interesting with temp before it is destroyed
Here is a useful context manager for this.
(In my opinion, this functionality should be part of the Python standard library.)
# python2 or python3
import contextlib
import os
#contextlib.contextmanager
def temporary_filename(suffix=None):
"""Context that introduces a temporary file.
Creates a temporary file, yields its name, and upon context exit, deletes it.
(In contrast, tempfile.NamedTemporaryFile() provides a 'file' object and
deletes the file as soon as that file object is closed, so the temporary file
cannot be safely re-opened by another library or process.)
Args:
suffix: desired filename extension (e.g. '.mp4').
Yields:
The name of the temporary file.
"""
import tempfile
try:
f = tempfile.NamedTemporaryFile(suffix=suffix, delete=False)
tmp_name = f.name
f.close()
yield tmp_name
finally:
os.unlink(tmp_name)
# Example:
with temporary_filename() as filename:
os.system('echo Hello >' + filename)
assert 6 <= os.path.getsize(filename) <= 8 # depending on text EOL
assert not os.path.exists(filename)
touch is a Unix utility that sets the modification and access times of files to the current time of day. If the file doesn't exist, it is created with default permissions.
How would you implement it as a Python function? Try to be cross platform and complete.
(Current Google results for "python touch file" are not that great, but point to os.utime.)
Looks like this is new as of Python 3.4 - pathlib.
from pathlib import Path
Path('path/to/file.txt').touch()
This will create a file.txt at the path.
--
Path.touch(mode=0o777, exist_ok=True)
Create a file at this given path. If mode is given, it is combined with the process’ umask value to determine the file mode and access flags. If the file already exists, the function succeeds if exist_ok is true (and its modification time is updated to the current time), otherwise FileExistsError is raised.
This tries to be a little more race-free than the other solutions. (The with keyword is new in Python 2.5.)
import os
def touch(fname, times=None):
with open(fname, 'a'):
os.utime(fname, times)
Roughly equivalent to this.
import os
def touch(fname, times=None):
fhandle = open(fname, 'a')
try:
os.utime(fname, times)
finally:
fhandle.close()
Now, to really make it race-free, you need to use futimes and change the timestamp of the open filehandle, instead of opening the file and then changing the timestamp on the filename (which may have been renamed). Unfortunately, Python doesn't seem to provide a way to call futimes without going through ctypes or similar...
EDIT
As noted by Nate Parsons, Python 3.3 will add specifying a file descriptor (when os.supports_fd) to functions such as os.utime, which will use the futimes syscall instead of the utimes syscall under the hood. In other words:
import os
def touch(fname, mode=0o666, dir_fd=None, **kwargs):
flags = os.O_CREAT | os.O_APPEND
with os.fdopen(os.open(fname, flags=flags, mode=mode, dir_fd=dir_fd)) as f:
os.utime(f.fileno() if os.utime in os.supports_fd else fname,
dir_fd=None if os.supports_fd else dir_fd, **kwargs)
def touch(fname):
if os.path.exists(fname):
os.utime(fname, None)
else:
open(fname, 'a').close()
Why not try this?:
import os
def touch(fname):
try:
os.utime(fname, None)
except OSError:
open(fname, 'a').close()
I believe this eliminates any race condition that matters. If the file does not exist then an exception will be thrown.
The only possible race condition here is if the file is created before open() is called but after os.utime(). But this does not matter because in this case the modification time will be as expected since it must have happened during the call to touch().
For a more low-level solution one can use
os.close(os.open("file.txt", os.O_CREAT))
This answer is compatible with all versions since Python-2.5 when keyword with has been released.
1. Create file if does not exist + Set current time
(exactly same as command touch)
import os
fname = 'directory/filename.txt'
with open(fname, 'a'): # Create file if does not exist
os.utime(fname, None) # Set access/modified times to now
# May raise OSError if file does not exist
A more robust version:
import os
with open(fname, 'a'):
try: # Whatever if file was already existing
os.utime(fname, None) # => Set current time anyway
except OSError:
pass # File deleted between open() and os.utime() calls
2. Just create the file if does not exist
(does not update time)
with open(fname, 'a'): # Create file if does not exist
pass
3. Just update file access/modified times
(does not create file if not existing)
import os
try:
os.utime(fname, None) # Set access/modified times to now
except OSError:
pass # File does not exist (or no permission)
Using os.path.exists() does not simplify the code:
from __future__ import (absolute_import, division, print_function)
import os
if os.path.exists(fname):
try:
os.utime(fname, None) # Set access/modified times to now
except OSError:
pass # File deleted between exists() and utime() calls
# (or no permission)
Bonus: Update time of all files in a directory
from __future__ import (absolute_import, division, print_function)
import os
number_of_files = 0
# Current directory which is "walked through"
# | Directories in root
# | | Files in root Working directory
# | | | |
for root, _, filenames in os.walk('.'):
for fname in filenames:
pathname = os.path.join(root, fname)
try:
os.utime(pathname, None) # Set access/modified times to now
number_of_files += 1
except OSError as why:
print('Cannot change time of %r because %r', pathname, why)
print('Changed time of %i files', number_of_files)
Here's some code that uses ctypes (only tested on Linux):
from ctypes import *
libc = CDLL("libc.so.6")
# struct timespec {
# time_t tv_sec; /* seconds */
# long tv_nsec; /* nanoseconds */
# };
# int futimens(int fd, const struct timespec times[2]);
class c_timespec(Structure):
_fields_ = [('tv_sec', c_long), ('tv_nsec', c_long)]
class c_utimbuf(Structure):
_fields_ = [('atime', c_timespec), ('mtime', c_timespec)]
utimens = CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))
futimens = CFUNCTYPE(c_int, c_char_p, POINTER(c_utimbuf))
# from /usr/include/i386-linux-gnu/bits/stat.h
UTIME_NOW = ((1l << 30) - 1l)
UTIME_OMIT = ((1l << 30) - 2l)
now = c_timespec(0,UTIME_NOW)
omit = c_timespec(0,UTIME_OMIT)
# wrappers
def update_atime(fileno):
assert(isinstance(fileno, int))
libc.futimens(fileno, byref(c_utimbuf(now, omit)))
def update_mtime(fileno):
assert(isinstance(fileno, int))
libc.futimens(fileno, byref(c_utimbuf(omit, now)))
# usage example:
#
# f = open("/tmp/test")
# update_mtime(f.fileno())
Simplistic:
def touch(fname):
open(fname, 'a').close()
os.utime(fname, None)
The open ensures there is a file there
the utime ensures that the timestamps are updated
Theoretically, it's possible someone will delete the file after the open, causing utime to raise an exception. But arguably that's OK, since something bad did happen.
with open(file_name,'a') as f:
pass
The following is sufficient:
import os
def func(filename):
if os.path.exists(filename):
os.utime(filename)
else:
with open(filename,'a') as f:
pass
If you want to set a specific time for touch, use os.utime as follows:
os.utime(filename,(atime,mtime))
Here, atime and mtime both should be int/float and should be equal to epoch time in seconds to the time which you want to set.
Complex (possibly buggy):
def utime(fname, atime=None, mtime=None)
if type(atime) is tuple:
atime, mtime = atime
if atime is None or mtime is None:
statinfo = os.stat(fname)
if atime is None:
atime = statinfo.st_atime
if mtime is None:
mtime = statinfo.st_mtime
os.utime(fname, (atime, mtime))
def touch(fname, atime=None, mtime=None):
if type(atime) is tuple:
atime, mtime = atime
open(fname, 'a').close()
utime(fname, atime, mtime)
This tries to also allow setting the access or modification time, like GNU touch.
write_text() from pathlib.Path can be used.
>>> from pathlib import Path
>>> Path('aa.txt').write_text("")
0
It might seem logical to create a string with the desired variables, and pass it to os.system:
touch = 'touch ' + dir + '/' + fileName
os.system(touch)
This is inadequate in a number of ways (e.g.,it doesn't handle whitespace), so don't do it.
A more robust method is to use subprocess :
subprocess.call(['touch', os.path.join(dirname, fileName)])
While this is much better than using a subshell (with os.system), it is still only suitable for quick-and-dirty scripts; use the accepted answer for cross-platform programs.
There is also a python module for touch
>>> from touch import touch
>>> touch(file_name)
You can install it with pip install touch
Why don't you try:
newfile.py
#!/usr/bin/env python
import sys
inputfile = sys.argv[1]
with open(inputfile, 'r+') as file:
pass
python newfile.py foobar.txt
or
use subprocess:
import subprocess
subprocess.call(["touch", "barfoo.txt"])
I have a program that I use for backups: https://stromberg.dnsalias.org/~strombrg/backshift/
I profiled it using vmprof, and identified that touch was by far the most time-consuming part of it.
So I looked into ways of touching files quickly.
I found that on CPython 3.11, this was the fastest:
def touch3(filename, flags=os.O_CREAT | os.O_RDWR):
"""Touch a file using os.open+os.close - fastest on CPython 3.11."""
os.close(os.open(filename, flags, 0o644))
And on Pypy3 7.3.9, this was the fastest:
def touch1(filename):
"""Touch a file using pathlib - fastest on pypy3, and fastest overall."""
Path(filename).touch()
Of the two, pypy3's best was only slightly faster cpython's best.
I may create a web page about this someday, but for now all I have is a Subversion repo:
https://stromberg.dnsalias.org/svn/touch/trunk
It includes the 4 ways of doing touches I tried.