I'm using the python module subprocess to call a program and redirect the possible std error to a specific file with the following command:
with open("std.err","w") as err:
subprocess.call(["exec"],stderr=err)
I want that the "std.err" file is created only if there are errors, but using the command above if there are no errors the code will create an empty file.
How i can make python create a file only if it's not empty?
I can check after execution if the file is empty and in case remove it, but i was looking for a "cleaner" way.
You could use Popen, checking stderr:
from subprocess import Popen,PIPE
proc = Popen(["EXEC"], stderr=PIPE,stdout=PIPE,universal_newlines=True)
out, err = proc.communicate()
if err:
with open("std.err","w") as f:
f.write(err)
On a side note, if you care about the return code you should use check_call, you could combine it with a NamedTemporaryFile:
from tempfile import NamedTemporaryFile
from os import stat,remove
from shutil import move
try:
with NamedTemporaryFile(dir=".", delete=False) as err:
subprocess.check_call(["exec"], stderr=err)
except (subprocess.CalledProcessError,OSError) as e:
print(e)
if stat(err.name).st_size != 0:
move(err.name,"std.err")
else:
remove(err.name)
You can create your own context manager to handle the cleanup for you -- you can't really do what you're describing here, which boils down to asking how you can see into the future. Something like this (with better error handling, etc.):
import os
from contextlib import contextmanager
#contextmanager
def maybeFile(fileName):
# open the file
f = open(fileName, "w")
# yield the file to be used by the block of code inside the with statement
yield f
# the block is over, do our cleanup.
f.flush()
# if nothing was written, remember that we need to delete the file.
needsCleanup = f.tell() == 0
f.close()
if needsCleanup:
os.remove(fileName)
...and then something like:
with maybeFile("myFileName.txt") as f:
import random
if random.random() < 0.5:
f.write("There should be a file left behind!\n")
will either leave behind a file with a single line of text in it, or will leave nothing behind.
Related
Is there a way in Python using gzip or other module to check the integrity of the gzip archive?
Basically is there equivalent in Python to what the following does:
gunzip -t my_archive.gz
Oops, first answer (now deleted) was result of misreading the question.
I'd suggest using the gzip module to read the file and just throw away what you read. You have to decode the entire file in order to check its integrity in any case. https://docs.python.org/2/library/gzip.html
Something like ( Untested code)
import gzip
chunksize=10000000 # 10 Mbytes
ok = True
with gzip.open('file.txt.gz', 'rb') as f:
try:
while f.read(chunksize) != b'':
pass
except:
ok = False
I don't know what exception reading a corrupt zipfile will throw, you might want to find out and then catch only this particular one.
you can use subprocess or os module to execute this command and read the output. something like this
Using os module
import os
output = os.popen('gunzip -t my_archive.gz').read()
Using Subprocess Module
import subprocess
proc = subprocess.Popen(["gunzip", "-t", "my_archive.gz"], stdout=subprocess.PIPE, shell=True)
(out, err) = proc.communicate()
I am trying to open a file and read from it and if the file is not there, I catch the exception and throw an error to stderr. The code that I have:
for x in l:
try:
f = open(x,'r')
except IOError:
print >> sys.stderr, "No such file" , x
but nothing is being printed to stderr, does open create a new file if the file name doesn't exist or is the problem somewhere else?
Try this:
from __future__ import print_statement
import sys
if os.path.exists(x):
with open(x, 'r') as f:
# Do Stuff with file
else:
print("No such file '{}'".format(x), file=sys.stderr)
The goal here is to be as clear as possible about what is happening. We first check if the file exists by calling os.path.exists(x). This returns True or False, allowing us to simply use it in an if statement.
From there you can open the file for reading, or handle exiting as you like. Using the Python3 style print function allows you to explicitly declare where your output goes, in this case to stderr.
You have the os.path.exists function:
import os.path
os.path.exists(file_path)
returns bool
It works for me. Why can't you make use of os.path.exists()
for x in l:
if not os.path.exists(x):
print >> sys.stderr , "No such file", x
I am trying to run a diff on 2 named temporary files, I did not use difflib because its output was different from the linux diff.
When I run this code, It does not output anything. I tried a diff on regular files and that works just fine.
#using python 2.6
temp_stage = tempfile.NamedTemporaryFile(delete = False)
temp_prod = tempfile.NamedTemporaryFile(delete = False)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)
#this does not work, shows no output, tried both call and popen
subprocess.Popen(["diff", temp_stage.name, temp_prod.name])
#subprocess.call(["diff", temp_stage.name, temp_prod.name])
You need to force the files to be written out to disk by calling flush(); or else the data you were writing to the file may only exist in a buffer.
In fact, if you do this, you can even use delete = True, assuming there's no other reason to keep the files around. This keeps the benefit of using tempfile.
#!/usr/bin/python2
temp_stage = tempfile.NamedTemporaryFile(delete = True)
temp_prod = tempfile.NamedTemporaryFile(delete = True)
temp_stage.write(stage_notes)
temp_prod.write(prod_notes)
temp_stage.flush()
temp_prod.flush()
subprocess.Popen(["diff", temp_stage.name, temp_prod.name])
Unrelated to your .flush() issue, you could pass one file via stdin instead of writing data to disk:
from tempfile import NamedTemporaryFile
from subprocess import Popen, PIPE
with NamedTemporaryFile() as file:
file.write(prod_notes)
file.flush()
p = Popen(['diff', '-', file.name], stdin=PIPE)
p.communicate(stage_notes) # diff reads the first file from stdin
if p.returncode == 0:
print('the same')
elif p.returncode == 1:
print('different')
else:
print('error %s' % p.returncode)
diff reads from stdin if input filename is -.
If you use a named pipe then you don't need to write data to disk at all:
from subprocess import Popen, PIPE
from threading import Thread
with named_pipe() as path:
p = Popen(['diff', '-', path], stdin=PIPE)
# use thread, to support content larger than the pipe buffer
Thread(target=p.communicate, args=[stage_notes]).start()
with open(path, 'wb') as pipe:
pipe.write(prod_notes)
if p.wait() == 0:
print('the same')
elif p.returncode == 1:
print('different')
else:
print('error %s' % p.returncode)
where named_pipe() context manager is defined as:
import os
import tempfile
from contextlib import contextmanager
from shutil import rmtree
#contextmanager
def named_pipe(name='named_pipe'):
dirname = tempfile.mkdtemp()
try:
path = os.path.join(dirname, name)
os.mkfifo(path)
yield path
finally:
rmtree(dirname)
The content of a named pipe doesn't touch the disk.
I would suggest bypassing the tempfile handling since with a NTF you're going to have to handle cleanup anyway. Create a new file and write your data then close it. Flush the buffer and then call the subprocess commands. See if that gets it to run.
f=open('file1.blah','w')
f2=open('file2.blah','w')
f.write(stage_notes)
f.flush()
f.close()
f2.write(prod_notes)
f2.flush()
f2.close()
then run your subprocess calls
Well, I have made a module that allows you to copy a file to a directory easier. Now, I also have some "try's" and "except's" in there to make sure it doesn't fail in the big messy way and doesn't close the terminal, but I also want it to display different error messages when a wrong string or variable is put in, and end the module, but not the...if I may say, Terminal running it, so I did this:
def copy():
import shutil
import os
try:
cpy = input("CMD>>> Name of file(with extension): ")
open(cpy, "r")
except:
print("ERROR>>> 02x00 No such file")
try:
dri = input("CMD>>> Name of Directory: ")
os.chdir(dri)
os.chdir("..")
except:
print("ERROR>>> 03x00 No such directory")
try:
shutil.copy(cpy, dri)
except:
print("ERROR>>> 04x00 Command Failure")
Problem is that it doesn't end the module if there is no file or directory, only at the finish.
You may be thinking that when an exception is raised, Python just stops what it's doing, but that's not quite true. The except: block actually catches the exception raised, and is supposed to handle it. After an except: block finishes, Python will continue on executing the rest of the code in the file.
In your case, I'd put a return after each print(...). That way, after Python prints out an error message, it will also return from the copy() function rather than continuing to ask for more input.
If you did want to make the module exit on error...
Here's how you'd do it.
def copy():
import shutil
import os
import sys
try:
cpy = input("CMD>>> Name of file(with extension): ")
open(cpy, "r")
except:
sys.exit("ERROR>>> 02x00 No such file")
try:
dri = input("CMD>>> Name of Directory: ")
os.chdir(dri)
os.chdir("..")
except:
sys.exit("ERROR>>> 03x00 No such directory")
try:
shutil.copy(cpy, dri)
except:
sys.exit("ERROR>>> 04x00 Command Failure")
sys.exit(0) (for success) and sys.exit(1) (for failure) are usually used but, since you want to output the error, the above example will output the error string to stderr.
Here's a link for more info on sys.exit().
How can you create a temporary FIFO (named pipe) in Python? This should work:
import tempfile
temp_file_name = mktemp()
os.mkfifo(temp_file_name)
open(temp_file_name, os.O_WRONLY)
# ... some process, somewhere, will read it ...
However, I'm hesitant because of the big warning in Python Docs 11.6 and potential removal because it's deprecated.
EDIT: It's noteworthy that I've tried tempfile.NamedTemporaryFile (and by extension tempfile.mkstemp), but os.mkfifo throws:
OSError -17: File already exists
when you run it on the files that mkstemp/NamedTemporaryFile have created.
os.mkfifo() will fail with exception OSError: [Errno 17] File exists if the file already exists, so there is no security issue here. The security issue with using tempfile.mktemp() is the race condition where it is possible for an attacker to create a file with the same name before you open it yourself, but since os.mkfifo() fails if the file already exists this is not a problem.
However, since mktemp() is deprecated you shouldn't use it. You can use tempfile.mkdtemp() instead:
import os, tempfile
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'myfifo')
print filename
try:
os.mkfifo(filename)
except OSError, e:
print "Failed to create FIFO: %s" % e
else:
fifo = open(filename, 'w')
# write stuff to fifo
print >> fifo, "hello"
fifo.close()
os.remove(filename)
os.rmdir(tmpdir)
EDIT: I should make it clear that, just because the mktemp() vulnerability is averted by this, there are still the other usual security issues that need to be considered; e.g. an attacker could create the fifo (if they had suitable permissions) before your program did which could cause your program to crash if errors/exceptions are not properly handled.
You may find it handy to use the following context manager, which creates and removes the temporary file for you:
import os
import tempfile
from contextlib import contextmanager
#contextmanager
def temp_fifo():
"""Context Manager for creating named pipes with temporary names."""
tmpdir = tempfile.mkdtemp()
filename = os.path.join(tmpdir, 'fifo') # Temporary filename
os.mkfifo(filename) # Create FIFO
try:
yield filename
finally:
os.unlink(filename) # Remove file
os.rmdir(tmpdir) # Remove directory
You can use it, for example, like this:
with temp_fifo() as fifo_file:
# Pass the fifo_file filename e.g. to some other process to read from.
# Write something to the pipe
with open(fifo_file, 'w') as f:
f.write("Hello\n")
How about using
d = mkdtemp()
t = os.path.join(d, 'fifo')
If it's for use within your program, and not with any externals, have a look at the Queue module. As an added benefit, python queues are thread-safe.
Effectively, all that mkstemp does is run mktemp in a loop and keeps attempting to exclusively create until it succeeds (see stdlib source code here). You can do the same with os.mkfifo:
import os, errno, tempfile
def mkftemp(*args, **kwargs):
for attempt in xrange(1024):
tpath = tempfile.mktemp(*args, **kwargs)
try:
os.mkfifo(tpath, 0600)
except OSError as e:
if e.errno == errno.EEXIST:
# lets try again
continue
else:
raise
else:
# NOTE: we only return the path because opening with
# os.open here would block indefinitely since there
# isn't anyone on the other end of the fifo.
return tpath
else:
raise IOError(errno.EEXIST, "No usable temporary file name found")
Why not just use mkstemp()?
For example:
import tempfile
import os
handle, filename = tempfile.mkstemp()
os.mkfifo(filename)
writer = open(filename, os.O_WRONLY)
reader = open(filename, os.O_RDONLY)
os.close(handle)