We Keep Coding

python try...except in handling argument from user input

my python file would read two files from command line using argparse.
I want to use 'try...except...' to handle incorrect user input, like non-exist file and incorrect command:
for example, I use '-f' to open file, then '-g' is incorrect command.
my codes are:
> parser = argparse.ArgumentParser(description = "Handle with input
files")
parser.add_argument('-m', dest = "model", type=argparse.FileType('r'))
parser.add_argument('-t', dest = "test", type=argparse.FileType('r'))
parser.add_argument('-d', dest = "permute",type=argparse.FileType('r'))
args = parser.parse_args()
# Parse file names from command line
model_file = open(args.model.name)
test_file = open(args.test.name)
decoy_file = open(args.permute.name)
I try to implement 'try...except' in my code but failed. Here is what I did:
> parser = argparse.ArgumentParser(description = "Handle with input
files")
parser.add_argument('-m', dest = "model", type=argparse.FileType('r'))
parser.add_argument('-t', dest = "test", type=argparse.FileType('r'))
parser.add_argument('-d', dest = "permute",type=argparse.FileType('r'))
args = parser.parse_args()
# Parse file names from command line
try:
model_file = open(args.model.name)
test_file = open(args.test.name)
decoy_file = open(args.permute.name)
except IOError:
print('Wrong file name')

I would suggest adding something like this to make sure all arguments are there. Also argparse.FileType('r') will already open the file (meaning you can do something like args.model.readlines()).
if not all([args.model, args.test, args.permute]):
print("All Arguments are required")
exit(1)

Firstly the error happend at args = parser.parse_args() when it parse your command param.
As we can see from argparse's origin code:
...
def parse_args(self, args=None, namespace=None):
args, argv = self.parse_known_args(args, namespace)
if argv:
msg = _('unrecognized arguments: %s')
self.error(msg % ' '.join(argv))
return args
...
def error(self, message):
"""error(message: string)
Prints a usage message incorporating the message to stderr and
exits.
If you override this in a subclass, it should not return -- it
should either exit or raise an exception.
"""
self.print_usage(_sys.stderr)
args = {'prog': self.prog, 'message': message}
self.exit(2, _('%(prog)s: error: %(message)s\n') % args)
# exit here ,that's why can not catch the exception
So when you input incorrectly, like what you mentioned above, like non-exist file or incorrect command , it will call error function in the end, and exit the thread, that's why you cannot catch a exception by try .. except synax.
If you want to catch the exception, you can inherit the main class ArgumentParser, and rewrite the method error. you can take my codes as reference:
import argparse
import sys as _sys
from argparse import ArgumentParser
class ArgumentParserSub(ArgumentParser):
def error(self, message):
self.print_usage(_sys.stderr)
args = {'prog': self.prog, 'message': message}
raise Exception(('error: %(message)s\n') % args)
parser = ArgumentParserSub(description = "Handle with input files")
parser.add_argument('-m', dest = "model", type=argparse.FileType('r'))
parser.add_argument('-t', dest = "test", type=argparse.FileType('r'))
parser.add_argument('-d', dest = "permute",type=argparse.FileType('r'))
try:
args = parser.parse_args()
# Parse file names from command line
model_file = open(args.model.name)
test_file = open(args.test.name)
decoy_file = open(args.permute.name)
except Exception as e:
print(e)
print('Wrong file name')

If you don't want the parser to catch and exit bad files, open the files yourself. Just ask the parser for names, not open files.
parser = argparse.ArgumentParser(description = "Handle with input files")
parser.add_argument('-m', dest = "model") # no type
parser.add_argument('-t', dest = "test")
parser.add_argument('-d', dest = "permute")
args = parser.parse_args()
# print(args) # good idea when debugging
# Open file names from command line
try:
model_file = open(args.model)
test_file = open(args.test)
decoy_file = open(args.permute)
except IOError:
print('Wrong file name')

python 2.6 linecache.getline() and stdin. How does it work?

I have a script which runs through lines of input to find the occurrence of an ID string while keeping track of the linenumber.
Then it runs backwards up the input to trace parentID/childID relationships. The script accepts either a logfile using a '-f' flag as an argument or the contents of stdin from a pipe.
The logfile as input portion works just fine, but reading from stdin seems not to work.
For the sake of reasonable clarity I've included the portion of the script that this concerns, but don't expect to be able to run it. It's just to show you sorta whats going on (anyone who works in financial services around FIX protocol would recognize a few things):
import os
import sys
import linecache
from types import *
from ____ import FixMessage # custom message class that is used throughout
# Feel free to ignore all the getArgs and validation crap
def getArgs():
import argparse
parser = argparse.ArgumentParser(
description='Get amendment history.')
parser.add_argument('-f', '--file',
help="input logfile.'")
args = parser.parse_args()
return validateArgs(args)
def validateArgs(args):
try:
if sys.stdin.isatty():
if args.file:
assert os.path.isfile(args.file.strip('\n')), \
'File "{0}" does not exist'.format(args.file)
args.file = open(args.file, 'r')
else:
args.file = sys.stdin
assert args.file, \
"Please either include a file with '-f' or pipe some text in"
except AssertionError as err:
print err
exit(1)
return args
defGetMessageTrail(logfile, orderId):
# some input validation
if isinstance(logfile, StringType):
try: logfile = open(logfile, 'r')
except IOError as err: exit(1)
elif not isinstance(logfile, FileType):
raise TypeError(
'Expected FileType and got {0}'.format(type(logfile)))
linenum = 0
# This retrieves the message containing the orderID as well as the linenum
for line in logfile:
linenum += 1
if orderId in line:
# FixMessage is a custom class that is treated here like
# a dictionary with some metadata
# Missing dict keys return 'None'
# .isvalid is bool results of some text validation
# .direction is either incoming or outgoing
# thats all you really need to know
msg = FixMessage(line)
if msg.isvalid and msg.direction == 'Incoming':
yield msg
break
# If there is a message parentID, it would be in msg['41']
if msg['41']:
messages = findParentMessages(logfile, startline=linenum, msg['41'])
for msg in messages: yield msg
def findParentMessages(logfile, startline, targetId):
# Some more input validation
assert isinstance(logfile, FileType)
assert isinstance(startline, IntType)
assert isinstance(targetId, StringType)
# should just make a integer decrementing generator,
# but this is fine for the example
for linenum in range(startline)[::-1]:
# *** This is where the question lies... ***
# print(logfile.name) # returns "<stdin>"
line = linecache.getline(logfile.name, linenum)
if 'Incoming' in line and '11=' + targetId in line:
msg = FixMessage(line)
yield msg
if msg['41']: findParentMessages(logfile, linenum, msg['41'])
else: break
def main():
log = getArgs().file
trail = getMessageTrail(log, 'ORDER123')
if __name__ == '__main__': main()
The question is, how does linecache.getline work when it comes to reading stdin as a file? is it different than how it would work if given a regular filename?

linecache.getline() accepts a file name, not a file object. It is not designed to work that way as filename is passed to calls like open() and os.stat().
For reference: https://github.com/python/cpython/blob/2.6/Lib/linecache.py

Error in def_main(), excecute takes 1 positional argument, but 2 was given

import shlex
import os
os.environ['SERVER_SOFTWARE'] = 'Dev'
import subprocess, socket, time, struct
def execute_command(cmd):
f = open(log.name, "w")
try:
subprocess.Popen(cmd, 0, f, f)
except WindowsError:
cmd[0] = cmd[0] + ".com"
subprocess.Popen(cmd, 0, f, f) #work-around
rc = process.wait()
if rc != 0:
print("Error: failed to execute command:", cmd)
print(error)
return result
def main():
execute_command(["tree", "C:\\"], "treelog.txt")
return

The error message is quite clear, and I'm suprised you need any explanation: you defined a function execute_command(cmd) that takes a single argument (cmd), then call it with two arguments.
There are quite a few other problems in your code but since you didn't ever bothered formatting your code and writing a real question (copy/pasting par of an error message as title doesn't count as "a real question"), I won't bother pointing them either.

strace a python function

Is it possible to strace a python function for opened files, and differentiate if they were opened by python or a subprocess?
read_python, read_external = [], []
#strace_read(read_python, read_external)
function test():
file = open("foo.txt", "r")
subprocess.call(["cat", "bar.txt"])
for file in read_python:
print("python: ", file)
for file in read_external:
print("external: ", file)
So the output is as:
>>> python: foo.txt
>>> external: bar.txt
I'm most interested in using a decorator. Differentiating isn't a priority.
Conceptually, my best guess is to replace instances of load_function(open) with wrappers ... actually, I have no idea, there are too many ways to access open.

I'd solve it in a much simpler way but with similar result. Instead of figuring out how to enable strace on a single function:
Create decorator like this: (untested)
-
def strace_mark(f):
def wrapper(*args, **kwargs):
try:
open('function-%s-start' % f.__name__, 'r')
except:
pass
ret = f(*args, **kwargs)
try:
open('function-%s-end' % f.__name__, 'r')
except:
pass
return ret
Run the whole app under strace -e file.
Get only the parts between calls open(function-something-start) and open(function-something-end).
If you do strace -f, you get the python/external separation for free. Just look at what pid calls the function.

This is the solution I used:
#!/usr/bin/env python3
import multiprocessing
import selectors
import os
import array
import fcntl
import termios
import subprocess
import decorator
import locale
import io
import codecs
import re
import collections
def strace(function):
StraceReturn = collections.namedtuple("StraceReturn", ["return_data", "pid", "strace_data"])
def strace_filter(stracefile, pid, exclude_system=False):
system = ( "/bin"
, "/boot"
, "/dev"
, "/etc"
, "/lib"
, "/proc"
, "/root"
, "/run"
, "/sbin"
, "/srv"
, "/sys"
, "/tmp"
, "/usr"
, "/var"
)
encoding = locale.getpreferredencoding(False)
for line in stracefile:
match = re.search(r'^(?:\[pid\s+(\d+)\]\s+)?open\(\"((?:\\x[0-9a-f]{2})+)\",', line, re.IGNORECASE)
if match:
p, f = match.groups(pid)
f = codecs.escape_decode(f.encode("ascii"))[0].decode(encoding)
if exclude_system and f.startswith(system):
continue
yield (p, f)
def strace_reader(conn_parent, conn_child, barrier, pid):
conn_parent.close()
encoding = locale.getpreferredencoding(False)
strace_args = ["strace", "-e", "open", "-f", "-s", "512", "-xx", "-p", str(pid)]
process_data = io.StringIO()
process = subprocess.Popen\
( strace_args
, stdout = subprocess.DEVNULL
, stderr = subprocess.PIPE
, universal_newlines = True
)
selector = selectors.DefaultSelector()
selector.register(process.stderr, selectors.EVENT_READ)
selector.select()
barrier.wait()
selector.register(conn_child, selectors.EVENT_READ)
while len(selector.get_map()):
events = selector.select()
for key, mask in events:
if key.fd == conn_child.fileno():
conn_child.recv()
selector.unregister(key.fd)
process.terminate()
try:
process.wait(5)
except TimeoutError:
process.kill()
process.wait()
else:
ioctl_buffer = array.array("i", [0])
try:
fcntl.ioctl(key.fd, termios.FIONREAD, ioctl_buffer)
except OSError:
read_bytes = 1024
else:
read_bytes = max(1024, ioctl_buffer[0])
data = os.read(key.fd, read_bytes)
if data:
# store all data, simpler but not as memory-efficient
# as:
# result, leftover_line = strace_filter\
# ( leftover_line + data.decode(encoding)
# , pid
# )
# process_data.append(result)
# with, after this loop, a final:
# result = strace_filter(leftover_line + "\n", pid)
# process_data.append(result)
process_data.write(data.decode(encoding))
else:
selector.unregister(key.fd)
selector.close()
process_data.seek(0, io.SEEK_SET)
for pidfile in strace_filter(process_data, pid):
conn_child.send(pidfile)
conn_child.close()
def strace_wrapper(function, *args, **kw):
strace_data = list()
barrier = multiprocessing.Barrier(2)
conn_parent, conn_child = multiprocessing.Pipe(duplex = True)
process = multiprocessing.Process\
( target=strace_reader
, args=(conn_parent, conn_child, barrier, os.getpid())
)
process.start()
conn_child.close()
barrier.wait()
function_return = function()
conn_parent.send(None)
while True:
try:
strace_data.append(conn_parent.recv())
except EOFError:
break
process.join(5)
if process.is_alive():
process.terminate()
process.join(5)
if process.is_alive():
os.kill(process.pid, signal.SIGKILL)
process.join()
conn_parent.close()
return StraceReturn(function_return, os.getpid(), strace_data)
return decorator.decorator(strace_wrapper, function)
#strace
def test():
print("Entering test()")
process = subprocess.Popen("cat +μυρτιὲς.txt", shell=True)
f = open("test\"test", "r")
f.close()
process.wait()
print("Exiting test()")
return 5
print(test())
Note that any information strace generates after the termination event will be collected. To avoid that, use a while not signaled loop, and terminate the subprocess after the loop (the FIONREAD ioctl is a holdover from this case; I didn't see any reason to remove it).
In hindsight, the decorator could have been greatly simplified had I used a temporary file, rather than multiprocessing/pipe.
A child process is forked to then fork strace - in other words, strace is tracing its grandparent. Some linux distributions only allow strace to trace its children. I'm not sure how to work around this restriction - having the main program continue executing in the child fork (while the parent execs strace) is probably a bad idea - the program will trade PIDs like a hot potato if the decorated functions are used too often.

popen3 and return code

I want to get stdout and stderr of a command along with the return code.
Can someone point me to a python function that already accomplishes this?
I modified a function that I found on this site as follows -- but I am not able to grab the return code of the command. In this snippet, sts always seems to be 0.
def Getstatusoutput(cmd):
"""Return (status, output) of executing cmd in a shell."""
import sys
mswindows = (sys.platform == "win32")
import os
if not mswindows:
cmd = '{ ' + cmd + '; }'
fi,fo,fe=os.popen3(cmd)
textOut = fo.read()
textErr = fe.read()
sts = fo.close()
if sts is None: sts = 0
if textOut[-1:] == '\n': textOut = textOut[:-1]
return sts, textOut, textErr

Use the subprocess module. This section shows how to replace os.popen3 calls.