I would like to create a python module that would be called with python -m mymodule somefile.py some_arg some_arg.
The idea is that I would be able to set up an alias alias="python -m mymodule" and call files normally with python somefile.py some_arg some_arg.
In the file mymodule/__main__.py, what is the best way to load somefile.py and pass it the argument list?
I am looking for a generic solution, that would be python2 and 3 compatible.
It would be great to be as little intrusive as possible. If somefile.py would raise an exception, mymodule should barely be seen in the traceback.
What the module does is not interesting here in detail, but it sets up some python things (traceback hooks etc.), so somefile.py should be ran pythonicly in the same process. os.system or subprocess.Popen do not fit.
Ok I found something good for python 3.5, and satisfying enough for python 2.7.
mymodule/main.py
import sys
# The following block of code removes the part of
# the traceback related to this very module, and runpy
# Negative limit support came with python 3.5, so it will not work
# with previous versions.
# https://docs.python.org/3.5/library/traceback.html#traceback.print_tb
def myexcepthook(type, value, tb):
nb_noise_lines = 3
traceback_size = len(traceback.extract_tb(tb))
traceback.print_tb(tb, nb_noise_lines - traceback_size)
if sys.version_info >= (3, 5):
sys.excepthook = myexcepthook
if len(sys.argv) > 1:
file = sys.argv[1]
sys.argv = sys.argv[1:]
with open(file) as f:
code = compile(f.read(), file, 'exec')
exec(code)
somefile.py
import sys
print sys.argv
raise Exception()
in the terminal
$ python3 -m mymodule somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "somefile.py", line 3, in <module>
raise Exception()
$ python2 -m mymodule somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "/usr/lib64/python3.5/runpy.py", line 184, in _run_module_as_main
"__main__", mod_spec)
File "/usr/lib64/python3.5/runpy.py", line 85, in _run_code
exec(code, run_globals)
File "/home/azmeuk/dev/testpy/mymodule/__main__.py", line 16, in <module>
exec(code)
File "somefile.py", line 3, in <module>
raise Exception()
$ python somefile.py some_arg some_arg
['somefile.py', 'some_arg', 'some_arg']
Traceback (most recent call last):
File "somefile.py", line 3, in <module>
raise Exception()
Exception
Still, if someone has a better proposition, it would be great!
I think the negative value of limit does not work in traceback module before python 3.5. Here is an ugly hack that works with python 2.7
import sys
import traceback
class ExcFile(object):
def __init__(self, file):
self.topline = True
self.file = file
def write(self, s):
if self.topline:
u, s = s.split('\n', 1)
self.file.write(u +'\n')
self.topline = False
if '#---\n' in s:
u, s = s.split('#---\n', 1)
self.file.write(s)
self.write = self.file.write
ExcFile._instance = ExcFile(sys.stdout)
# The following block of code removes the part of
# the traceback related to this very module, and runpy
def myexcepthook(type, value, tb):
traceback.print_exception(type, value, tb, file=ExcFile._instance)
sys.excepthook = myexcepthook
if len(sys.argv) > 1:
file = sys.argv[1]
sys.argv = sys.argv[1:]
with open(file) as f:
code = compile(f.read(), file, 'exec')
exec(code) #---
All this should be written in a separate file, to avoid clutter __main__.py.
Related
import fileinput
def main()
try:
lines = fileinput.input()
res = process_lines(lines)
...more code
except Exception:
print('is your file path bad?')
if __name__ == '__main__':
main()
When I run this code with a bad path, it doesn't throw an error, yet the docs say that an OS error will be thrown if there's an IO error. How do I test for bad paths then?
fileinput.input() returns an iterator, not an ad-hoc list:
In [1]: fileinput.input()
Out[1]: <fileinput.FileInput at 0x7fa9bea55a50>
Proper usage of this function is done via a for loop:
with fileinput.input() as files:
for line in files:
process_line(line)
or using a conversion to list:
lines = list(fileinput.input())
I.e. the files are opened only when you actually iterate over this object.
Although I wouldn't recommend the second way, as it is counter to the philosophy of how such scripts are supposed to work.
You are supposed to parse as little as you need to output data, and then output it as soon as possible. This avoids issues with large inputs, and if your script is used within a larger pipeline, speeds up the processing significantly.
With regards to checking whether the path is correct or not:
As soon as you'll iterate down to the file that doesn't exist, the iterator will throw an exception:
# script.py
import fileinput
with fileinput.input() as files:
for line in files:
print(repr(line))
$ echo abc > /tmp/this_exists
$ echo xyz > /tmp/this_also_exists
$ python script.py /tmp/this_exists /this/does/not /tmp/this_also_exists
'abc\n'
Traceback (most recent call last):
File "/tmp/script.py", line 6, in <module>
for line in files:
File "/home/mrmino/.pyenv/versions/3.7.7/lib/python3.7/fileinput.py", line 252, in __next__
line = self._readline()
File "/home/mrmino/.pyenv/versions/3.7.7/lib/python3.7/fileinput.py", line 364, in _readline
self._file = open(self._filename, self._mode)
FileNotFoundError: [Errno 2] No such file or directory: '/this/does/not'
#! python3
from contextlib import redirect_stderr
import io
f = io.StringIO()
with redirect_stderr(f):
# simulates an error
erd
As seen above, I have used the redirect_stderr function to redirect stderr to a StringIO object. However, it doesn't work, as the error message is still printed out in command prompt:
Traceback (most recent call last):
File "C:\Users\max\testerr.py", line 8, in <module>
erd
NameError: name 'erd' is not defined
I tested it on Python 3.5.1 64 bit and 3.5.2 64 bit with the same results.
A similar issue in this thread
I have also tried writing the error to a file as described in the linked thread, but the file is empty after running the script.
You need to actually write to stderr, it is not a tool to catch exceptions.
>>> from contextlib import redirect_stderr
>>> import io
>>> f = io.StringIO()
>>> import sys
>>> with redirect_stderr(f):
... print('Hello', file=sys.stderr)
...
>>> f.seek(0)
0
>>> f.read()
'Hello\n'
To catch exceptions, you need to do some more work. You can use a logging library (external), or write your own exception handler, and then use your custom output for it.
Here is something quick, which uses a logger instance to help write to the stream:
log = logging.getLogger('TEST')
log.addHandler(logging.StreamHandler(stream=f))
def exception_handler(exc_type, exc_value, exc_traceback):
if issubclass(exc_type, KeyboardInterrupt):
# Let the system handle things like CTRL+C
sys.__excepthook__(*args)
log.error('Exception: ', exc_info=(exc_type, exc_value, exc_traceback))
sys.excepthook = exception_handler
raise RuntimeError('foo')
Here f is the same StringIO instance from above. After you run this code, you should not see any traceback on your console, but it will be stored in the stream object:
>>> f.seek(0)
0
>>> print(f.read())
Hello
Exception:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
RuntimeError: foo
I have simple Python code that uses dockets
#!/usr/bin/python
# http://stackoverflow.com/questions/2708178/python-using-doctests-for-classes
class Test:
def __init__(self, number):
self._number=number
def multiply_by_2(self):
"""
>>> t.multiply_by_2()
4
"""
return self._number*2
if __name__ == "__main__":
import doctest
doctest.testmod(extraglobs={'t': Test(2)})
I can use it with python interpreter:
> python simple.py
However, when I execute the code from doctest module, I get this error:
> python -m doctest simple.py
**********************************************************************
File "simple.py", line 10, in simple.Test.multiply_by_2
Failed example:
t.multiply_by_2()
Exception raised:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/doctest.py", line 1289, in __run
compileflags, 1) in test.globs
File "<doctest simple.Test.multiply_by_2[0]>", line 1, in <module>
t.multiply_by_2()
NameError: name 't' is not defined
**********************************************************************
1 items had failures:
1 of 1 in simple.Test.multiply_by_2
***Test Failed*** 1 failures.
Why is this difference? How to resolve this issue?
The difference is that when you execute via doctest, it is the __main__ module compared to executing directly where your script's if __name__ == '__main__' block will execute.
I don't know of a good solution other than to put all the information you need in the docstring itself:
def multiply_by_2(self):
"""
>>> t = Test(2)
>>> t.multiply_by_2()
4
"""
return self._number * 2
This will have the added benefit that users who are reading your docstrings will know what's going on ... They won't have to stumble upon your extraglobs keyword to figure out what t is and how it was initialized.
In Python Packaging: Hate, hate, hate everywhere, Armin says:
[...] Python tracebacks no longer included the source lines with the traceback. However there was no technical limitation for why it should not be able to show the correct line numbers there. It was just a bug in the Python interpreter.
I'm seeing exactly this issue for eggs in my virtualenv:
Traceback (most recent call last):
File "/users/example/venv/current/bin/my_script", line 37, in <module>
sys.exit(demo.scripts.foo.main())
File "build/bdist.linux-x86_64/egg/example/demo/scripts/my_script.py", line 90, in main
File "build/bdist.linux-x86_64/egg/example/demo/lib/bar.py", line 18, in func_x
File "build/bdist.linux-x86_64/egg/example/demo/lib/bar.py", line 55, in func_y
AttributeError: 'tuple' object has no attribute 'sort'
Since this is a known bug, are there workarounds? Is there an issue in the Python bug tracker (I can't find one)?
This is a proof of concept
import os
import sys
import traceback
import linecache
def recurse(depth=10):
if depth:
recurse(depth-1)
os.path.join(None, None)
def locate_filename(filename):
def generate_segments():
parts = filename.split(os.sep)
for i in xrange(len(parts) - 1, 0, -1):
yield os.sep.join(os.path.join(parts[i:]))
for segment in generate_segments():
for path in sys.path:
candidate = os.path.join(path, segment)
if os.path.exists(candidate):
return candidate
try:
recurse()
except:
_, _, tb = sys.exc_info()
for filename, lineno, functionname, _ in traceback.extract_tb(tb):
print filename, lineno, functionname
relocated_filename = locate_filename(filename)
if relocated_filename:
print linecache.getline(relocated_filename, lineno)
When I get a runtime exception from the standard library, it's almost always a problem in my code and not in the library code. Is there a way to truncate the exception stack trace so that it doesn't show the guts of the library package?
For example, I would like to get this:
Traceback (most recent call last):
File "./lmd3-mkhead.py", line 71, in <module>
main()
File "./lmd3-mkhead.py", line 66, in main
create()
File "./lmd3-mkhead.py", line 41, in create
headver1[depotFile]=rev
TypeError: Data values must be of type string or None.
and not this:
Traceback (most recent call last):
File "./lmd3-mkhead.py", line 71, in <module>
main()
File "./lmd3-mkhead.py", line 66, in main
create()
File "./lmd3-mkhead.py", line 41, in create
headver1[depotFile]=rev
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/__init__.py", line 276, in __setitem__
_DeadlockWrap(wrapF) # self.db[key] = value
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/dbutils.py", line 68, in DeadlockWrap
return function(*_args, **_kwargs)
File "/usr/anim/modsquad/oses/fc11/lib/python2.6/bsddb/__init__.py", line 275, in wrapF
self.db[key] = value
TypeError: Data values must be of type string or None.
update: added an answer with the code, thanks to the pointer from Alex.
The traceback module in Python's standard library lets you emit error tracebacks in a way that accords to your liking, while an exception is propagating. You can use this power either in the except leg of a try/except statement, or in a function you've installed as sys.excepthook, which gets called if and when an exception propagates all the way; quoting the docs:
In an interactive session this happens
just before control is returned to the
prompt; in a Python program this
happens just before the program exits.
The handling of such top-level
exceptions can be customized by
assigning another three-argument
function to sys.excepthook.
Here's a simple, artificial example:
>>> import sys
>>> import traceback
>>> def f(n):
... if n<=0: raise ZeroDivisionError
... f(n-1)
...
>>> def excepthook(type, value, tb):
... traceback.print_exception(type, value, tb, 3)
...
>>> sys.excepthook = excepthook
>>> f(8)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 3, in f
File "<stdin>", line 3, in f
ZeroDivisionError
as you see, without needing a try/except, you can easily limit the traceback to (for example) the first three levels -- even though we know by design that there were 9 nested levels when the exception was raised.
You want something more sophisticated than a simple limit on levels, so you'll need to call traceback.format_exception, which gives you a list of lines rather than printing it, then "prune" from that list the lines that are about modules you never want to see in your tracebacks, and finally emit the remaining lines (typically to sys.stderr, but, whatever!-).
Thanks to the pointer from Alex, here's teh codez:
def trimmedexceptions(type, value, tb, pylibdir=None, lev=None):
"""trim system packages from the exception printout"""
if pylibdir is None:
import traceback, distutils.sysconfig
pylibdir = distutils.sysconfig.get_python_lib(1,1)
nlev = trimmedexceptions(type, value, tb, pylibdir, 0)
traceback.print_exception(type, value, tb, nlev)
else:
fn = tb.tb_frame.f_code.co_filename
if tb.tb_next is None or fn.startswith(pylibdir):
return lev
else:
return trimmedexceptions(type, value, tb.tb_next, pylibdir, lev+1)
import sys
sys.excepthook=trimmedexceptions
# --- test code ---
def f1(): f2()
def f2(): f3()
def f3():
import xmlrpclib
proxy = xmlrpclib.ServerProxy('http://nosuchserver')
proxy.f()
f1()
Which yields this stack trace:
Traceback (most recent call last):
File "./tsttraceback.py", line 47, in <module>
f1()
File "./tsttraceback.py", line 40, in f1
def f1(): f2()
File "./tsttraceback.py", line 41, in f2
def f2(): f3()
File "./tsttraceback.py", line 45, in f3
proxy.f()
gaierror: [Errno -2] Name or service not known
The Traceback library is probably what you want. Here's one example that might help:
import traceback
try:
your_main()
except:
lines = traceback.format_exc()
print lines[:lines.find('File "/usr')]
(This obviously won't work if there's an exception outside the library, and might not exactly fit your needs, but it's one way of using the traceback library)
Put an unqualified try...except at the top of your code (ie: in your "main") or set sys.excepthook. You can then format the stack trace however you'd like.