Let's say I have the following function which has an attribute which marks it for special handling within a callback subsystem:
def my_func(msg):
print msg
my_func.my_marker = SPECIAL_CONSTANT
The problem is that if other bits of code wrap my_func with functools.partial or another decorator, then my_marker will be lost.
my_partial = partial(my_func, 'hello world')
print my_partial.my_marker
>>> AttributeError...
Is there a way to protect attributes on functions when wrapping? Is there a better way to store the metadata I'm currently storing in my_marker? It seems like storing a reference to the original function suffers from the same problem.
If you know that the partial you use is actually a partial, you can use it's func attribute.
E. g.
from functools import partial
SPECIAL_CONSTANT = 'bam'
def my_func(msg):
print msg
my_func.my_marker = SPECIAL_CONSTANT
my_partial = partial(my_func, 'hello world')
print my_partial.func.my_marker
If you really have to handle meta_data, maybe it is a better approach to write classes and override the __call__() method.
Here is another solution, using functools.update_wrapper.
from functools import partial, update_wrapper, WRAPPER_ASSIGNMENTS
SPECIAL_CONSTANT = 'bam'
def my_func(msg):
print msg
my_func.my_marker = SPECIAL_CONSTANT
my_partial = partial(my_func, 'hello world')
update_wrapper(my_partial, my_func, WRAPPER_ASSIGNMENTS + ('my_marker', ))
print my_partial.my_marker
Inspired by kindall's comment, cbo's answer and the source code:
from functools import partial as _partial, update_wrapper
def partial(func, *args, **keywords):
return update_wrapper(_partial(func, *args, **keywords), func)
An example showing it working, and caveats:
def my_func(msg):
print msg
my_func.my_marker = 'FOO'
my_partial = partial(my_func, 'hello world')
print my_func.my_marker
print my_partial.my_marker
# check other stuff is still there
print my_partial.func
print my_partial.args
print my_partial.keywords
# this works fine for READ ONLY stuff.
# so just remember:
my_partial.my_marker = 'BAR' # this _only_ updates partial, not the original
print my_func.my_marker
print my_partial.my_marker
my_func.my_marker = 'BAZ' # this _only_ updates original, not the partial
print my_func.my_marker
print my_partial.my_marker
You can do something like:
import functools
setattr(functools, 'partial', partial) # from above code
However, this is probably a bad idea as (1) it would need to be imported before any code which relied on it, (2) it could break imported code, (3) it might confuse future people and (4) the alternative, keeping it locally, is easy. Only do this if you want to force third party code to run your version.
Related
I am using cocotb version 1.5.2, and I would like to write an utility function to create reports/plots per test.
MWE: Implementing the get_test_name function so that the following test will print my_wonderful_test.
import cocotb
#cocotb.test()
async def my_wonderful_test(dut):
print(get_test_name(dut));
def get_test_name(dut):
pass # how do I get the current test from here?
You can use "name" attribute :
import cocotb
#cocotb.test()
async def my_wonderful_test(dut):
print(my_wonderful_test.name);
But not sure that exactly you want.
Thank you for the commenters, and thank you #FabienM for trying to give an answer. Thank you for #Tzane for trying to find an answer. You were close.
If you want to know a one liner answer
import cocotb;
def get_test_name():
return cocotb.regression_manager._test.name
but the underline prefix in _test maybe it will break in the future, but for since I was only concerned about version 1.5.2 this is OK for me.
Any way I implemented another method that scans the stack one level at a time and check if the frame is in a cocotb.test decorated function. This is also the method that cocotb uses to _discover_tests
It won't work if the test is in a closure, but I never use that, and I don't know if it is even supported.
import cocotb
import inspect;
import sys
#cocotb.test()
async def test_get_testname(dut):
print('Runnign from test ', get_test_name())
def get_test_name():
try:
return cocotb.regression_manager._test.name
except:
pass
cocotbdir = '/'.join(cocotb.__file__.split('/')[:-1])
frame = sys._getframe();
prev_frame = None
while frame is not None:
try:
# the [documentation](https://docs.python.org/2/library/inspect.html#inspect.getmodule)
# says
# Try to guess which module an object was defined in.
# Implying it may fail, wrapp in a try block and everything is fine
module = inspect.getmodule(frame.f_code)
func_name = inspect.getframeinfo(frame).function
if hasattr(module, func_name):
ff = getattr(module, func_name)
if isinstance(ff, cocotb.test):
return func_name
except:
pass
prev_frame = frame;
frame = frame.f_back;
# return None if fail to determine the test name
I don't know why my questions are so badly received
It was something simple that I preferred to engage more people
In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?
There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)
For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html
Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)
You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function
I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18
wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func
import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())
Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456
To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).
Function foo prints to console. I want to test the console print. How can I achieve this in python?
Need to test this function, has NO return statement :
def foo(inStr):
print "hi"+inStr
My test :
def test_foo():
cmdProcess = subprocess.Popen(foo("test"), stdout=subprocess.PIPE)
cmdOut = cmdProcess.communicate()[0]
self.assertEquals("hitest", cmdOut)
You can easily capture standard output by just temporarily redirecting sys.stdout to a StringIO object, as follows:
import StringIO
import sys
def foo(inStr):
print "hi"+inStr
def test_foo():
capturedOutput = StringIO.StringIO() # Create StringIO object
sys.stdout = capturedOutput # and redirect stdout.
foo('test') # Call unchanged function.
sys.stdout = sys.__stdout__ # Reset redirect.
print 'Captured', capturedOutput.getvalue() # Now works as before.
test_foo()
The output of this program is:
Captured hitest
showing that the redirection successfully captured the output and that you were able to restore the output stream to what it was before you began the capture.
Note that the code above in for Python 2.7, as the question indicates. Python 3 is slightly different:
import io
import sys
def foo(inStr):
print ("hi"+inStr)
def test_foo():
capturedOutput = io.StringIO() # Create StringIO object
sys.stdout = capturedOutput # and redirect stdout.
foo('test') # Call function.
sys.stdout = sys.__stdout__ # Reset redirect.
print ('Captured', capturedOutput.getvalue()) # Now works as before.
test_foo()
This Python 3 answer uses unittest.mock. It also uses a reusable helper method assert_stdout, although this helper is specific to the function being tested.
import io
import unittest
import unittest.mock
from .solution import fizzbuzz
class TestFizzBuzz(unittest.TestCase):
#unittest.mock.patch('sys.stdout', new_callable=io.StringIO)
def assert_stdout(self, n, expected_output, mock_stdout):
fizzbuzz(n)
self.assertEqual(mock_stdout.getvalue(), expected_output)
def test_only_numbers(self):
self.assert_stdout(2, '1\n2\n')
Note that the mock_stdout arg is passed automatically by the unittest.mock.patch decorator to the assert_stdout method.
A general-purpose TestStdout class, possibly a mixin, can in principle be derived from the above.
For those using Python ≥3.4, contextlib.redirect_stdout also exists, but it seems to serve no benefit over unittest.mock.patch.
If you happen to use pytest, it has builtin output capturing. Example (pytest-style tests):
def eggs():
print('eggs')
def test_spam(capsys):
eggs()
captured = capsys.readouterr()
assert captured.out == 'eggs\n'
You can also use it with unittest test classes, although you need to passthrough the fixture object into the test class, for example via an autouse fixture:
import unittest
import pytest
class TestSpam(unittest.TestCase):
#pytest.fixture(autouse=True)
def _pass_fixtures(self, capsys):
self.capsys = capsys
def test_eggs(self):
eggs()
captured = self.capsys.readouterr()
self.assertEqual('eggs\n', captured.out)
Check out Accessing captured output from a test function for more info.
You can also use the mock package as shown below, which is an example from
https://realpython.com/lessons/mocking-print-unit-tests.
from mock import patch
def greet(name):
print('Hello ', name)
#patch('builtins.print')
def test_greet(mock_print):
# The actual test
greet('John')
mock_print.assert_called_with('Hello ', 'John')
greet('Eric')
mock_print.assert_called_with('Hello ', 'Eric')
The answer of #Acumenus says:
It also uses a reusable helper method assert_stdout, although this helper is specific to the function being tested.
the bold part seems a big drawback, thus I would do the following instead:
# extend unittest.TestCase with new functionality
class TestCase(unittest.TestCase):
def assertStdout(self, expected_output):
return _AssertStdoutContext(self, expected_output)
# as a bonus, this syntactical sugar becomes possible:
def assertPrints(self, *expected_output):
expected_output = "\n".join(expected_output) + "\n"
return _AssertStdoutContext(self, expected_output)
class _AssertStdoutContext:
def __init__(self, testcase, expected):
self.testcase = testcase
self.expected = expected
self.captured = io.StringIO()
def __enter__(self):
sys.stdout = self.captured
return self
def __exit__(self, exc_type, exc_value, tb):
sys.stdout = sys.__stdout__
captured = self.captured.getvalue()
self.testcase.assertEqual(captured, self.expected)
this allows for the much nicer and much more re-usable:
# in a specific test case, the new method(s) can be used
class TestPrint(TestCase):
def test_print1(self):
with self.assertStdout("test\n"):
print("test")
by using a straight forward context manager. (It might also be desirable to append "\n" to expected_output since print() adds a newline by default. See next example...)
Furthermore, this very nice variant (for an arbitrary number of prints!)
def test_print2(self):
with self.assertPrints("test1", "test2"):
print("test1")
print("test2")
is possible now.
You can also capture the standard output of a method using contextlib.redirect_stdout:
import unittest
from contextlib import redirect_stdout
from io import StringIO
class TestMyStuff(unittest.TestCase):
# ...
def test_stdout(self):
with redirect_stdout(StringIO()) as sout:
my_command_that_prints_to_stdout()
# the stream replacing `stdout` is available outside the `with`
# you may wish to strip the trailing newline
retval = sout.getvalue().rstrip('\n')
# test the string captured from `stdout`
self.assertEqual(retval, "whatever_retval_should_be")
Gives you a locally scoped solution. It is also possible to capture the standard error using contextlib.redirect_stderr().
Another variant is leaning on the logging module rather than print(). This module also has a suggestion of when to use print in the documentation:
Display console output for ordinary usage of a command line script or program
PyTest has built-in support for testing logging messages.
I have many Python files (submission1.py, submission2.py, ... , submissionN.py) in the following format,
#submission1.py
def fun():
print('some fancy function')
fun()
I want to write a tester to test these submissions. (They are actually homeworks that I am grading.). I have a tester for the fun() which is able to test the function itself. However, my problem is, when I import submission.py, it runs the fun() since it calls it at the end of file.
I know that, using if __name__ == "__main__": is the correct way of handling this issue, however, our submissions does not have it since we did not teach it.
So, my question is, is there any way that I can import only fun() from the submission.py files without running the rest of the python file?
For simple scripts with just functions the following will work:
submission1.py:
def fun(x):
print(x)
fun("foo")
def fun2(x):
print(x)
fun2("bar")
print("debug print")
You can remove all bar the FunctionDef nodes then recompile:
import ast
import types
with open("submission1.py") as f:
p = ast.parse(f.read())
for node in p.body[:]:
if not isinstance(node, ast.FunctionDef):
p.body.remove(node)
module = types.ModuleType("mod")
code = compile(p, "mod.py", 'exec')
sys.modules["mod"] = module
exec(code, module.__dict__)
import mod
mod.fun("calling fun")
mod.fun2("calling fun2")
Output:
calling fun
calling fun2
The module body contains two Expr and one Print node which we remove in the loop keeping just the FunctionDef's.
[<_ast.FunctionDef object at 0x7fa33357f610>, <_ast.Expr object at 0x7fa330298a90>,
<_ast.FunctionDef object at 0x7fa330298b90>, <_ast.Expr object at 0x7fa330298cd0>,
<_ast.Print object at 0x7fa330298dd0>]
So after the loop out body only contains the functions:
[<_ast.FunctionDef object at 0x7f49a786a610>, <_ast.FunctionDef object at 0x7f49a4583b90>]
This will also catch where the functions are called with print which if the student was calling the function from an IDE where the functions have return statements is pretty likely, also to keep any imports of there are any you can keep ast.Import's and ast.ImportFrom's:
submission.py:
from math import *
import datetime
def fun(x):
print(x)
fun("foo")
def fun2(x):
return x
def get_date():
print(pi)
return datetime.datetime.now()
fun2("bar")
print("debug print")
print(fun2("hello world"))
print(get_date())
Compile then import:
for node in p.body[:]:
if not isinstance(node, (ast.FunctionDef,ast.Import, ast.ImportFrom)):
p.body.remove(node)
.....
import mod
mod.fun("calling fun")
print(mod.fun2("calling fun2"))
print(mod.get_date())
Output:
calling fun
calling fun2
3.14159265359
2015-05-09 12:29:02.472329
Lastly if you have some variables declared that you need to use you can keep them using ast.Assign:
submission.py:
from math import *
import datetime
AREA = 25
WIDTH = 35
def fun(x):
print(x)
fun("foo")
def fun2(x):
return x
def get_date():
print(pi)
return datetime.datetime.now()
fun2("bar")
print("debug print")
print(fun2("hello world"))
print(get_date()
Add ast.Assign:
for node in p.body[:]:
if not isinstance(node, (ast.FunctionDef,
ast.Import, ast.ImportFrom,ast.Assign)):
p.body.remove(node)
....
Output:
calling fun
calling fun2
3.14159265359
2015-05-09 12:34:18.015799
25
35
So it really all depends on how your modules are structured and what they should contain as to what you remove. If there are literally only functions then the first example will do what you want. If there are other parts that need to be kept it is just a matter of adding them to the isinstance check.
The listing of all the abstract grammar definitions is in the cpython source under Parser/Python.asdl.
You could use sys.settrace() to catch function definitions.
Whenever your fun() is defined, you save it somewhere, and you place a stub into the module you are importing, so that it won't get executed.
Assuming that fun() gets defined only once, this code should do the trick:
import sys
fun = None
def stub(*args, **kwargs):
pass
def wait_for_fun(frame, event, arg):
global fun
if frame.f_code.co_filename == '/path/to/module.py':
if 'fun' in frame.f_globals:
# The function has just been defined. Save it.
fun = frame.f_globals['fun']
# And replace it with our stub.
frame.f_globals['fun'] = stub
# Stop tracing the module execution.
return None
return wait_for_fun
sys.settrace(wait_for_fun)
import my_module
# Now fun() is available and we can test it.
fun(1, 2, 3)
# We can also put it again inside the module.
# This is important if other functions in the module need it.
my_module.fun = fun
This code can be improved in many ways, but it does its job.
maybe if you just want to import the fun () function from submission.py try
from submission import fun
To perform the function of fun, you must include the fun module
submission.fun()
or if you want to make it easier when calling the fun () function, give it a try
from submission import fun as FUN
FUN ()
In C++, I can print debug output like this:
printf(
"FILE: %s, FUNC: %s, LINE: %d, LOG: %s\n",
__FILE__,
__FUNCTION__,
__LINE__,
logmessage
);
How can I do something similar in Python?
There is a module named inspect which provides these information.
Example usage:
import inspect
def PrintFrame():
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename) # __FILE__ -> Test.py
print(info.function) # __FUNCTION__ -> Main
print(info.lineno) # __LINE__ -> 13
def Main():
PrintFrame() # for this line
Main()
However, please remember that there is an easier way to obtain the name of the currently executing file:
print(__file__)
For example
import inspect
frame = inspect.currentframe()
# __FILE__
fileName = frame.f_code.co_filename
# __LINE__
fileNo = frame.f_lineno
There's more here http://docs.python.org/library/inspect.html
Building on geowar's answer:
class __LINE__(object):
import sys
def __repr__(self):
try:
raise Exception
except:
return str(sys.exc_info()[2].tb_frame.f_back.f_lineno)
__LINE__ = __LINE__()
If you normally want to use __LINE__ in e.g. print (or any other time an implicit str() or repr() is taken), the above will allow you to omit the ()s.
(Obvious extension to add a __call__ left as an exercise to the reader.)
You can refer my answer:
https://stackoverflow.com/a/45973480/1591700
import sys
print sys._getframe().f_lineno
You can also make lambda function
I was also interested in a __LINE__ command in python.
My starting point was https://stackoverflow.com/a/6811020 and I extended it with a metaclass object. With this modification it has the same behavior like in C++.
import inspect
class Meta(type):
def __repr__(self):
# Inspiration: https://stackoverflow.com/a/6811020
callerframerecord = inspect.stack()[1] # 0 represents this line
# 1 represents line at caller
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
# print(info.filename) # __FILE__ -> Test.py
# print(info.function) # __FUNCTION__ -> Main
# print(info.lineno) # __LINE__ -> 13
return str(info.lineno)
class __LINE__(metaclass=Meta):
pass
print(__LINE__) # print for example 18
wow, 7 year old question :)
Anyway, taking Tugrul's answer, and writing it as a debug type method, it can look something like:
def debug(message):
import sys
import inspect
callerframerecord = inspect.stack()[1]
frame = callerframerecord[0]
info = inspect.getframeinfo(frame)
print(info.filename, 'func=%s' % info.function, 'line=%s:' % info.lineno, message)
def somefunc():
debug('inside some func')
debug('this')
debug('is a')
debug('test message')
somefunc()
Output:
/tmp/test2.py func=<module> line=12: this
/tmp/test2.py func=<module> line=13: is a
/tmp/test2.py func=<module> line=14: test message
/tmp/test2.py func=somefunc line=10: inside some func
import inspect
.
.
.
def __LINE__():
try:
raise Exception
except:
return sys.exc_info()[2].tb_frame.f_back.f_lineno
def __FILE__():
return inspect.currentframe().f_code.co_filename
.
.
.
print "file: '%s', line: %d" % (__FILE__(), __LINE__())
Here is a tool to answer this old yet new question!
I recommend using icecream!
Do you ever use print() or log() to debug your code? Of course, you
do. IceCream, or ic for short, makes print debugging a little sweeter.
ic() is like print(), but better:
It prints both expressions/variable names and their values.
It's 40% faster to type.
Data structures are pretty printed.
Output is syntax highlighted.
It optionally includes program context: filename, line number, and parent function.
For example, I created a module icecream_test.py, and put the following code inside it.
from icecream import ic
ic.configureOutput(includeContext=True)
def foo(i):
return i + 333
ic(foo(123))
Prints
ic| icecream_test.py:6 in <module>- foo(123): 456
To get the line number in Python without importing the whole sys module...
First import the _getframe submodule:
from sys import _getframe
Then call the _getframe function and use its' f_lineno property whenever you want to know the line number:
print(_getframe().f_lineno) # prints the line number
From the interpreter:
>>> from sys import _getframe
... _getframe().f_lineno # 2
Word of caution from the official Python Docs:
CPython implementation detail: This function should be used for internal and specialized purposes only. It is not guaranteed to exist in all implementations of Python.
In other words: Only use this code for personal testing / debugging reasons.
See the Official Python Documentation on sys._getframe for more information on the sys module, and the _getframe() function / submodule.
Based on Mohammad Shahid's answer (above).