I am looking for a way to quickly print a variable name and value while rapidly developing/debugging a small Python script on a Unix command line/ssh session.
It seems like a very common requirement and it seems wasteful (on keystrokes and time/energy) to duplicate the variable_names on every line which prints or logs its value. i.e. rather than
print 'my_variable_name:', my_variable_name
I want to be able to do the following for str, int, list, dict:
log(my_variable_name)
log(i)
log(my_string)
log(my_list)
and get the following output:
my_variable_name:some string
i:10
my_string:a string of words
my_list:[1, 2, 3]
Ideally the output would also log the function name.
I have seen some solutions attempting to use locals, globals, frames etc., But I have not yet seen something that works for ints, strings, lists, and works inside functions too.
Thanks!
Sorry to Necro this ancient thread, but this was surprisingly difficult to find a good answer for.
Using the '=' sign after the variable achieves this. For instance:
import pathlib as pl
import logging
logging.basicConfig(level=logging.DEBUG)
data_root = pl.Path("D:\Code_Data_Dev\Data\CSV_Workspace_Data")
logging.debug(f'{data_root=}')
This outputs
DEBUG:root:data_root=WindowsPath('D:/Code_Data_Dev/Data/CSV_Workspace_Data')
If the tool you need is only for developing and debugging, there's a useful package called q.
It has been submitted to pypi, it can be installed with pip install q or easy_install q.
import q; q(foo)
# use #q to trace a function's arguments and return value
#q
def bar():
...
# to start an interactive console at any point in your code:
q.d()
The results are output to the file /tmp/q (or any customized paths) by default, so they won't be mixed with stdout and normal logs. You can check the output with tail -f /tmp/q. The output is highlighted with different colors.
The author introduced his library in a lightning talk of PyconUS 2013. The video is here, begins at 25:15.
Here is another evil hack:
import inspect
def log(a):
call_line = inspect.stack()[1][4][0].strip()
assert call_line.strip().startswith("log(")
call_line = call_line[len("log("):][:-1]
print "Log: %s = %s" % (call_line, a)
b=3
log(b)
This obviously would need some range checks, better parsing, etc.
Also works only if the calls is made always in the same way and has probably more - unknown to me - assumptions...
I don't know any way to simply get the name of a variable of a string.
you can however get the list of argument of the current fonction log.
import inspect
def log(a):
frame = inspect.currentframe()
args, _, _, values = inspect.getargvalues(frame)
print "%s:%s" % (args[0], values[args[0]])
Related
I am writing a Python interpreter and want to redirect the function's return values to stdout, like the Python Interpreter in Interactive Mode. Within this mode, when the user calls a function, its return value is printed on the screen. The same occurs with expressions.
E.g.
>>> foo()
'Foo return value'
>>> 2+4
6
>>> print('Hello!')
'Hello!'
Changing the sys.stdout only affects the print function. How do I redirect the other expressions to stdout?
Thank you
First, the interactive mode does not print the return value from any function called. Instead, it prints the result of whatever expression the user typed in. If that's not a function call, it still gets printed. If it has 3 function calls in it, it still prints one result, not 3 lines. And so on.
So, trying to redirect function return values to stdout is the wrong thing to do.
What the interactive interpreter does is something sort of like this:
line = raw_input(sys.ps1)
_ = eval(line)
if _ is not None:
print repr(_)
(You may notice that you can change sys.ps1 from the interactive prompt to change what the prompt looks like, access _ to get the last value, etc.)
However, that's not what it really does. And that's not how you should go about this yourself either. If you try, you'll have to deal with complexities like keeping your own globals separate from the user's, handling statements as well as expressions, handling multi-line statements and expressions (doing raw_input(sys.ps2) is easy, but how do you know when to do that?), interacting properly with readline and rlcomplete, etc.
There's a section of the documentation called Custom Python Interpreters which explains the easy way to do this:
The modules described in this chapter allow writing interfaces similar to Python’s interactive interpreter. If you want a Python interpreter that supports some special feature in addition to the Python language, you should look at the code module.
And code:
… provides facilities to implement read-eval-print loops in Python. Two classes and convenience functions are included which can be used to build applications which provide an interactive interpreter prompt.
The idea is that you let Python do all the hard stuff, up to whatever level you want to take over, and then you just write the part on top of that.
You may want to look at the source for IDLE, ipython, bpython, etc. for ideas.
Instead of using exec() to run the user input, try eval():
retval = eval(user_input)
sys.stdout.write(repr(retval) + "\n")
Sometimes I have a lot of prints scattered around function to print debug output.
To switch this debug outputs I came up with this:
def f(debug=False):
print = __builtins__.print if debug else lambda *p: None
Or if I need to print something apart from debug message, I create dprint function for debug messages.
The problem is, when debug=False, this print statements slow down the code considerably, because lambda *p: None is still called, and function invocation are known to be slow.
So, my question is: Is there any better way to efficiently disable all these debug prints for them not to affect code performance?
All the answers are regarding my not using logging module. This is a good to notice, but this doesn't answer the question how to avoid function invocations that slow down the code considerably - in my case 25 times (if it's possible (for example by tinkering with function code object to through away all the lines with print statements or somehow else)). What these answers suggest is replacing print with logging.debug, which should be even slower. And this question is about getting rid of those function calls completely.
I tried using logging instead of lambda *p: None, and no surprise, code became even slower.
Maybe someone would like to see the code where those prints caused 25 slowdown: http://ideone.com/n5PGu
And I don't have anything against logging module. I think it's a good practice to always stick to robust solutions without some hacks. But I thinks there is nothing criminal if I used those hacks in 20-line one-time code snippet.
Not as a restriction, but as a suggestion, maybe it's possible to delete some lines (e.g. starting with print) from function source code and recompile it? I laid out this approach in the answer below. Though I would like to see some comments on that solution, I welcome other approaches to solving this problem.
You should use the logging module instead. See http://docs.python.org/library/logging.html
Then you can set the log level depending on your needs, and create multiple logger objects, that log about different subjects.
import logging
#set your log level
logging.basicConfig(level=logging.DEBUG)
logging.debug('This is a log message')
In your case: you could simply replace your print statement with a log statement, e.g.:
import logging
print = __builtins__.print if debug else logging.debug
now the function will only be print anything if you set the logging level to debug
logging.basicConfig(level=logging.DEBUG)
But as a plus, you can use all other logging features on top! logging.error('error!')
Ned Batchelder wrote in the comment:
I suspect the slow down is in the calculation of the arguments to
your debug function. You should be looking for ways to avoid those
calculations. Preprocessing Python is just a distraction.
And he is right as slowdown is actually caused by formatting string with format method which happens regardless if the resulting string will be logged or not.
So, string formatting should be deferred and dismissed if no logging will occur. This may be achieved by refactoring dprint function or using log.debug in the following way:
log.debug('formatted message: %s', interpolated_value)
If message won't be logged, it won't be formatted, unlike print, where it's always formatted regardless of if it'll be logged or discarded.
The solution on log.debug's postponed formatting gave Martijn Pieters here.
Another solution could be to dynamically edit code of f and delete all drpint calls. But this solution is highly unrecommended to be used:
You are correct, you should never resort to this, there are so many
ways it can go wrong. First, Python is not a language designed for
source-level transformations, and it's hard to write it a transformer
such as comment_1 without gratuitously breaking valid code. Second,
this hack would break in all kinds of circumstances - for example,
when defining methods, when defining nested functions, when used in
Cython, when inspect.getsource fails for whatever reason. Python is
dynamic enough that you really don't need this kind of hack to
customize its behavior.
Here is the code of this approach, for those who like to get acquainted with it:
from __future__ import print_function
DEBUG = False
def dprint(*args,**kwargs):
'''Debug print'''
print(*args,**kwargs)
_blocked = False
def nodebug(name='dprint'):
'''Decorator to remove all functions with name 'name' being a separate expressions'''
def helper(f):
global _blocked
if _blocked:
return f
import inspect, ast, sys
source = inspect.getsource(f)
a = ast.parse(source) #get ast tree of f
class Transformer(ast.NodeTransformer):
'''Will delete all expressions containing 'name' functions at the top level'''
def visit_Expr(self, node): #visit all expressions
try:
if node.value.func.id == name: #if expression consists of function with name a
return None #delete it
except(ValueError):
pass
return node #return node unchanged
transformer = Transformer()
a_new = transformer.visit(a)
f_new_compiled = compile(a_new,'<string>','exec')
env = sys.modules[f.__module__].__dict__
_blocked = True
try:
exec(f_new_compiled,env)
finally:
_blocked = False
return env[f.__name__]
return helper
#nodebug('dprint')
def f():
dprint('f() started')
print('Important output')
dprint('f() ended')
print('Important output2')
f()
More information: Replacing parts of the function code on-the-fly
As a hack, yes, that works. (And there is no chance in hell those lambda no-ops are your app's bottleneck.)
However, you really should be doing logging properly by using the logging module.
See http://docs.python.org/howto/logging.html#logging-basic-tutorial for a basic example of how this should be done.
You definitely need to use the logging module of Python, it's very practical and you can change the log level of your application. Example:
>>> import logging
>>> logging.basicConfig(level=logging.DEBUG)
>>> logging.debug('Test.')
DEBUG:root:Test.
in c code I frequently use printf debugging macros like
#define DPRINT_INT(i) fprintf(stderr,"%s has the value %i at line %i", #i,i, __LINE__)
and then i can do things like
DPRINT_INT(height)
where it will print the variable or things like
DPRINT_INT(from_cm_to_inch(get_average(heights)))
and it will print out the whole expression for the name.
To do this for python, since python doesn't have c-like macros
I pass a string and use inspect to get the calling functions environment to call eval with.
But I don't like passing strings, its ugly and easy to forget(I have it check type and call an exception if it gets passed a non string) and doesn't work as well with the ide.
There isn't any way to extract the variable names and expressions from the python code for the debug function? is there?
In Python we tend to write modules that we can unittest and / or import into the REPL and drive them there if necessary.
If you can unit-test your functions, you can prove their behaviour for any given input. However, if you must write debug statements, you should use debug() from the standard logging module.
For example:
#!/usr/bin/env python
import logging
import inspect
def buggy_fn():
d = 42;
if d != 69:
logging.debug('%s(%d): %s is not what we expected. [%s]',
inspect.currentframe().f_back.f_code.co_filename,
inspect.currentframe().f_back.f_lineno,
'd',
repr(d),
)
if __name__ == '__main__':
logging.basicConfig(level=logging.DEBUG)
buggy_fn()
Will output...
DEBUG:root:./buggy.py(19): d is not what we expected. [42]
There's a wealth of useful stuff in inspect that may help you out in your hour of need.
Is there a way to view a list of all of my variables in python while the program is running without setting breakpoints? Printing is too messy because I have a lot of variables that are constantly changing.
Thanks
Maybe inspect helps you, but you have to filter the information then.
Usage like:
> import inspect
> a = 5
> f = inspect.currentframe()
> print f.f_locals
...
...
'a': 5
...
Maybe it's worth to mention that you cannot iterate over the resulting dictionary in a for loop because asignment to a variable would change that dictionary. You have to iterate only over the keys (at least that's what I just found out).
Example:
for v in f.f_locals.keys():
if not v.startswith("_"):
print v
Look at the first line: simply writing for v in f.f_locals would not succeed.
If you are running on Pydev (python extension for eclipse), you can easily watch your variables, however you'll need to set a breakpoint initially, then only step into/over your code.
Grab a debugger: http://winpdb.org/ and then you can watch them and see them in there; however if you're just looking for a function to print all the variables defined in the local scope, just calls locals().
If your question is referring to python as a language (as opposed to "python as an ecosystem of applications and utilities"), AFAIK, the answer is "no" (not out-of-the-box anyhow). Of course there are a number of IDE's that allow you to use debug to see the values of variables [see Sumit answer, for example].
However if during development you wanted a "live monitor" of a number of variables, you could use the logging module, and define your own Handler class so as to redirect the information "live" somewhere. Setting the message to log.debug level would make trivial to activate/deactivate this feature by simply changing the minimum log level of your application.
HTH!
I have a bunch of print calls that I need to write to a file instead of stdout. (I don't need stdout at all.)
I am considering three approaches. Are there any advantages (including performance) to any one of them?
Full redirect, which I saw here:
import sys
saveout = sys.stdout
fsock = open('out.log', 'w')
sys.stdout = fsock
print(x)
# and many more print calls
# later if I ever need it:
# sys.stdout = saveout
# fsock.close()
Redirect in each print statement:
fsock = open('out.log', 'w')
print(x, file = fsock)
# and many more print calls
Write function:
fsock = open('out.log', 'w')
fsock.write(str(x))
# and many more write calls
I would not expect any durable performance differences among these approaches.
The advantage of the first approach is that any reasonably well-behaved code which you rely upon (modules you import) will automatically pick up your desired redirection.
The second approach has no advantage. It's only suitable for debugging or throwaway code ... and not even a good idea for that. You want your output decisions to be consolidated in a few well-defined places, not scattered across your code in every call to print(). In Python3 print() is a function rather than a statement. This allows you to re-define it, if you like. So you can def print(*args) if you want. You can also call __builtins__.print() if you need access to it, within the definition of your own custom print(), for example.
The third approach ... and by extension the principle that all of your output should be generated in specific functions and class methods that you define for that purpose ... is probably best.
You should keep your output and formatting separated from your core functionality as much as possible. By keeping them separate you allow your core to be re-used. (For example you might start with something that's intended to run from a text/shell console, and later need to provide a Web UI, a full-screen (curses) front end or a GUI for it. You may also build entirely different functionality around it ... in situations where the resulting data needs to be returned in its native form (as objects) rather than pulled in as text (output) and re-parsed into new objects.
For example I've had more than one occasional where something I wrote to perform some complex queries and data gathering from various sources and print a report ... say of the discrepancies ... later need to be adapted into a form which could spit out the data in some form (such as YAML/JSON) that could be fed into some other system (say, for reconciling one data source against another.
If, from the outset, you keep the main operations separate from the output and formatting then this sort of adaptation is relatively easy. Otherwise it entails quite a bit of refactoring (sometimes tantamount to a complete re-write).
From the filenames you're using in your question, it sounds like you're wanting to create a log file. Have you consider the Python logging module instead?
I think that semantics is imporant:
I would suggest first approach for situation when you printing the same stuff you would print to console. Semantics will be the same. For more complex situation I would use standard logging module.
The second and third approach are a bit different in case you are printing text lines. Second approach - print adds the newline and write does not.
I would use the third approach in writing mainly binary or non-textual format and I would use redirect in print statement in the most other cases.