I am using pycallgraph to analyze my code performance. However, the call graph is pretty messy with many calls to system functions as well as certain functions I would not like to document. How can I stop pycallgraph from reporting these calls?
Pycallgraph provides filtering capabilities to filter out any module, class or function you would like to exclude from call graph. Following function should be defined before you start the trace and passed to pycallgraph
Example
def filtercalls(call_stack, modul, clas, func, full):
mod_ignore = ['shutil','scipy.optimize','re','os','sys','json']
func_ignore = ['CustomFunctionName','pdbcall']
clas_ignore = ['pdb']
return modul not in mod_ignore and func not in func_ignore and clas not in clas_ignore
The pycallgraph trace start is
pycallgraph.start_trace(filter_func=filtercalls)
This way, any module, class or function you provide in filtercalls will be removed. Please note that many time in standard libraries providing just the module name is not enough. Thus, including numpy in mod_ignore will still result in numpy.core being included
Related
I'm working with Tensorflow in Python. In a custom written function I found #tf_export() before the function definition like below, the function of which I don't understand. Could somebody explain?
#tf_export("signal.ifftshift")
def ifftshift(x, axes=None, name=None):
As I understand, it allows Tensorflow to expose a function or class under a different name. For example, the Server class within the distribute module actually lives in the training/server_lib.py file within the repo, but, since it is exported as distribute.Server, you can use it like tf.distribute.Server().
# training/server_lib.py
#tf_export("distribute.Server", v1=["distribute.Server", "train.Server"])
#deprecation.deprecated_endpoints("train.Server")
class Server(object):
...
It makes it confusing to find the code, but I imagine it's a more flexible way to create these "logical" modules.
It is a convenient way to output dot delimited symbols directly to the tf API. Namely, a user can access ifftshift() with tf.signal.ifftshift(), without caring about the true path (here tf.python.ops.signal.fft_ops.ifftshif()).
I'm working on creating a large .py file that can be imported and used to solve mathematical formulas. I'd like to store the formulas in a procedure that is called input1_input2_input3(): for example the formual distance=speed*time is called dis_spe_tim().
The code so far is:
def dis_spe_tim():
def distance(speed, time):
result = speed*time
unit = input("What unit are you measuring the distance in?")
print(resule,unit)
def speed():
print("speed")
and I would ideally like the user to use this like so:
import equations #name of the .py file
from equations import *
dis_spe_tim.distance(1,2)
Unfortunately, this is my first time ever doing something like this so I have absolutely no idea how to go about calling the procedure inside of the procedure and providing its arguments.
Short answer: you can't. Nested functions are local to the function they're defined in and only exists during the outer function's execution (def is an executable statement that, at runtime, creates a function object and bind it to it's name in the enclosing namespace).
The canonical python solution is to use modules as namespaces (well, Python modules ARE, mainly, namespaces), ie have a distinct module for each "formula", and define the functions at the module's top-level:
# dis_spe_tim.py
def distance(speed, time):
# code here
def speed():
# code here
Then put all those modules in an equations package (mostly: a folder containing modules and an __init__.py file). Then you can do:
from equations import dis_spe_tim
dis_spe_tim.distance(1,2)
You can check the doc for more on modules and packages here: https://docs.python.org/3/tutorial/modules.html#packages
Also note that
1/ "star imports" (also named "wildcard imports"), ie from somemodule import *, are highly discouraged as they tend to make the code harder to read and maintain and can cause unexpected (and sometimes subtles enough to be hard to spot) breakages.
2/ you shouldn't mix "domain" code (code that do effective computations) with UI code (code that communicates with the user), so any call to input(), print() etc should be outside the "domain" code. This is key to make your domain code usable with different UIs (command-line, text-based (curse etc), GUI, web, whatever), but also, quite simply, to make sure your domain code is easily testable in isolation (unit testing...).
I'm writing some code for an esp8266 micro controller using micro-python and it has some different class as well as some additional methods in the standard built in classes. To allow me to debug on my desktop I've built some helper classes so that the code will run. However I've run into a snag with micro-pythons time function which has a time.sleep_ms method since the standard time.sleep method on micropython does not accept floats. I tried using the following code to extend the built in time class but it fails to import properly. Any thoughts?
class time(time):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
def sleep_ms(self, ms):
super().sleep(ms/1000)
This code exists in a file time.py. Secondly I know I'll have issues with having to import time.time that I would like to fix. I also realize I could call this something else and put traps for it in my micro controller code however I would like to avoid any special functions in what's loaded into the controller to save space and cycles.
You're not trying to override a class, you're trying to monkey-patch a module.
First off, if your module is named time.py, it will never be loaded in preference to the built-in time module. Truly built-in (as in compiled into the interpreter core, not just C extension modules that ship with CPython) modules are special, they are always loaded without checking sys.path, so you can't even attempt to shadow the time module, even if you wanted to (you generally don't, and doing so is incredibly ugly). In this case, the built-in time module shadows you; you can't import your module under the plain name time at all, because the built-in will be found without even looking at sys.path.
Secondly, assuming you use a different name and import it for the sole purpose of monkey-patching time (or do something terrible like adding the monkey patch to a custom sitecustomize module, it's not trivial to make the function truly native to the monkey-patched module (defining it in any normal way gives it a scope of the module where it was defined, not the same scope as other functions from the time module). If you don't need it to be "truly" defined as part of time, the simplest approach is just:
import time
def sleep_ms(ms):
return time.sleep(ms / 1000)
time.sleep_ms = sleep_ms
Of course, as mentioned, sleep_ms is still part of your module, and carries your module's scope around with it (that's why you do time.sleep, not just sleep; you could do from time import sleep to avoid qualifying it, but it's still a local alias that might not match time.sleep if someone else monkey-patches time.sleep later).
If you want to make it behave like it's part of the time module, so you can reference arbitrary things in time's namespace without qualification and always see the current function in time, you need to use eval to compile your code in time's scope:
import time
# Compile a string of the function's source to a code object that's not
# attached to any scope at all
# The filename argument is garbage, it's just for exception traceback
# reporting and the like
code = compile('def sleep_ms(ms): sleep(ms / 1000)', 'time.py', 'exec')
# eval the compiled code with a scope of the globals of the time module
# which both binds it to the time module's scope, and inserts the newly
# defined function directly into the time module's globals without
# defining it in your own module at all
eval(code, vars(time))
del code, time # May as well leave your monkey-patch module completely empty
I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.
I've looked at several things:
Method name parsing: the Main module would import the plugin module
and scan it for method names that match a certain format. For
example, it might add the download_file_command(file) method to its
dict as "download file" -> download_file_command. However, getting a
concise, easy-to-type command name (say, "dl") requires that the
function's name also be short, which isn't good for code
readability. It also requires the plugin developer to conform to a
precise naming format.
Cross-module decorators: decorators would let
the plugin developer name his function whatever he wants and simply
add something like #Main.register("dl"), but they would necessarily
require that I both modify another module's namespace and keep
global state in the Main module. I understand this is very bad.
Same-module decorators: using the same logic as above, I could add a
decorator that adds the function's name to some command name->function mapping local to the
plugin module and retrieve the mapping to the Main module with an
API call. This requires that certain methods always be present or
inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time
thereafter.
Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.
Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.
Building or Standing on the first part of #ericstalbot's answer, you might find it convenient to use a decorator like the following.
################################################################################
import functools
def register(command_name):
def wrapped(fn):
#functools.wraps(fn)
def wrapped_f(*args, **kwargs):
return fn(*args, **kwargs)
wrapped_f.__doc__ += "(command=%s)" % command_name
wrapped_f.command_name = command_name
return wrapped_f
return wrapped
################################################################################
#register('cp')
def copy_all_the_files(*args, **kwargs):
"""Copy many files."""
print "copy_all_the_files:", args, kwargs
################################################################################
print "Command Name: ", copy_all_the_files.command_name
print "Docstring : ", copy_all_the_files.__doc__
copy_all_the_files("a", "b", keep=True)
Output when run:
Command Name: cp
Docstring : Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}
User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:
def a():
return 1
a.command_name = 'get_one'
Then, in your module you could build a mapping like this:
import inspect #from standard library
import plugin
mapping = {}
for v in plugin.__dict__.itervalues():
if inspect.isfunction(v) and v.hasattr('command_name'):
mapping[v.command_name] = v
To read about arbitrary attributes for user-defined functions see the docs
There are two parts in a plugin system:
Discover plugins
Trigger some code execution in a plugin
The proposed solutions in your question address only the second part.
There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:
plugins = some_package.plugin_for_your_app
another_plugin_module
# ...
To implement loading of the plugin modules:
plugins = [importlib.import_module(name) for name in config.get("plugins")]
To get a dictionary: command name -> function:
commands = {name: func
for plugin in plugins
for name, func in plugin.get_commands().items()}
Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.
For example, some_plugin.py (full source):
def f(a, b):
return a + b
def get_commands():
return {"add": f, "multiply": lambda x,y: x*y}
It defines two commands add, multiply.
I manage a fairly large python-based quantum chemistry suite, PyQuante. I'm currently struggling with how to set various defaults so that users can choose among different options at runtime.
For example, I have three different methods for computing electron repulsion integrals. Let's call them a,b,c. I used to simply pick the one I liked best (say, c), and have that hard-wired into the module that computes these integrals.
I have now modified this to use a module, Defaults.py, that contains all such hard-wires. But this is set at compile/install time. I would now like users to be able to override these options at runtime, say, using a .pyquanterc.py file.
In my integral routines, I currently have something like
from Defaults import integral_method
I know about dictionaries, and the .update() method. But I don't know how I would use this in real life. My defaults module looks like
integral_method = c
should I modify the end of Defaults.py to look for a .pythonrc.py file and override these values? E.g.
if os.path.exists('$HOME/.pythonrc.py'): do_something
If so, what should do_something look like?
With your current setup, the user can change the default functions in his scripts quite easily:
import Defaults
Defaults.integral_method = somefunc
If the user adds this to his script, all your modules that use integral_method from Defaults will use somefunc to calculate integrals.
I might do this via a factory class.
class IntegralSolver:
"""
Factory class containing methods for solving integrals.
>>> solver = IntegralSolver("method1")
>>> solver(x)
# solution via method1
Can also be used directly:
>>> IntegralSolver.method2(x)
# solution via method2
"""
def __init__(self, method):
self.__call__ = getattr(self, method)
#staticmethod
def method1(x):
return method1_solution
#staticmethod
def method2(x):
return method2_solution
It really depends on how your user runs the toolset. If they twiddle the python code each time, just setting a block at the top labeled OPTIONS should be good. If they run it off the command line, use the argparse library to allow them to switch options on the command line. Perhaps have it read the options out of a file with configParser to read a default file with your options, and if the user sets it, an additional file with their options.