I'm using SQLAlchemy and am trying to import the function group_by into the interpreter, but I can't seem to find it. Is there an easy way to search the module tree to see where this function lives?
Of course I've tried from sqlalchemy import + tab and searching manually, but at each tree level there are too many options to check.
The easy way to do this is to step outside the interpreter and just search the docs.
Meanwhile, it looks to me like there is no such function to import. There are group_by methods on sqlalchemy.orm.query.Query and sqlalchemy.sql.expression.[Compound]Select[Base] objects.
But if you really want to recursively walk through all the modules in a package looking for a name, here's how you'd do it:
import inspect
def find_name(package, name):
if hasattr(package, name):
yield package
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
yield from find_name(submodule, name)
For Python 3.2 or earlier, you need to replace the yield from with a loop:
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
for result in find_name(submodule, name):
yield result
And if you just want the first result, instead of all results, you can just return instead of yielding:
def find_name(package, name):
if hasattr(package, name):
return package
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
result = find_name(submodule, name)
if result:
return result
try printing SQLAlchemy.__all__ which will return you a list of all functions in that module which are publicly available.
Related
I would like to customize the behavior of my module when it is imported.
For example, let say I want my module to print an incremented number each time another file use import my_module. And when from my_module import some_string is used, it should print "some_string".
How could I do that?
I read several questions here and there but this does not seems to work.
# my_module.py
import sys
class MyImporter:
def find_module(self, module_name, package_path):
print(module_name, package_path)
return self
def load_module(self, module_name):
print(module_name)
return self
sys.meta_path.append(MyImporter())
# file.py
import my_module # Nothing happens
What you're asking for is to have Python work not like Python. Whenever it imports a module it parses and executes the 'opened' code only once so it can pick up the definitions, functions, classes, etc. - every subsequent import of the module just references the cached & parsed first import.
That's why even if you put something like vars()["counter"] = vars().get("counter", 0) + 1 at your module's 'root', the counter will never go above 1 indicating that the module was indeed executed only once. You can force module reload using reload() (or importlib.reload() on Python 3.6+) but then you'd lose your counter if you keep it in the module itself.
Of course, you can have an external counter to be called when your module is imported, but that would have to be a contract with the users of your module at which point the question becomes - can't you just contract your users to call a function to increase your counter whenever they import your module instead of having to reload it for you to capture the count? Reloading a module will also make it have a potentially different state in every context it was reloaded which will make Python behave unexpectedly and should be avoided at any cost.
So, a short answer would be - no, you cannot do that and you should not attempt to do it. If you want something that doesn't work like Python - use something that isn't Python.
However... If you have a really, REALLY good reason to do this (and you don't!) and you don't mind hacking how Python fundamentally behaves (and you should mind) then you might attempt to do this by wrapping the built-in import and checking whenever it gets fired for your module. Something like:
your_module.py:
# HERE BE DRAGONS!!!
import sys
try:
import builtins # Python 3.4+
except ImportError:
import __builtin__ as builtins # Python 2.6+
__builtin_import__ = builtins.__import__ # store a reference to the built-in import
def __custom_import__(name, *args, **kwargs):
# execute builtin first so that the import fails if badly requested
ret = __builtin_import__(name, *args, **kwargs)
if ret is sys.modules[__name__]: # we're trying to load this module
if len(args) > 1 and args[2]: # using the `from your_module import whatever` form
if "some_string" in args[2]: # if some_string is amongst requested properties
print("some_string")
else: # using the `import your_module` form...
print_counter() # increase and print the latest count
return ret # return back the actual import result
builtins.__import__ = __custom_import__ # override the built-in import with our method
counter = 0
# a convinience function, you can do all of this through the `__custom_import__` function
def print_counter():
global counter
counter += 1
print(counter)
print_counter() # call it immediately on the first import to print out the counter
some_string = "I'm holding some string value" # since we want to import this
# HAVE I FORGOT TO TELL YOU NOT TO DO THIS? WELL, DON'T!!!
Keep in mind that this will not account for the first import (be it in the pure import your_module or in the from your_module import whatever form) as the import override won't exist until your module is loaded - that's why it calls print_counter() immediately in hope that the first import of the module was in the form of import your_module and not in the from..import form (if not it will wrongly print out the count instead of some_string the first time). To solve the first-import issue, you can move this 'ovverride' to the __init__.py in the same folder so that the override loads before your module starts and then delegate the counter change / some_string print to the module once loaded, just make sure you do your module name check properly in that case (you need to account for the package as well) and make sure it doesn't automatically execute the counter.
You also, technically, don't need the some_string property at all - by moving the execution of the built-in import around you can do your from..import check first, find the position of some_string in args[2] and pop it before calling the builtin import, then return None in the same position once executed. You can also do your printing and counter incrementing from within the overriden import function.
Again, for the love of all things fluffy and the poor soul who might have to rely on your code one day - please don't do this!
Actually, it does look like it's possible to do what you're looking for in python3.5. It's probably a bad idea, and I've carefully written my code to demonstrate the concept without being polished enough to use as-is, because I'd think carefully before doing something like this in a production project.
If you need to look at a more-or-less production example of this, take a look at the SelfWrapper class in the sh module.
Meanwhile, you can override your own entry in sys.modules to be a subclass of Module. Then you can override getattribute and detect accesses to attributes.
As best I can tell:
Every subsiquent import of the module references spec so you could probably count accesses to spec to count total imports
Each from foo import bar accesses bar as an attribute. I don't think you can distinguish between "from foo import bar" and "import foo; foo.bar"
import sys, types
class Wrapper(types.ModuleType):
def __getattribute__(self, attr):
print(attr)
return super().__getattribute__(attr)
test = "test"
sys.modules[__name__].__class__ = Wrapper
Here is how you can dynamically import modules-
from importlib import import_module
def import_from(module, name):
module = import_module(module, name)
return getattr(module, name)
and use it like this-
funcObj = import_from("<file_name>", "<method_name>")
response = funcObj(arg1,arg2)
I'm trying to put together a system that will handle lazy-loading of modules that don't explicitly exist. Basically I have an http server with a number of endpoints that I don't know ahead of time that I would like to programmatically offer for import. These modules would all have a uniform method signature, they just wouldn't exist ahead of time.
import lazy.route as test
import lazy.fake as test2
test('Does this exist?') # This sends a post request.
test2("This doesn't exist.") # Also sends a post request
I can handle all the logic I need around these imports with a uniform decorator, I just can't find any way of "decorating" imports in python, or actually interacting with them in any kind of programmatic way.
Does anyone have experience with this? I've been hunting around, and the closest thing I've found is the ast module, which would lead to a really awful kind of hacky implementation in my current under my current understanding (something like finding all import statements and manually over-writing the import function)
Not looking for a handout, just a piece of the python codebase to start looking at, or an example of someone that's done something similar.
I got a little clever in my googling and managed to find a PEP that specifically addressed this issue, it just happens to be relatively unknown, probably because the subset of reasonable uses for this is pretty narrow.
I found an excellent piece of example code showing off the new sys.meta_path implementation. I've posted it below for information on how to dynamically bootstrap your import statements.
import sys
class VirtualModule(object):
def hello(self):
return 'Hello World!'
class CustomImporter(object):
virtual_name = 'my_virtual_module'
def find_module(self, fullname, path):
"""This method is called by Python if this class
is on sys.path. fullname is the fully-qualified
name of the module to look for, and path is either
__path__ (for submodules and subpackages) or None (for
a top-level module/package).
Note that this method will be called every time an import
statement is detected (or __import__ is called), before
Python's built-in package/module-finding code kicks in."""
if fullname == self.virtual_name:
# As per PEP #302 (which implemented the sys.meta_path protocol),
# if fullname is the name of a module/package that we want to
# report as found, then we need to return a loader object.
# In this simple example, that will just be self.
return self
# If we don't provide the requested module, return None, as per
# PEP #302.
return None
def load_module(self, fullname):
"""This method is called by Python if CustomImporter.find_module
does not return None. fullname is the fully-qualified name
of the module/package that was requested."""
if fullname != self.virtual_name:
# Raise ImportError as per PEP #302 if the requested module/package
# couldn't be loaded. This should never be reached in this
# simple example, but it's included here for completeness. :)
raise ImportError(fullname)
# PEP#302 says to return the module if the loader object (i.e,
# this class) successfully loaded the module.
# Note that a regular class works just fine as a module.
return VirtualModule()
if __name__ == '__main__':
# Add our import hook to sys.meta_path
sys.meta_path.append(CustomImporter())
# Let's use our import hook
import my_virtual_module
print my_virtual_module.hello()
The full blog post is here
I am looking for all the functions that have a parameter called adjustable.
One of these function is matplotlib.pyplot.figure ().add_axes. help(matplotlib.pyplot.figure ().add_axes) describes that parameter, that can be present in kwargs dictionary.
I tried pydoc.apropos
from pydoc import apropos
In [8]: apropos ('adjustable')
No handlers could be found for logger "OpenGL.Tk"
In [9]:
This is all what it returned apropos(key).
One brute force way to find what I am looking for is to grep the source code of python from /usr/share/. But I need to do it from the current python environment (only what is present loaded in evaluator).
I just noticed you're using IPython, if so, there's an extension called grasp which implements its own version of apropos that may be useful here.
The documentation even uses matplotlib in its example.
apropos only searches the one-line summaries of all modules, not each docstring of each single function, as that wouldn't be feasable (already importing every single module as apropos does and searching it's docstring is quite some overhead).
And even if it would, in the case of add_axes it wouldn't find anything because your search term isn't part of the one-line description.
I'm affraid if you need such functionality, you'd need to implement it yourself.
Here is a quickly written function that walks all current packages in the specified path, using ast to find matching parameters, and returning (filename, funcname, line_no) for each match.
import ast
import pkgutil
import os.path
class FindParameter(ast.NodeVisitor):
def __init__(self, parameter):
self.parameter = parameter
self.found = []
def visit_FunctionDef(self, node):
for arg in node.args.args:
if getattr(arg, 'id', None) == self.parameter:
self.found.append(node)
def apropos(parameter, path=None):
paramFinder = FindParameter(parameter)
for importer, modname, is_package in pkgutil.iter_modules(path=path):
try:
loader = importer.find_module(modname)
loader.get_code()
if loader.source:
tree = ast.parse(loader.source, filename=loader.filename)
paramFinder.visit(tree)
except SyntaxError:
pass # avoid any broken code in the current path
return [ (loader.filename, found.name, found.lineno) for found in paramFinder.found ]
all_selfs = apropos('self')
I am building a very basic platform in the form of a Python 2.7 module. This module has a read-eval-print loop where entered user commands are mapped to function calls. Since I am trying to make it easy to build plugin modules for my platform, the function calls will be from my Main module to an arbitrary plugin module. I'd like a plugin builder to be able to specify the command that he wants to trigger his function, so I've been looking for a Pythonic way to remotely enter a mapping in the command->function dict in the Main module from the plugin module.
I've looked at several things:
Method name parsing: the Main module would import the plugin module
and scan it for method names that match a certain format. For
example, it might add the download_file_command(file) method to its
dict as "download file" -> download_file_command. However, getting a
concise, easy-to-type command name (say, "dl") requires that the
function's name also be short, which isn't good for code
readability. It also requires the plugin developer to conform to a
precise naming format.
Cross-module decorators: decorators would let
the plugin developer name his function whatever he wants and simply
add something like #Main.register("dl"), but they would necessarily
require that I both modify another module's namespace and keep
global state in the Main module. I understand this is very bad.
Same-module decorators: using the same logic as above, I could add a
decorator that adds the function's name to some command name->function mapping local to the
plugin module and retrieve the mapping to the Main module with an
API call. This requires that certain methods always be present or
inherited though, and - if my understanding of decorators is correct - the function will only register itself the first time it is run and will unnecessarily re-register itself every subsequent time
thereafter.
Thus, what I really need is a Pythonic way to annotate a function with the command name that should trigger it, and that way can't be the function's name. I need to be able to extract the command name->function mapping when I import the module, and any less work on the plugin developer's side is a big plus.
Thanks for the help, and my apologies if there are any flaws in my Python understanding; I'm relatively new to the language.
Building or Standing on the first part of #ericstalbot's answer, you might find it convenient to use a decorator like the following.
################################################################################
import functools
def register(command_name):
def wrapped(fn):
#functools.wraps(fn)
def wrapped_f(*args, **kwargs):
return fn(*args, **kwargs)
wrapped_f.__doc__ += "(command=%s)" % command_name
wrapped_f.command_name = command_name
return wrapped_f
return wrapped
################################################################################
#register('cp')
def copy_all_the_files(*args, **kwargs):
"""Copy many files."""
print "copy_all_the_files:", args, kwargs
################################################################################
print "Command Name: ", copy_all_the_files.command_name
print "Docstring : ", copy_all_the_files.__doc__
copy_all_the_files("a", "b", keep=True)
Output when run:
Command Name: cp
Docstring : Copy many files.(command=cp)
copy_all_the_files: ('a', 'b') {'keep': True}
User-defined functions can have arbitrary attributes. So you could specify that plug-in functions have an attribute with a certain name. For example:
def a():
return 1
a.command_name = 'get_one'
Then, in your module you could build a mapping like this:
import inspect #from standard library
import plugin
mapping = {}
for v in plugin.__dict__.itervalues():
if inspect.isfunction(v) and v.hasattr('command_name'):
mapping[v.command_name] = v
To read about arbitrary attributes for user-defined functions see the docs
There are two parts in a plugin system:
Discover plugins
Trigger some code execution in a plugin
The proposed solutions in your question address only the second part.
There many ways to implement both depending on your requirements e.g., to enable plugins, they could be specified in a configuration file for your application:
plugins = some_package.plugin_for_your_app
another_plugin_module
# ...
To implement loading of the plugin modules:
plugins = [importlib.import_module(name) for name in config.get("plugins")]
To get a dictionary: command name -> function:
commands = {name: func
for plugin in plugins
for name, func in plugin.get_commands().items()}
Plugin author can use any method to implement get_commands() e.g., using prefixes or decorators — your main application shouldn't care as long as get_commands() returns the command dictionary for each plugin.
For example, some_plugin.py (full source):
def f(a, b):
return a + b
def get_commands():
return {"add": f, "multiply": lambda x,y: x*y}
It defines two commands add, multiply.
In a Python package, I have a string containing (presumably) the name of a subpackage. From that subpackage, I want to retrieve a tuple of constants...I'm really not even sure how to proceed in doing this, though.
#!/usr/bin/python
"" The Alpha Package
Implements functionality of a base package under the 'alpha' namespace
""
def get_params(packagename):
# Here, I want to get alpha.<packagename>.REQUIRED_PARAMS
pass
So, later in my code I might have:
#!/usr/bin/python
import alpha
alpha.get_params('bravo') # should return alpha.bravo.REQUIRED_PARAMS
alpha.get_params('charlie') # should return alpha.charlie.REQUIRED_PARAMS
If I correctly understand what you want, I think something roughly like this should work:
def get_params(packagename):
module = __import__('alpha.%s' % packagename)
return module.__dict__['REQUIRED_PARAMS']