I am looking for all the functions that have a parameter called adjustable.
One of these function is matplotlib.pyplot.figure ().add_axes. help(matplotlib.pyplot.figure ().add_axes) describes that parameter, that can be present in kwargs dictionary.
I tried pydoc.apropos
from pydoc import apropos
In [8]: apropos ('adjustable')
No handlers could be found for logger "OpenGL.Tk"
In [9]:
This is all what it returned apropos(key).
One brute force way to find what I am looking for is to grep the source code of python from /usr/share/. But I need to do it from the current python environment (only what is present loaded in evaluator).
I just noticed you're using IPython, if so, there's an extension called grasp which implements its own version of apropos that may be useful here.
The documentation even uses matplotlib in its example.
apropos only searches the one-line summaries of all modules, not each docstring of each single function, as that wouldn't be feasable (already importing every single module as apropos does and searching it's docstring is quite some overhead).
And even if it would, in the case of add_axes it wouldn't find anything because your search term isn't part of the one-line description.
I'm affraid if you need such functionality, you'd need to implement it yourself.
Here is a quickly written function that walks all current packages in the specified path, using ast to find matching parameters, and returning (filename, funcname, line_no) for each match.
import ast
import pkgutil
import os.path
class FindParameter(ast.NodeVisitor):
def __init__(self, parameter):
self.parameter = parameter
self.found = []
def visit_FunctionDef(self, node):
for arg in node.args.args:
if getattr(arg, 'id', None) == self.parameter:
self.found.append(node)
def apropos(parameter, path=None):
paramFinder = FindParameter(parameter)
for importer, modname, is_package in pkgutil.iter_modules(path=path):
try:
loader = importer.find_module(modname)
loader.get_code()
if loader.source:
tree = ast.parse(loader.source, filename=loader.filename)
paramFinder.visit(tree)
except SyntaxError:
pass # avoid any broken code in the current path
return [ (loader.filename, found.name, found.lineno) for found in paramFinder.found ]
all_selfs = apropos('self')
Related
I'm trying to debug Hy's use of bytecode. In particular, each time a module is imported, I want to see the path it was actually imported from, whether source or bytecode. Under the hood, Hy manages modules with importlib. It doesn't explicitly read or write bytecode; that's taken care of by importlib.machinery.SourceFileLoader. So it looks like what I want to do is monkey-patch Python's importing system to print the import path each time an import happens. How can I do that? I should be able to figure out how to do it for Hy once I understand how to do it for Python.
The easiest way that does not involve coding, is to start Python with two(!) verbose flags:
python -vv myscript.py
you'll get a lot of output, including all the import statements and all the files Python tries to access in order to load the module. In this example I have a simple python script that does import json:
lots of output!
[...]
# trying /tmp/json.cpython-310-x86_64-linux-gnu.so
# trying /tmp/json.abi3.so
# trying /tmp/json.so
# trying /tmp/json.py
# trying /tmp/json.pyc
# /usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc matches /usr/lib/python3.10/json/__init__.py
# code object from '/usr/lib/python3.10/json/__pycache__/__init__.cpython-310.pyc'
[...]
Alternatively but more complex: you could change the import statement itself. For that, you can overwrite __import__, which is invoked by the import statement itself. This way you could print out all the files import actually opens.
Seems like a good option would be to dynamically patch importlib.machinery.SourceFileLoader(fullname, path) and importlib.machinery.SourcelessFileLoader(fullname, path) to each print or write to a variable (a) the calling method and (b) the argument passed to the function.
If all you need to do is:
I want to see the path it was actually imported from, whether source or bytecode
And you don't need the import to "work properly", perhaps you can do a modified version of something like this. For example, I quickly modified their sample code to get this, I have not tested it so it may not work exactly, but it should get you on the right track:
# custom class to be the mock return value
class MockSourceLoader:
# mock SourceFileLoader method always returns that the module was loaded from source and its path
def SourceFileLoader(fullname, path):
return {"load type": "source", "fullname": fullname, "path": path}
def check_how_imported(monkeypatch):
# Any arguments may be passed and mock_get() will always return our
# mocked object
def mock_get(*args, **kwargs):
return MockSourceLoader
# apply the monkeypatch
monkeypatch.setattr(importlib.machinery, SourceFileLoader, SourceFileLoader)
You would of course provide a similar mock for Sourceless file loading for SourcelessFileLoader
For reference:
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourceFileLoader(fullname%2C%20path)%C2%B6
https://docs.python.org/3/library/importlib.html#:~:text=importlib.machinery.SourcelessFileLoader(fullname%2C%20path)
How can I mock the path (".test/locations.yml), because it does not exist in this project where I run my test. It exists in the CI environment.
I test my function get_matches_mr and then it says path location file not found
Do you have any idea?
Code
def read_location_file():
locations_file_path = os.path.join(".test/location.yml")
if not os.path.isfile(locations_file_path):
raise RuntimeError("Location file not found: " + locations_file_path)
with open(locations_file_path, "r") as infile:
location_file = yaml.safe_load(infile.read())
test_locations= location_file["paths"]
return test_locations
def get_matches_mr(self):
merge_request = MergeRequest()
locations = self.read_location_file()
data_locations= merge_request.get_matches(locations)
return data_locations
Like suggested in the comment, I would also say the best way to test such a scenario is to mock read_location_file. Because mocking the file system methods like os.path.join would mean that you limit the test to a certain implementation, which is a bad practice. The unit test suite should not know about the implementation detail, but only about the interfaces to be tested. Usually, in test driven development you write the test before the logic is implemented. This way you would not even know os.path.join is used.
The following code shows how to mock the read_location_file method. Assuming the class containing your two methods is called ClassToBeTested (replace with your actual class name).
import os.path
from class_to_test import ClassToBeTested
def test_function_to_test(tmpdir, monkeypatch):
def mockreturn():
return [
os.path.join(tmpdir, "sample/path/a"),
os.path.join(tmpdir, "sample/path/b"),
os.path.join(tmpdir, "sample/path/c"),
]
monkeypatch.setattr(ClassToBeTested, 'read_location_file', mockreturn)
c = ClassToBeTested()
assert c.get_matches_mr()
Note: I use the fixtures tmpdir and monkeypatch, which are both built-ins of pytest:
See this answer to find some info about tmpdir (in the linked answer I explained tmp_path, but it provides the same concept as tmpdir; the difference is tmp_path returns a pathlib.Path object, and tmpdir returns a py.path.local object).
monkeypatch is a pytest fixture that provides methods for mocking/patching of objects.
Split your function into two parts:
Finding and opening the correct file.
Reading and parsing the opened file.
Your function only does the second part; the call can be responsible for the first part.
def read_location_file(infile):
location_file = yaml.safe_load(infile.read())
test_locations= location_file["paths"]
return test_locations
Your test code can then use something like io.StringIO to verify that your function can parse it correctly.
def test_read_location():
assert read_location_file(io.StringIO("...")) == ...
Your production code will handle opening the file:
with open(location_file_path) as f:
locations = read_location_file(f)
I'm trying to put together a system that will handle lazy-loading of modules that don't explicitly exist. Basically I have an http server with a number of endpoints that I don't know ahead of time that I would like to programmatically offer for import. These modules would all have a uniform method signature, they just wouldn't exist ahead of time.
import lazy.route as test
import lazy.fake as test2
test('Does this exist?') # This sends a post request.
test2("This doesn't exist.") # Also sends a post request
I can handle all the logic I need around these imports with a uniform decorator, I just can't find any way of "decorating" imports in python, or actually interacting with them in any kind of programmatic way.
Does anyone have experience with this? I've been hunting around, and the closest thing I've found is the ast module, which would lead to a really awful kind of hacky implementation in my current under my current understanding (something like finding all import statements and manually over-writing the import function)
Not looking for a handout, just a piece of the python codebase to start looking at, or an example of someone that's done something similar.
I got a little clever in my googling and managed to find a PEP that specifically addressed this issue, it just happens to be relatively unknown, probably because the subset of reasonable uses for this is pretty narrow.
I found an excellent piece of example code showing off the new sys.meta_path implementation. I've posted it below for information on how to dynamically bootstrap your import statements.
import sys
class VirtualModule(object):
def hello(self):
return 'Hello World!'
class CustomImporter(object):
virtual_name = 'my_virtual_module'
def find_module(self, fullname, path):
"""This method is called by Python if this class
is on sys.path. fullname is the fully-qualified
name of the module to look for, and path is either
__path__ (for submodules and subpackages) or None (for
a top-level module/package).
Note that this method will be called every time an import
statement is detected (or __import__ is called), before
Python's built-in package/module-finding code kicks in."""
if fullname == self.virtual_name:
# As per PEP #302 (which implemented the sys.meta_path protocol),
# if fullname is the name of a module/package that we want to
# report as found, then we need to return a loader object.
# In this simple example, that will just be self.
return self
# If we don't provide the requested module, return None, as per
# PEP #302.
return None
def load_module(self, fullname):
"""This method is called by Python if CustomImporter.find_module
does not return None. fullname is the fully-qualified name
of the module/package that was requested."""
if fullname != self.virtual_name:
# Raise ImportError as per PEP #302 if the requested module/package
# couldn't be loaded. This should never be reached in this
# simple example, but it's included here for completeness. :)
raise ImportError(fullname)
# PEP#302 says to return the module if the loader object (i.e,
# this class) successfully loaded the module.
# Note that a regular class works just fine as a module.
return VirtualModule()
if __name__ == '__main__':
# Add our import hook to sys.meta_path
sys.meta_path.append(CustomImporter())
# Let's use our import hook
import my_virtual_module
print my_virtual_module.hello()
The full blog post is here
I'm using SQLAlchemy and am trying to import the function group_by into the interpreter, but I can't seem to find it. Is there an easy way to search the module tree to see where this function lives?
Of course I've tried from sqlalchemy import + tab and searching manually, but at each tree level there are too many options to check.
The easy way to do this is to step outside the interpreter and just search the docs.
Meanwhile, it looks to me like there is no such function to import. There are group_by methods on sqlalchemy.orm.query.Query and sqlalchemy.sql.expression.[Compound]Select[Base] objects.
But if you really want to recursively walk through all the modules in a package looking for a name, here's how you'd do it:
import inspect
def find_name(package, name):
if hasattr(package, name):
yield package
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
yield from find_name(submodule, name)
For Python 3.2 or earlier, you need to replace the yield from with a loop:
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
for result in find_name(submodule, name):
yield result
And if you just want the first result, instead of all results, you can just return instead of yielding:
def find_name(package, name):
if hasattr(package, name):
return package
for modulename, submodule in inspect.getmembers(package, inspect.ismodule):
result = find_name(submodule, name)
if result:
return result
try printing SQLAlchemy.__all__ which will return you a list of all functions in that module which are publicly available.
(Important: See update below.)
I'm trying to write a function, import_something, that will important certain modules. (It doesn't matter which for this question.) The thing is, I would like those modules to be imported at the level from which the function is called. For example:
import_something() # Let's say this imports my_module
my_module.do_stuff() #
Is this possible?
Update:
Sorry, my original phrasing and example were misleading. I'll try to explain my entire problem. What I have is a package, which has inside it some modules and packages. In its __init__.py I want to import all the modules and packages. So somewhere else in the program, I import the entire package, and iterate over the modules/packages it has imported.
(Why? The package is called crunchers, and inside it there are defined all kinds of crunchers, like CruncherThread, CruncherProcess, and in the future perhaps MicroThreadCruncher. I want the crunchers package to automatically have all the crunchers that are placed in it, so later in the program when I use crunchers I know it can tell exactly which crunchers I have defined.)
I know I can solve this if I avoid using functions at all, and do all imports on the main level with for loops and such. But it's ugly and I want to see if I can avoid it.
If anything more is unclear, please ask in comments.
Functions have the ability to return something to where they were called. Its called their return value :p
def import_something():
# decide what to import
# ...
mod = __import__( something )
return mod
my_module = import_something()
my_module.do_stuff()
good style, no hassle.
About your update, I think adding something like this to you __init__.py does what you want:
import os
# make a list of all .py files in the same dir that dont start with _
__all__ = installed = [ name for (name,ext) in ( os.path.splitext(fn) for fn in os.listdir(os.path.dirname(__file__))) if ext=='.py' and not name.startswith('_') ]
for name in installed:
# import them all
__import__( name, globals(), locals())
somewhere else:
import crunchers
crunchers.installed # all names
crunchers.cruncherA # actual module object, but you can't use it since you don't know the name when you write the code
# turns out the be pretty much the same as the first solution :p
mycruncher = getattr(crunchers, crunchers.installed[0])
You can monkey with the parent frame in CPython to install the modules into the locals for that frame (and only that frame). The downsides are that a) this is really quite hackish and b) sys._getframe() is not guaranteed to exist in other python implementations.
def importer():
f = sys._getframe(1) # Get the parent frame
f.f_locals["some_name"] = __import__(module_name, f.f_globals, f.f_locals)
You still have to install the module into f_locals, since import won't actually do that for you - you just supply the parent frame locals and globals for the proper context.
Then in your calling function you can have:
def foo():
importer() # Magically makes 'some_name' available to the calling function
some_name.some_func()
Are you looking for something like this?
def my_import(*names):
for name in names:
sys._getframe(1).f_locals[name] = __import__(name)
then you can call it like this:
my_import("os", "re")
or
namelist = ["os", "re"]
my_import(*namelist)
According to __import__'s help:
__import__(name, globals={}, locals={}, fromlist=[], level=-1) -> module
Import a module. The globals are only used to determine the context;
they are not modified. ...
So you can simply get the globals of your parent frame and use that for the __import__ call.
def import_something(s):
return __import__(s, sys._getframe(1).f_globals)
Note: Pre-2.6, __import__'s signature differed in that it simply had optional parameters instead of using kwargs. Since globals is the second argument in both cases, the way it's called above works fine. Just something to be aware of if you decided to use any of the other arguments.