My Python package depends on an external library for a few of it's functions. This is a non-Python package and can be difficult to install, so I'd like users to still be able to use my package but have it fail when using any functions that depend on this non-Python package.
What is the standard practice for this? I could only import the non-Python package inside the methods that use it, but I really hate doing this
My current setup:
myInterface.py
myPackage/
--classA.py
--classB.py
The interfaces script myInterface.py imports classA and classB and classB imports the non-Python package. If the import fails I print a warning. If myMethod is called and the package isn't installed there will be some error downstream but I do not catch it anywhere, nor do I warn the user.
classB is imported every time the interface script is called so I can't have anything fail there, which is why I included the pass. Like I said above, I could import inside the method and have it fail there, but I really like keeping all of my imports in one place.
From classB.py
try:
import someWeirdPackage
except ImportError:
print("Cannot import someWeirdPackage")
pass
class ClassB():
...
def myMethod():
swp = someWeirdPackage()
...
If you are only importing one external library, I would go for something along these lines:
try:
import weirdModule
available = True
except ImportError:
available = False
def func_requiring_weirdmodule():
if not available:
raise ImportError('weirdModule not available')
...
The conditional and error checking is only needed if you want to give more descriptive errors. If not you can omit it and let python throw the corresponding error when trying to calling a non-imported module, as you do in your current setup.
If multiple functions do use weirdModule, you can wrap the checking into a function:
def require_weird_module():
if not available:
raise ImportError('weirdModule not available')
def f1():
require_weird_module()
...
def f2():
require_weird_module()
...
On the other hand, if you have multiple libraries to be imported by different functions, you can load them dynamically. Although it doesn't look pretty, python caches them and there is nothing wrong with it. I would use importlib
import importlib
def func_requiring_weirdmodule():
weirdModule = importlib.import_module('weirdModule')
Again, if multiple of your functions import complicated external modules you can wrap them into:
def import_external(name):
return importlib.import_module(name)
def f1():
weird1 = import_external('weirdModule1')
def f2():
weird2 = import_external('weirdModule2')
And last, you could create a handler to prevent importing the same module twice, something along the lines of:
class Importer(object):
__loaded__ = {}
#staticmethod
def import_external(name):
if name in Importer.__loaded__:
return Importer.__loaded__[name]
mod = importlib.import_module(name)
Importer.__loaded__[name] = mod
return mod
def f1():
weird = Importer.import_external('weird1')
def f2():
weird = Importer.import_external('weird1')
Although I'm pretty sure that importlib does caching behing the scenes and you don't really need for manual caching.
In short, although it does look ugly, there is nothing wrong with importing modules dynamically in python. In fact, a lot of libraries rely on this. On the other hand, if it is just for an special case of 3 methods accessing 1 external function, do use your approach or my first one in case you cant to add custom sception handling.
I'm not really sure that there's any best practice in this situation, but I would redefine the function if it's not supported:
def warn_import():
print("Cannot import someWeirdPackage")
try:
import someWeirdPackage
external_func = someWeirdPackage
except ImportError:
external_func = warn_import
class ClassB():
def myMethod(self):
swp = external_func()
b = ClassB()
b.myMethod()
You can create two separate classes for the two cases. The first will be used when the the package exist . The second will used when the package does not exist.
class ClassB1():
def myMethod(self):
print("someWeirdPackage exist")
# do something
class ClassB2(ClassB1):
def myMethod(self):
print("someWeirdPackage does not exist")
# do something or raise Exception
try:
import someWeirdPackage
class ClassB(ClassB1):
pass
except ImportError:
class ClassB(ClassB2):
pass
You can also use given below approach to overcome the problem that you're facing.
class UnAvailableName(object):
def __init__(self, name):
self.target = name
def __getattr_(self, attr):
raise ImportError("{} is not available.".format(attr))
try:
import someWeirdPackage
except ImportError:
print("Cannot import someWeirdPackage")
someWeirdPackage = someWeirdPackage("someWeirdPackage")
class ClassB():
def myMethod():
swp = someWeirdPackage.hello()
a = ClassB()
a.myMethod()
Related
As the doc "Where to patch" says, we need to patch where an object is looked up, (not where it's defined); so I understand that it's not possible to - let's say - create a reusable patch for a particular path
Imagine you have several modules importing an object you'd like to mock
# file_a.py
from foo.goo.hoo import settings
# file_b.py
from foo.goo.hoo import settings
# file_c.py
from foo.goo.hoo import settings
I was wondering if there is a way to create a decorator such as:
#mock_settings
def test_whatever(self, settings_mock):
...
instead of this solution:
#patch("some_module.file_a.settings")
def test_whatever(self, settings_mock):
...
#patch("some_module.file_b.settings")
def test_whatever(self, settings_mock):
...
#patch("some_module.file_c.settings")
def test_whatever(self, settings_mock):
...
As mentioned in the question, to patch an object you have to patch its reference in the module to be tested (in case it is imported using from ...import).
To have it patched in several modules, you can patch all of these modules with the same mock, and use that mock. If you know in advance which modules you want to patch, you can just do this. If you don't know them in advance, you have to try to patch the object in all loaded modules -- this may get a bit more complicated.
I will show an example using pytest and a pytest fixture, as this is more compact; you could wrap that in a decorator for usage in unittest, but that will not change the basics. Consider we have a class that needs to be mocked in several modules:
class_to_mock.py
class ClassToMock:
def foo(self, msg):
return msg
module1.py
from class_to_mock import ClassToMock
def do_something():
inst = ClassToMock()
return inst.foo("module1")
module2.py
from class_to_mock import ClassToMock
def do_something_else():
inst = ClassToMock()
return inst.foo("module2")
You can now write a fixture that mocks the class in all of these modules at once (here using pytest-mock for simplicity):
#pytest.fixture
def mocked_class(mocker):
mocked = Mock()
for module in ('module1', 'module2'):
mocker.patch(module + '.ClassToMock', mocked)
yield mocked
This can be used to test both modules:
def test_module1(mocked_class):
mocked_class.return_value.foo.return_value = 'mocked!'
assert module1.do_something() == 'mocked!'
def test_module2(mocked_class):
mocked_class.return_value.foo.return_value = 'mocked!'
assert module2.do_something_else() == 'mocked!'
If you want a generic version that mocks the class in all loaded modules, you can replace the fixture with something like this:
#pytest.fixture
def mocked_class(mocker):
mocked = Mock()
for name, module in list(sys.modules.items()):
if not inspect.ismodule(module):
continue
for cls_name, cls in module.__dict__.items():
try: # need that as inspect may raise for some modules
if inspect.isclass(cls) and cls_name == "ClassToMock":
mocker.patch(name + ".ClassToMock", mocked)
except Exception:
continue
yield mocked
This will work for this specific example - to generalize this, it has to consider more object types, the class shall be configurable, and there may be some more issues - opposed to the more simple version where you just enumerate the modules you want to patch, which will always work.
You could do something similar in unittest.setUp by putting the mock in an instance variable, though that is less elegant, because you are also responsible for stopping the mocking:
class ModulesTest(unittest.TestCase):
def setUp(self):
self.mocked_class = Mock()
self.mocks = []
for module in ('module1', 'module2'):
mocked = mock.patch(module + '.ClassToMock', self.mocked_class)
self.mocks.append(mocked)
mocked.start()
def tearDown(self):
for mocked in self.mocks:
mocked.stop()
def test_module1(self):
self.mocked_class.return_value.foo.return_value = 'mocked!'
assert module1.do_something() == 'mocked!'
And you can also wrap this in a decorator, to answer your original question at least partially:
def mocked_class_to_mock(f):
#wraps(f)
def _mocked_class_to_mock(*args, **kwargs):
mocked_class = Mock()
mocks = []
for module in ('module1', 'module2'):
mocked = mock.patch(module + '.ClassToMock', mocked_class)
mocks.append(mocked)
mocked.start()
kwargs['mocked_class'] = mocked_class # use a keyword arg for simplicity
f(*args, **kwargs)
for mocked in mocks:
mocked.stop()
return _mocked_class_to_mock
...
#mocked_class_to_mock
def test_module3(self, mocked_class):
mocked_class.return_value.foo.return_value = 'mocked!'
assert module3.do_something() == 'mocked!'
Of course, you can do the same with the more generic version, if needed.
Also note that I skipped the simpler case where the object is imported using import .... In this case, you have to patch the original module. In the generic fixture, you probably want to add that case always.
Does the interpreter somehow keep a timestamp of when a module is imported? Or is there an easy way of hooking into the import machinery to do this?
The scenario is a long-running Python process that at various points imports user-provided modules. I would like the process to be able to check "should I restart to load the latest code changes?" by checking the module file's timestamps against the time the module was imported.
Here's a way to automatically have an attribute (named _loadtime in the example code below) added to modules when they're imported. The code is based on Recipe 10.12 titled "Patching Modules on Import" in the book Python Cookbook, by David Beazley and Brian Jones, O'Reilly, 2013, which shows a technique that I adapted to do what you want.
For testing purposes I created this trivial target_module.py file:
print('in target_module')
Here's the example code:
import importlib
import sys
import time
class PostImportFinder:
def __init__(self):
self._skip = set() # To prevent recursion.
def find_module(self, fullname, path=None):
if fullname in self._skip: # Prevent recursion
return None
self._skip.add(fullname)
return PostImportLoader(self)
class PostImportLoader:
def __init__(self, finder):
self._finder = finder
def load_module(self, fullname):
importlib.import_module(fullname)
module = sys.modules[fullname]
# Add a custom attribute to the module object.
module._loadtime = time.time()
self._finder._skip.remove(fullname)
return module
sys.meta_path.insert(0, PostImportFinder())
if __name__ == '__main__':
import time
try:
print('importing target_module')
import target_module
except Exception as e:
print('Import failed:', e)
raise
loadtime = time.localtime(target_module._loadtime)
print('module loadtime: {} ({})'.format(
target_module._loadtime,
time.strftime('%Y-%b-%d %H:%M:%S', loadtime)))
Sample output:
importing target_module
in target_module
module loadtime: 1604683023.2491636 (2020-Nov-06 09:17:03)
I don't think there's any way to get around how hacky this is, but how about something like this every time you import? (I don't know exactly how you're importing):
import time
from types import ModuleType
# create a dictionary to keep track
# filter globals to exclude things that aren't modules and aren't builtins
MODULE_TIMES = {k:None for k,v in globals().items() if not k.startswith("__") and not k.endswith("__") and type(v) == ModuleType}
for module_name in user_module_list:
MODULE_TIMES[module_name] = time.time()
eval("import {0}".format(module_name))
And then you can reference this dictionary in a similar way later.
I have an application that will take in a string and later run it as arbitrary python code. I wish to validate this string before I attempt to run it and evaluate it for a few things:
Syntactically correct (this can be done via the compile(stringCode, foo.py, "execute") builtin)
All imports are available locally
Whether a class in the arbitrary code string inherits from a specific class
Whether the class from #3 also implements a specifically named method (so I can later call foo.bar() on the arbitrary code without too much hassle)
I've looked around at code objects, but they don't seem to be able to do anything unless I try to run the code directly, when I would rather validate that it works beforehand
You can use ast.parse to create a syntax tree of your string. Then you can iterate over the tree and validate whatever parse-time qualities you like.
As internet_user says, this will not tell you about the run-time qualities of your code; if modules are imported through a mechanism other than the usual import statement, those won't be validated. If your classes are dynamically changed to add or remove methods, you won't know that just from looking at the defs in their class definition.
Provided that you're not worried about any of that, here's a sample implementation:
import ast
import sys
import os
import imp
s = """
import math, gzip
from os import system
import numpy
import obviouslyFakeModuleName
class A(int):
def troz(self):
return 23
class B(str):
def zort(self):
return 42
"""
def can_be_imported(name):
try:
imp.find_module(name)
return True
except ImportError:
return False
def iter_nodes_by_type(code, type_or_types):
for node in ast.walk(code):
if isinstance(node, type_or_types):
yield node
def iter_imported_module_names(code):
for node in iter_nodes_by_type(code, ast.Import):
for alias in node.names:
yield alias.name
for node in iter_nodes_by_type(code, ast.ImportFrom):
yield node.module
def iter_globally_defined_classes(code):
for child in ast.iter_child_nodes(code):
if isinstance(child, ast.ClassDef):
yield child
def iter_methods(class_):
for node in ast.iter_child_nodes(class_):
if isinstance(node, ast.FunctionDef):
yield node
try:
code = ast.parse(s)
except SyntaxError:
print("That string is not valid Python.")
sys.exit(0)
#inspection of imports
for name in iter_imported_module_names(code):
if can_be_imported(name):
print("module {} is available for import.".format(name))
else:
print("module {} is not available for import.".format(name))
#inspection of classes
for class_ in iter_globally_defined_classes(code):
class_name = class_.name
base_class_names = [name.id for name in class_.bases]
function_names = [func.name for func in iter_methods(class_)]
print("Inspecting class {}...".format(class_name))
#we want to know if this class inherits directly from int
if "int" in base_class_names:
print(" Does inherit from int.")
else:
print(" Does not inherit from int.")
#and does it implement zort()?
if "zort" in function_names:
print(" Implements `zort`.")
else:
print(" Does not implement `zort`.")
Result:
module math is available for import.
module gzip is available for import.
module numpy is not available for import.
module obviouslyFakeModuleName is not available for import.
module os is available for import.
Inspecting class A...
Does inherit from int.
Does not implement `zort`.
Inspecting class B...
Does not inherit from int.
Implements `zort`.
I am using a custom logging module in my project. If it is not available, I'd like to substitute it with a dummy instead of raising an ImportError.
Here's the code which currently does that:
try:
import logger
except ImportError:
print 'Couldn\'t load logger'
class DummyLogger(object):
def __init__(self):
pass
def log(self, image):
pass
logger = DummyLogger()
I don't think it's an elegant solution. It works, sure, but it ain't nice. Is there a better way?
I would put the dummy implementation into a separate module, called dummy_loggger, and write:
try:
import logger
except ImportError:
import dummy_logger as logger
I've done this in the past with JSON parsers:
try:
import ujson as json # very fast but might not be available in some cases
except ImportError:
import json
You can make it more concise quite easily:
try:
import logger
except ImportError:
print 'Couldn\'t load logger'
class logger(object):
#classmethod
def log(cls, image):
pass
Note that, even in your current version, the empty __init__ should be removed -- it adds no value.
I have the following interface :
class Interface(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def run(self):
"""Run the process."""
return
I have a collections of modules that are all in the same directory. Each module contains a single class that implements my interface.
For example Launch.py :
class Launch(Interface):
def run(self):
pass
Let's say I have 20 modules, that implements 20 classes. I would like to be able to launch a module that would check if some of the classes do not implement the Interface.
I know I have to use :
issubclass(Launch, ProcessInterface) to know if a certain class implements my process interface.
introspection to get the class that is in my module.
import modules at runtime
I am just not sure how to do that.
I can manage to use issubclass inside a module.
But I cannot use issubclass if I am outside the module.
I need to :
get the list of all modules in the directory
get the class in each module
do issubclass on each class
I would need a draf of a function that could do that.
You're probably looking for something like this:
from os import listdir
from sys import path
modpath = "/path/to/modules"
for modname in listdir(modpath):
if modname.endswith(".py"):
# look only in the modpath directory when importing
oldpath, path[:] = path[:], [modpath]
try:
module = __import__(modname[:-3])
except ImportError:
print "Couldn't import", modname
continue
finally: # always restore the real path
path[:] = oldpath
for attr in dir(module):
cls = getattr(module, attr)
if isinstance(cls, type) and not issubclass(cls, ProcessInterface):
# do whatever