I have a Python module that tries to import a module, and if it fails, adds sets up a lightweight class to replicate some of the functionality as a fallback.
How can I write a unit test that covers both cases - when the module is present, and when the module is not present?
try:
from foo import bar
except ImportError:
class bar(object):
def doAThing(self):
return "blah"
For example, if bar is present, the unit tests will report that only the 2nd line is covered, and the rest of the module isn't. Python's coverage will report missing tests on the line with "except" to the end.
Modules are loaded from the sys.path array. If you clear that array then any non-core import will fail. The following commands were run in ipython in a virtual env with access to django
import sys
sys.path = []
import django
... ImportError ...
import os
... No ImportError ...
Your other alternative is to use two virtual environments. One which has the module and one which does not. That would make it slightly more difficult to run the tests though.
If this doesn't work then you can add a path to the start of the sys.path that has a module that will produce an ImportError when loaded. This works because the first match found when searching the sys.path is loaded, so this can replace the real module. The following is sufficient:
test.py
import sys
sys.path.insert(0, '.')
import foo
foo/__init__.py
raise ImportError()
Related
Real goal: I have a module that is common between two packages (say, bar and bar2). I want to use exact same test files for both cases so I want to change the test imports to not name the package explicitly. (Why? This can be useful during the process of extracting modules from a mega-package into separate packages.)
My idea was to add another module that imports a particular package and provides an "alias" for it. It almost worked, but I got a problem.
Initially I had:
# test.py:
from bar import some_function
If I do nothing magical, there will be two versions of test.py: one with from bar import some_function and another with from new_bar import some_function. I want to avoid this and have the test code files remain the same.
After I added indirection:
#foo.py:
import bar as baz
#test.py:
from .foo import baz # Works!
from .foo.baz import some_function # ModuleNotFoundError: No module named 'cur_dir.foo.baz'; 'cur_dir.foo' is not a package
I can make foo a package:
#foo/__init__.py:
import bar as baz
#test.py:
from .foo import baz # Works!
from .foo.baz import some_function # ModuleNotFoundError: No module named 'cur_dir.foo.baz'
The error changes a bit, but still remains.
I know that I can work around the problem by writing
# test.py:
from .foo import baz
some_function = baz.some_function
Is there any other way? I want my imports to be "normal".
Is there a way to create an "alias" for a package that can be used with the standard import mechanism?
The import statement only looks at actual modules and their paths, not at aliases inside the loaded modules. An actual module alias in Python's module registry, sys.modules, is required.
import sys
import os
sys.modules["os_alias"] = os # alias `os` to `os_alias`
import os_alias # alias import works now
from os_alias import chdir # even as from ... import ...
Once a module alias has been added to sys.modules, it is available for import in the entire application.
Note that module aliasing can lead to subtle bugs when submodules of aliased modules are loaded. In specific, if the submodules are not aliased explicitly, separate versions are created that are not identical. This means that any tests based on object identity, including isinstance(original.submodule.someclass(), alias.submodule.someclass), will fail if the versions are mixed.
To avoid this, you must alias all submodules of any aliased package.
I have a program (like a macro) that runs within a parent program and imports an API module from that program (lets call it foo). The problem is that that module only exists within that program, so I can't do things like run pydocmd outside the software because the script throws a ReferenceError. To aid in my own development I have create a type stub file, foo.pyi, in my project directory. What I would like to do is import that type stub as a normal Python file if the import fails, to provide dummy functions and properties. Something like:
try:
import foo
except ImportError:
from . import foo.pyi
This raises an error, however, as it's trying to import pyi from the foo library that does not exist in the project folder. The only other option I can think of is to have an identical copy of the .pyi file as, say "dummy_foo.py" but then I have to maintain two copies of the same file in one repo. I'd rather not do that.
I wrote this a while back; should still work I think:
import importlib
def import_stub(stubs_path, module_name):
sys.path_hooks.insert(0,
importlib.machinery.FileFinder.path_hook(
(importlib.machinery.SourceFileLoader, ['.pyi']))
)
sys.path.insert(0, stubs_path)
try:
return importlib.import_module(module_name)
finally:
sys.path.pop(0)
sys.path_hooks.pop(0)
I found this question, but my problem was about type checking. In my case pyi file contains class definition (so type hints are working), but the library doesn't. Solution is in checking typing.TYPE_CHECKING:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from library import _PrivateClass
def foo(x: "_PrivateClass"):
...
I have written a package (http://github.com/anntzer/parsedcmd) that runs with both Python2 and Python3. However, I had to write separate (py.test) unit tests for Python2 and Python3 (mainly because I want to test extra features of Python3, in particular keyword-only arguments), so I have a test_py2.py and a test_py3.py in a test subpackage. Now, if I run, say py.test2 mypkg, test_py2 passes, but test_py3 fails with a SyntaxError. Likewise, for py.test3 mypkg, test_py3 passes but test_py2 fails (I could make this one work though, it's just an issue of StringIO having moved to io).
I can design the test subpackage so that import mypkg.test only imports the proper version of the tests, but apparently py.test doesn't care -- it just sees two files matching test_* and grabs all tests in both of them, ignoring what __init__.py tells him to import.
So right now I have to do both py.test2 mypkg/test/test_py2.py and py.test3 mypkg/test/test_py3.py. Is there a way to set up the whole thing so that py.test2 mypkg and py.test3 mypkg would "just work"?
Thanks.
If you can then making your modules importable on all interpreters and skipping tests as appropriate is a common solution. Otherwise you can put the following as "conftest.py" into the test directory:
import sys
py3 = sys.version_info[0] >= 3
class DummyCollector(pytest.collect.File):
def collect(self):
return []
def pytest_pycollect_makemodule(path, parent):
bn = path.basename
if "py3" in bn and not py3 or ("py2" in bn and py3):
return DummyCollector(path, parent=parent)
This gets picked up a project-specific plugin and will properly ignore a test module with a filename containing a "py2" or "py3" substring on the wrong interpreter version. Of course you can refine it to rather have an explicit list directly in the conftest.py file instead of checking the filename etc.pp.
HTH, holger
You can put your tests in different packages and run only the tests in the appropriate package. Or you can load the appropriate test module in a script:
import sys, unittest
cur_version = sys.version_info
if cur_version[0] < 3:
import myApp.test.test_py2
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py2).run()
else:
import myApp.test.test_py3
unittest.TestLoader().loadTestsFromModule(myApp.test.test_py3).run()
Alternatively, use a setup.py file so you can run:
python setup.py test
and put the versioning logic in there:
versionedTestSuite = "parsedcmd.test.test_py2" # do something as above here
setup(name='parsedcmd',
...
test_suite=versionedTestSuite,
)
Is it possible to programmatically detect dependencies given a python project residing in SVN?
Here is a twist which adds some precision, and which might be useful if you find you're frequently checking dependencies of miscellaneous code:
Catches only import statements executed by the code being analyzed.
Automatically excludes all system-loaded modules, so you don't have to weed through it.
Also reports the symbols imported from each module.
Code:
import __builtin__
import collections
import sys
IN_USE = collections.defaultdict(set)
_IMPORT = __builtin__.__import__
def _myimport(name, globs=None, locs=None, fromlist=None, level=-1):
global IN_USE
if fromlist is None:
fromlist = []
IN_USE[name].update(fromlist)
return _IMPORT(name, globs, locs, fromlist, level)
# monkey-patch __import__
setattr(__builtin__, '__import__', _myimport)
# import and run the target project here and run the routine
import foobar
foobar.do_something()
# when it finishes running, dump the imports
print 'modules and symbols imported by "foobar":'
for key in sorted(IN_USE.keys()):
print key
for name in sorted(IN_USE[key]):
print ' ', name
Example foobar module:
import byteplay
import cjson
def _other():
from os import path
from sys import modules
def do_something():
import hashlib
import lxml
_other()
Output:
modules and symbols imported by "foobar":
_hashlib
array
array
byteplay
cStringIO
StringIO
cjson
dis
findlabels
foobar
hashlib
itertools
lxml
opcode
*
__all__
operator
os
path
sys
modules
types
warnings
Absolutely! If you are working from a UNIX or Linux shell, a simple combination of grep and awk would work; basically, all you want to do is search for lines containing the "import" keyword.
However, if you are working from any environment, you could just write a small Python script to do the searching for you (don't forget that strings are treated as immutable sequences, so you can do something like if "import" in line: ....
The one sticky spot, would be associating those imported modules to their package name (the first one that comes to mind is the PIL module, in Ubuntu it's provided by the python-imaging package).
Python code can import modules using runtime-constructed strings, so the only surefire way would be to run the code. Real-world example: when you open a database with SQLAlchemy's dbconnect, the library will load one or more db-api modules depending on the content of your database string.
If you're willing to run the code, here is a relatively simple way to do this by examining sys.modules when it finishes:
>>> from sys import modules
>>> import codeofinterest
>>> execute_code_of_interest()
>>> print modules
[ long, list, of, loaded, modules ]
Here, too, you should keep in mind that this could theoretically fail if execute_code_of_interest() modifies sys.modules, but I believe that's quite rare in production code.
I've got some python code in a library that attempts to load a simple value from a module that will exist for the applications that use this library
from somemodule import simplevalue
Normally, the application that uses the library will have the module file and everything works fine. However, in the unit tests for this library the module does not exist. I know that I can create a temporary file and add that file to my path at runtime, but I was curious if there is a way in python to load something in to memory that would allow the above import to work.
This is more of a curiosity, saying "add the module to your test path" is not helpful :P
It is. Use types.ModuleType to create a new module object, then add it to sys.modules:
sys.modules["somename"] = types.ModuleType("somename")
You can then do import somename. If you need to add classes or functions to it, import it before calling your test script, and just add functions to it:
def myfunc(x, y, z):
...
somename.myfunc = myfunc
It should go without saying, but just in case: this is largely an academic curiosity. It has some uses for testing, but other than that, stick to importing things the usual way.
Incidentally, how I know about this: I've come across the technique used in testing, to make a "stub" for the _winreg module on non-Windows systems. Here it is in use.
It isn't necessary to create a module. No Python code cares whether somemodule.simplevalue is actually a reference to an attribute of a module. To do so, a program would need to check the type of somemodule. Why bother?
Since you just want the single value from the module and are importing it into your own namespace, just define it:
simplevalue = 42
If you like, use try/except to try to import the real module first.
try:
from somemodule import simplevalue
except ImportError:
simplevalue = 42
If you are importing the entire module but only using one value, you can use a class to define a namespace.
try:
import somemodule
except ImportError:
class somemodule(object):
simplevalue = 42
Now somemodule.simplevalue refers to the value regardless of whether the module is available.
If you want other modules that also import somemodule to see your faked-up class-as-module, as you would in your unit test, just do this afterward:
import sys
sys.modules["somemodule"] = somemodule
Your system under test (sut in my example) needs to be able to cope with the fact that somemodule may not exist, so you can trap the ImportError:
#!/usr/bin/env python
try:
from somemodule import simplevalue
except ImportError, e:
if 'somemodule' in e:
'''We expect that to happen in the unittest but you should log something for when
this happens in production'''
def fn():
return simplevalue
Then you can inject a value in your unittest:
#!/usr/bin/env python
import unittest
import sut
class T(unittest.TestCase):
def test_fn(self):
sut.simplevalue = 42
self.assertEquals(42, sut.fn())
if __name__ == '__main__':
unittest.main()