We Keep Coding

Python 3 how to write unit tests for try except outside functions in modules

I would like to know how to write Python 3 unittest for try exceptblocks
that are defined outside of function definitions in Python's module.
Imagine that in package/module.py I have a block of code like:
import os
try:
CONSTANT = os.environ['NOT_EXISTING_KEY']
except KeyError:
CONSTANT = False
finally:
del os
(please don't mind the actual code, I know I could have used os.getenv('NOT_EXISTING_KEY', False)in this specific case, what I am interested in is really testing that the try-except block in a module (outside of a function) behaves as expected.
How can I write a unit test that checks that package.module.CONSTANT is set to the expected value?
In the unittest file (I use pytest) I have something like:
from package.module import CONSTANT
def test_constant_true():
assert CONSTANT == 'expected_value'
to test that if the try block executed correctly then CONSTANT is as expected.
I don't know, however, how to mock the import machinery so that the os.environ in the try block raises an exception and I can test that CONSTANT is set to False.
How can I do that?

You can use monkeypatch to set the environment variable, but you have to reload the module for the change to take effect:
from importlib import reload
from package import module
def test_constant_true(monkeypatch):
monkeypatch.setenv('MY_KEY', '42')
reload(module)
assert module.CONSTANT == '42'
def test_constant_false():
reload(module)
assert not module.CONSTANT
Given this content of package/module.py:
import os
try:
CONSTANT = os.environ['MY_KEY']
except KeyError:
CONSTANT = False

You could mock the environment using mock.patch.dict and import the value inside your unit test method. Like so:
from unittest import TestCase, mock
class YourTest(TestCase):
#mock.patch.dict('os.environ', {'NOT_EXISTING_KEY': 'value'})
def test_constant_key_defined(self, mocked):
""" Tests when the key is defined """
from package.module import CONSTANT
self.assertEqual(CONSTANT, 'value')
def test_constant_key_not_defined(self):
""" Tests when the key is not defined """
from package.module import CONSTANT
self.assertEqual(CONSTANT, 'value')
You may use the importlib.reload, like #mrbean-bremen's answer, which I am not familiar with.

How to customize a module import in Python?

I would like to customize the behavior of my module when it is imported.
For example, let say I want my module to print an incremented number each time another file use import my_module. And when from my_module import some_string is used, it should print "some_string".
How could I do that?
I read several questions here and there but this does not seems to work.
# my_module.py
import sys
class MyImporter:
def find_module(self, module_name, package_path):
print(module_name, package_path)
return self
def load_module(self, module_name):
print(module_name)
return self
sys.meta_path.append(MyImporter())
# file.py
import my_module # Nothing happens

What you're asking for is to have Python work not like Python. Whenever it imports a module it parses and executes the 'opened' code only once so it can pick up the definitions, functions, classes, etc. - every subsequent import of the module just references the cached & parsed first import.
That's why even if you put something like vars()["counter"] = vars().get("counter", 0) + 1 at your module's 'root', the counter will never go above 1 indicating that the module was indeed executed only once. You can force module reload using reload() (or importlib.reload() on Python 3.6+) but then you'd lose your counter if you keep it in the module itself.
Of course, you can have an external counter to be called when your module is imported, but that would have to be a contract with the users of your module at which point the question becomes - can't you just contract your users to call a function to increase your counter whenever they import your module instead of having to reload it for you to capture the count? Reloading a module will also make it have a potentially different state in every context it was reloaded which will make Python behave unexpectedly and should be avoided at any cost.
So, a short answer would be - no, you cannot do that and you should not attempt to do it. If you want something that doesn't work like Python - use something that isn't Python.
However... If you have a really, REALLY good reason to do this (and you don't!) and you don't mind hacking how Python fundamentally behaves (and you should mind) then you might attempt to do this by wrapping the built-in import and checking whenever it gets fired for your module. Something like:
your_module.py:
# HERE BE DRAGONS!!!
import sys
try:
import builtins # Python 3.4+
except ImportError:
import __builtin__ as builtins # Python 2.6+
__builtin_import__ = builtins.__import__ # store a reference to the built-in import
def __custom_import__(name, *args, **kwargs):
# execute builtin first so that the import fails if badly requested
ret = __builtin_import__(name, *args, **kwargs)
if ret is sys.modules[__name__]: # we're trying to load this module
if len(args) > 1 and args[2]: # using the `from your_module import whatever` form
if "some_string" in args[2]: # if some_string is amongst requested properties
print("some_string")
else: # using the `import your_module` form...
print_counter() # increase and print the latest count
return ret # return back the actual import result
builtins.__import__ = __custom_import__ # override the built-in import with our method
counter = 0
# a convinience function, you can do all of this through the `__custom_import__` function
def print_counter():
global counter
counter += 1
print(counter)
print_counter() # call it immediately on the first import to print out the counter
some_string = "I'm holding some string value" # since we want to import this
# HAVE I FORGOT TO TELL YOU NOT TO DO THIS? WELL, DON'T!!!
Keep in mind that this will not account for the first import (be it in the pure import your_module or in the from your_module import whatever form) as the import override won't exist until your module is loaded - that's why it calls print_counter() immediately in hope that the first import of the module was in the form of import your_module and not in the from..import form (if not it will wrongly print out the count instead of some_string the first time). To solve the first-import issue, you can move this 'ovverride' to the __init__.py in the same folder so that the override loads before your module starts and then delegate the counter change / some_string print to the module once loaded, just make sure you do your module name check properly in that case (you need to account for the package as well) and make sure it doesn't automatically execute the counter.
You also, technically, don't need the some_string property at all - by moving the execution of the built-in import around you can do your from..import check first, find the position of some_string in args[2] and pop it before calling the builtin import, then return None in the same position once executed. You can also do your printing and counter incrementing from within the overriden import function.
Again, for the love of all things fluffy and the poor soul who might have to rely on your code one day - please don't do this!

Actually, it does look like it's possible to do what you're looking for in python3.5. It's probably a bad idea, and I've carefully written my code to demonstrate the concept without being polished enough to use as-is, because I'd think carefully before doing something like this in a production project.
If you need to look at a more-or-less production example of this, take a look at the SelfWrapper class in the sh module.
Meanwhile, you can override your own entry in sys.modules to be a subclass of Module. Then you can override getattribute and detect accesses to attributes.
As best I can tell:
Every subsiquent import of the module references spec so you could probably count accesses to spec to count total imports
Each from foo import bar accesses bar as an attribute. I don't think you can distinguish between "from foo import bar" and "import foo; foo.bar"
import sys, types
class Wrapper(types.ModuleType):
def __getattribute__(self, attr):
print(attr)
return super().__getattribute__(attr)
test = "test"
sys.modules[__name__].__class__ = Wrapper

Here is how you can dynamically import modules-
from importlib import import_module
def import_from(module, name):
module = import_module(module, name)
return getattr(module, name)
and use it like this-
funcObj = import_from("<file_name>", "<method_name>")
response = funcObj(arg1,arg2)

Mock an entire module in python

I have an application that imports a module from PyPI.
I want to write unittests for that application's source code, but I do not want to use the module from PyPI in those tests.
I want to mock it entirely (the testing machine will not contain that PyPI module, so any import will fail).
Currently, each time I try to load the class I want to test in the unittests, I immediately get an import error. so I thought about maybe using
try:
except ImportError:
and catch that import error, then use command_module.run().
This seems pretty risky/ugly and I was wondering if there's another way.
Another idea was writing an adapter to wrap that PyPI module, but I'm still working on that.
If you know any way I can mock an entire python package, I would appreciate it very much.
Thanks.

If you want to dig into the Python import system, I highly recommend David Beazley's talk.
As for your specific question, here is an example that tests a module when its dependency is missing.
bar.py - the module you want to test when my_bogus_module is missing
from my_bogus_module import foo
def bar(x):
return foo(x) + 1
mock_bogus.py - a file in with your tests that will load a mock module
from mock import Mock
import sys
import types
module_name = 'my_bogus_module'
bogus_module = types.ModuleType(module_name)
sys.modules[module_name] = bogus_module
bogus_module.foo = Mock(name=module_name+'.foo')
test_bar.py - tests bar.py when my_bogus_module is not available
import unittest
from mock_bogus import bogus_module # must import before bar module
from bar import bar
class TestBar(unittest.TestCase):
def test_bar(self):
bogus_module.foo.return_value = 99
x = bar(42)
self.assertEqual(100, x)
You should probably make that a little safer by checking that my_bogus_module isn't actually available when you run your test. You could also look at the pydoc.locate() method that will try to import something, and return None if it fails. It seems to be a public method, but it isn't really documented.

While #Don Kirkby's answer is correct, you might want to look at the bigger picture. I borrowed the example from the accepted answer:
import pypilib
def bar(x):
return pypilib.foo(x) + 1
Since pypilib is only available in production, it is not suprising that you have some trouble when you try to unit test bar. The function requires the external library to run, therefore it has to be tested with this library. What you need is an integration test.
That said, you might want to force unit testing, and that's generally a good idea because it will improve the confidence you (and others) have in the quality of your code. To widen the unit test area, you have to inject dependencies. Nothing prevents you (in Python!) from passing a module as a parameter (the type is types.ModuleType):
try:
import pypilib # production
except ImportError:
pypilib = object() # testing
def bar(x, external_lib = pypilib):
return external_lib.foo(x) + 1
Now, you can unit test the function:
import unittest
from unittest.mock import Mock
class Test(unittest.TestCase):
def test_bar(self):
external_lib = Mock(foo = lambda x: 3*x)
self.assertEqual(10, bar(3, external_lib))
if __name__ == "__main__":
unittest.main()
You might disapprove the design. The try/except part is a bit cumbersome, especially if you use the pypilib module in several modules of your application. And you have to add a parameter to each function that relies on the external library.
However, the idea to inject a dependency to the external library is useful, because you can control the input and test the output of your class methods, even if the external library is not within your control. Especially if the imported module is stateful, the state might be difficult to reproduce in a unit test. In this case, passing the module as a parameter may be a solution.
But the usual way to deal with this situation is called dependency inversion principle (the D of SOLID): you should define the (abstract) boundaries of your application, ie what you need from the outside world. Here, this is bar and other functions, preferably grouped in one or many classes:
import pypilib
import other_pypilib
class MyUtil:
"""
All I need from outside world
"""
#staticmethod
def bar(x):
return pypilib.foo(x) + 1
#staticmethod
def baz(x, y):
return other_pypilib.foo(x, y) * 10.0
...
# not every method has to be static
Each time you need one of these functions, just inject an instance of the class in your code:
class Application:
def __init__(self, util: MyUtil):
self._util = util
def something(self, x, y):
return self._util.baz(self._util.bar(x), y)
The MyUtil class must be as slim as possible, but must remain abstract from the underlying library. It is a tradeoff. Obviously, Application can be unit tested (just inject a Mock instead of an instance of MyUtil) while, under some circumstances (like a PyPi library not available during tests, a module that runs inside a framework only, etc.), MyUtil can be only tested within an integration test. If you need to unit test the boundaries of your application, you can use #Don Kirkby's method.
Note that the second benefit, after unit testing, is that if you change the libraries you are using (deprecation, license issue, cost, ...), you just have to rewrite the MyUtil class, using some other libraries or coding it from scratch. Your application is protected from the wild outside world.
Clean Code by Robert C. Martin has a full chapter on the boundaries.
Summary Before using #Don Kirkby's method or any other method, be sure to define the boundaries of your application irrespective of the specific libraries you are using. This, of course, does not apply to the Python standard library...

For a more explicit and granular approach:
import unittest
from unittest.mock import MagicMock, patch
try:
import bogus_module
except ModuleNotFoundError:
bogus_module = MagicMock()
#patch.dict('sys.modules', bogus_module=bogus_module)
class PlatformTests(unittest.TestCase):
...
Using the patch.dict decorator gives you granular control: it only applies to the class / method it is applied to.

Making a copy of an entire namespace?

I'd like to make a copy of an entire namespace while replacing some functions with dynamically constructed versions.
In other words, starting with namespace (import tensorflow as tf), I want to make a copy of it, replace some functions with my own versions, and update __globals__ of all the symbols to stay within the new namespace. This needs to be done in topological order of dependency.
I started doing something like it here but now I'm starting to wonder if I'm reinventing the wheel. Care is needed to deal with circular dependencies in system modules, functions/types/objects need to be updated differently, etc.
Can anyone point to existing code that solves a similar task?

To patch a set of functions while importing second instances of a set of functions, you can override the standard Python import hook and apply the patches directly at import time. This will make sure that no other module will ever see the unpatched versions of any of the modules, so even if they import functions from another module directly by name, they will only see the patched functions. Here is a proof-of-concept implementation:
import __builtin__
import collections
import contextlib
import sys
#contextlib.contextmanager
def replace_import_hook(new_import_hook):
original_import = __builtin__.__import__
__builtin__.__import__ = new_import_hook
yield original_import
__builtin__.__import__ = original_import
def clone_modules(patches, additional_module_names=None):
"""Import new instances of a set of modules with some objects replaced.
Arguments:
patches - a dictionary mapping `full.module.name.symbol` to the new object.
additional_module_names - a list of the additional modules you want new instances of, without
replacing any objects in them.
Returns:
A dictionary mapping module names to the new patched module instances.
"""
def import_hook(module_name, *args):
result = original_import(module_name, *args)
if module_name not in old_modules or module_name in new_modules:
return result
# The semantics for the return value of __import__() are a bit weird, so we need some logic
# to determine the actual imported module object.
if len(args) >= 3 and args[2]:
module = result
else:
module = reduce(getattr, module_name.split('.')[1:], result)
for symbol, obj in patches_by_module[module_name].items():
setattr(module, symbol, obj)
new_modules[module_name] = module
return result
# Group patches by module name
patches_by_module = collections.defaultdict(dict)
for dotted_name, obj in patches.items():
module_name, symbol = dotted_name.rsplit('.', 1) # Only allows patching top-level objects
patches_by_module[module_name][symbol] = obj
try:
# Remove the old module instances from sys.modules and store them in old_modules
all_module_names = list(patches_by_module)
if additional_module_names is not None:
all_module_names.extend(additional_module_names)
old_modules = {}
for name in all_module_names:
old_modules[name] = sys.modules.pop(name)
# Re-import modules to create new patched versions
with replace_import_hook(import_hook) as original_import:
new_modules = {}
for module_name in all_module_names:
import_hook(module_name)
finally:
sys.modules.update(old_modules)
return new_modules
And here some test code for this implementation:
from __future__ import print_function
import math
import random
def patched_log(x):
print('Computing log({:g})'.format(x))
return math.log(x)
patches = {'math.log': patched_log}
cloned_modules = clone_modules(patches, ['random'])
new_math = cloned_modules['math']
new_random = cloned_modules['random']
print('Original log: ', math.log(2.0))
print('Patched log: ', new_math.log(2.0))
print('Original expovariate: ', random.expovariate(2.0))
print('Patched expovariate: ', new_random.expovariate(2.0))
The test code has this output:
Computing log(4)
Computing log(4.5)
Original log: 0.69314718056
Computing log(2)
Patched log: 0.69314718056
Original expovariate: 0.00638038735379
Computing log(0.887611)
Patched expovariate: 0.0596108277801
The first two lines of output result from these two lines in random, which are executed at import time. This demonstrates that random sees the patched function right away. The rest of the output demonstrates that the original math and random still use the unpatched version of log, while the cloned modules both use the patched version.
A cleaner way of overriding the import hook might be to use a meta import hook as defined in PEP 302, but providing a full implementation of that approach is beyond the scope of StackOverflow.

Instead of trying to make a copy of the contents of a module and patch everything in it to use the correct globals, you could trick Python into importing everything you want to copy a second time. This will give you a newly initialized copy of all modules, so it won't copy any global state the modules might have (not sure whether you would need that).
import importlib
import sys
def new_module_instances(module_names):
old_modules = {}
for name in module_names:
old_modules[name] = sys.modules.pop(name)
new_modules = {}
for name in module_names:
new_modules[name] = importlib.import_module(name)
sys.modules.update(old_modules)
return new_modules
Note that we first delete all modules we want to replace from sys.modules, so they all get import a second time, and the dependencies between these modules are set up correctly automatically. At the end of the function, we restore the original state of sys.modules, so everything else continues to see the original versions of these modules.
Here's an example:
>>> import logging.handlers
>>> new_modules = new_module_instances(['logging', 'logging.handlers'])
>>> logging_clone = new_modules['logging']
>>> logging
<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>
>>> logging_clone
<module 'logging' from '/usr/lib/python2.7/logging/__init__.pyc'>
>>> logging is logging_clone
False
>>> logging is logging.handlers.logging
True
>>> logging_clone is logging_clone.handlers.logging
True
The last three expressions show that the two versions of logging are different modules, and both versions of the handlers module use the correct version of the logging module.

To my mind, you can do this easily:
import imp, string
st = imp.load_module('st', *imp.find_module('string')) # copy the module
def my_upper(a):
return "a" + a
def my_lower(a):
return a + "a"
st.upper = my_upper
st.lower = my_lower
print string.upper("hello") # HELLO
print string.lower("hello") # hello
print st.upper("hello") # ahello
print st.lower("hello") # helloa
And when you call st.upper("hello"), it will result in "hello".
So, you don't really need to mess with globals.

How to patch a module's internal functions with mock?

By "internal function", I mean a function that is called from within the same module it is defined in.
I am using the mock library, specifically the patch decorators, in my unit tests. They're Django unit tests, but this should apply to any python tests.
I have one module with several functions, many of which call each other. For example (fictitious code, ignore the lack of decimal.Decimal):
TAX_LOCATION = 'StateName, United States'
def add_tax(price, user):
tax = 0
if TAX_LOCATION == 'StateName, UnitedStates':
tax = price * .75
return (tax, price+tax)
def build_cart(...):
# build a cart object for `user`
tax, price = add_tax(cart.total, cart.user)
return cart
These are part of a deeper calling chain (func1 -> func2 -> build_cart -> add_tax), all of which are in the same module.
In my unit tests, I'd like to disable taxes to get consistent results. As I see it, my two options are 1) patch out TAX_LOCATION (with an empty string, say) so that add_tax doesn't actually do anything or 2) patch out add_tax to simply return (0, price).
However, when I try to patch either of these the patch seems to work externally (I can import the patched part inside the test and print it out, getting expected values), but seems to have no effect internally (the results I get from the code behave as if the patch were not applied).
My tests are like this (again, fictitious code):
from mock import patch
from django.test import TestCase
class MyTests(TestCase):
#patch('mymodule.TAX_LOCATION', '')
def test_tax_location(self):
import mymodule
print mymodule.TAX_LOCATION # ''
mymodule.func1()
self.assertEqual(cart.total, original_price) # fails, tax applied
#patch('mymodule.add_tax', lambda p, u: (0, p))
def test_tax_location(self):
import mymodule
print mymodule.add_tax(50, None) # (0, 50)
mymodule.func1()
self.assertEqual(cart.total, original_price) # fails, tax applied
Does anyone know if it's possible for mock to patch out functions used internally like this, or am I out of luck?

The answer: Clean up your darned imports
#patch('mymodule.TAX_LOCATION', '') did indeed patch things appropriately, but since our imports at the time were very haphazard -- sometimes we imported mymodule.build_cart, sometimes we imported project.mymodule.build_cart -- instances of the "full" import were not patched at all. Mock couldn't be expected to know about the two separate import paths... without being told explicitly, anyway.
We've since standardized all our imports on the longer path, and things behave much more nicely now.

another option is to explicitly call patch on the function:
mock.patch('function_name')
and to support both running directly or from py.test etc:
mock.patch(__name__ + '.' + 'function_name')

I'd like to add solution other than accepted one. You can also patch the module before it's been imported in any other modules and remove patch at the end of your test case.
#import some modules that don't use module you are going to patch
import unittest
from mock import patch
import json
import logging
...
patcher = patch('some.module.path.function', lambda x: x)
patcher.start()
import some.module.path
class ViewGetTests(unittest.TestCase):
#classmethod
def tearDownClass(cls):
patcher.stop()

I'm pretty sure your problem is that you are importing 'mymodule' inside your test functions, and therefore the patch decorator has no chance of actually patching. Do the import at the top of the module, like any other import.

If your module is in a folder with an __init__.py file that has from [module_file] import * make sure your patch argument has the folder and file name (module_folder.module_file), or the patch will succeed (no 'module does not have this attribute' error) but not function (calls will go to the actual function not the mock), no matter how the function under test is imported.