I've got some python code in a library that attempts to load a simple value from a module that will exist for the applications that use this library
from somemodule import simplevalue
Normally, the application that uses the library will have the module file and everything works fine. However, in the unit tests for this library the module does not exist. I know that I can create a temporary file and add that file to my path at runtime, but I was curious if there is a way in python to load something in to memory that would allow the above import to work.
This is more of a curiosity, saying "add the module to your test path" is not helpful :P
It is. Use types.ModuleType to create a new module object, then add it to sys.modules:
sys.modules["somename"] = types.ModuleType("somename")
You can then do import somename. If you need to add classes or functions to it, import it before calling your test script, and just add functions to it:
def myfunc(x, y, z):
...
somename.myfunc = myfunc
It should go without saying, but just in case: this is largely an academic curiosity. It has some uses for testing, but other than that, stick to importing things the usual way.
Incidentally, how I know about this: I've come across the technique used in testing, to make a "stub" for the _winreg module on non-Windows systems. Here it is in use.
It isn't necessary to create a module. No Python code cares whether somemodule.simplevalue is actually a reference to an attribute of a module. To do so, a program would need to check the type of somemodule. Why bother?
Since you just want the single value from the module and are importing it into your own namespace, just define it:
simplevalue = 42
If you like, use try/except to try to import the real module first.
try:
from somemodule import simplevalue
except ImportError:
simplevalue = 42
If you are importing the entire module but only using one value, you can use a class to define a namespace.
try:
import somemodule
except ImportError:
class somemodule(object):
simplevalue = 42
Now somemodule.simplevalue refers to the value regardless of whether the module is available.
If you want other modules that also import somemodule to see your faked-up class-as-module, as you would in your unit test, just do this afterward:
import sys
sys.modules["somemodule"] = somemodule
Your system under test (sut in my example) needs to be able to cope with the fact that somemodule may not exist, so you can trap the ImportError:
#!/usr/bin/env python
try:
from somemodule import simplevalue
except ImportError, e:
if 'somemodule' in e:
'''We expect that to happen in the unittest but you should log something for when
this happens in production'''
def fn():
return simplevalue
Then you can inject a value in your unittest:
#!/usr/bin/env python
import unittest
import sut
class T(unittest.TestCase):
def test_fn(self):
sut.simplevalue = 42
self.assertEquals(42, sut.fn())
if __name__ == '__main__':
unittest.main()
Related
I have a program (like a macro) that runs within a parent program and imports an API module from that program (lets call it foo). The problem is that that module only exists within that program, so I can't do things like run pydocmd outside the software because the script throws a ReferenceError. To aid in my own development I have create a type stub file, foo.pyi, in my project directory. What I would like to do is import that type stub as a normal Python file if the import fails, to provide dummy functions and properties. Something like:
try:
import foo
except ImportError:
from . import foo.pyi
This raises an error, however, as it's trying to import pyi from the foo library that does not exist in the project folder. The only other option I can think of is to have an identical copy of the .pyi file as, say "dummy_foo.py" but then I have to maintain two copies of the same file in one repo. I'd rather not do that.
I wrote this a while back; should still work I think:
import importlib
def import_stub(stubs_path, module_name):
sys.path_hooks.insert(0,
importlib.machinery.FileFinder.path_hook(
(importlib.machinery.SourceFileLoader, ['.pyi']))
)
sys.path.insert(0, stubs_path)
try:
return importlib.import_module(module_name)
finally:
sys.path.pop(0)
sys.path_hooks.pop(0)
I found this question, but my problem was about type checking. In my case pyi file contains class definition (so type hints are working), but the library doesn't. Solution is in checking typing.TYPE_CHECKING:
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from library import _PrivateClass
def foo(x: "_PrivateClass"):
...
Ok, this is extremely hacky and silly programming. I have a function that can import a module file from a URL. It works fine and is hardly secure. I want to import a module file that has a problematic dependency (the system I'm on can't support that particular dependency) but the functionality I want from that module doesn't rely on the problematic dependency, so it's not a problem if it can be ignored.
My thinking is that I could use this smuggle function (shown below) to get the module file and then somehow import it using FuckIt.py, but I'm not sure how to make these two ideas work together.
How could this be done?
import imp
import urllib
def smuggle(
module_name = None,
URL = None
):
if module_name is None:
module_name = URL
try:
module = __import__(module_name)
return(module)
except:
try:
module_string = urllib.urlopen(URL).read()
module = imp.new_module("module")
exec module_string in module.__dict__
return(module)
except:
raise(
Exception(
"module {module_name} import error".format(
module_name = module_name
)
)
)
sys.exit()
damned_silly_module = smuggle(
module_name = "damned_silly_module",
URL = "https://raw.githubusercontent.com/https://github.com/justsomefuckingguy/damned_silly_module/master/damned_silly_module.py"
)
damned_silly_module.some_function_or_other()
Putting aside Fuckit.py, if this is about a particular module with particular failing dependencies, the best way to get this to work is by making the import of the dependency succeed: Provide a mock sub-module with the same name, with stubs for whatever will be asked for. For example, if damn_silly_module tries to import silly_walks, which you don't have, make a mock silly_walks module and arrange for it to be found.
import sys
sys.path.insert(0, "path/to/mock/modules")
module = imp.new_module("module")
Or something like that. You could even catch ImportError and do this only if the module in question is absent. This is analogous to the python 2 custom of importing, say, cPickle as pickle and failing back to import pickle if that is unavailable.
If you want this to work in general, with modules you'll see in the future, you'd need to catch ImportError, examine it to figure out what's missing, mock it on the fly and try again.
Incidentally, your exception handling needs some work. Never catch everything (with except: with no arguments); catch ImportError (and perhaps NameError if the import succeeds but a later name lookup fails). Never raise an undifferentiated Exception, raise ImportError. In this case, it may be better to re-raise the exception you just caught, with a simple
raise
And get rid of sys.exit(). It's dead code-- it'll never be reached. (Also: raise is a keyword, you don't need function brackets around its arguments).
I have a module that features numerous functions. Some of these functions are dependent on other modules. The module is to be used in some environments that are not going to have these other modules. In these environments, I want the functionality of the module that is not dependent on these other unavailable modules to be usable. What is a reasonable way of coding this?
I am happy for an exception to be raised when a function's dependency module is not met, but I want the module to be usable for other functions that do not have dependency problems.
At present, I'm thinking of a module-level function something like the following:
tryImport(moduleName = None):
try:
module = __import__(moduleName)
return(module)
except:
raise(Exception("module {moduleName} import error".format(
moduleName = moduleName)))
sys.exit()
This would then be used within functions in a way such as the following:
def function1():
pyfiglet = tryImport(moduleName = "pyfiglet")
For your use case, it sounds like there's nothing wrong with putting the imports you need inside functions, rather than at the top of your module (don't worry, it costs virtually nothing to reimport):
def some_function():
import foo # Throws if there is no foo
return foo.bar ()
Full stop, please.
The first thing is that the problem you described indicates that you should redesign your module. The most obvious solution is to break it into a couple of modules in one package. Each of them would contain group of functions with common external dependencies. You can easily define how many groups you need if you only know which dependencies might be missing on the target machine. Then you can import them separately. Basically in a given environment you import only those you need and the problem doesn't exist anymore.
If you still believe it's not the case and insist on keeping all the functions in one module then you can always do something like:
if env_type == 'env1':
import pyfiglet
if env_type in ('env1', 'env2'):
import pynose
import gzio
How you deduct the env_type is up to you. Might come from some configuration file or the environment variable.
Then you have your functions in this module. No problem occurs if none of the module consumers calls function which makes use of the module which is unavailable in a given environment.
I don't see any point in throwing your custom exception. NameError exception will be thrown anyway upon trial to access not imported name. By the way your sys.exit() would never be executed anyway.
If you don't want to define environment types, you can still achieve the goal with the following code:
try: import pyfiglet
except ImportError: pass
try: import pynose
except ImportError: pass
try: import gzio
except ImportError: pass
Both code snippets are supposed to be used on module level and not inside functions.
TL;DR I would just break this module into several parts. But if you really have to keep it monolithic, just use the basic language features and don't over-engineer it by using __import__ and a dedicated function.
I need to make a copy of a socket module to be able to use it and to have one more socket module monkey-patched and use it differently.
Is this possible?
I mean to really copy a module, namely to get the same result at runtime as if I've copied socketmodule.c, changed the initsocket() function to initmy_socket(), and installed it as my_socket extension.
You can always do tricks like importing a module then deleting it from sys.modules or trying to copy a module. However, Python already provides what you want in its Standard Library.
import imp # Standard module to do such things you want to.
# We can import any module including standard ones:
os1=imp.load_module('os1', *imp.find_module('os'))
# Here is another one:
os2=imp.load_module('os2', *imp.find_module('os'))
# This returns True:
id(os1)!=id(os2)
Python3.3+
imp.load_module is deprecated in python3.3+, and recommends the use of importlib
#!/usr/bin/env python3
import sys
import importlib.util
SPEC_OS = importlib.util.find_spec('os')
os1 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os1)
sys.modules['os1'] = os1
os2 = importlib.util.module_from_spec(SPEC_OS)
SPEC_OS.loader.exec_module(os2)
sys.modules['os2'] = os2
del SPEC_OS
assert os1 is not os2, \
"Module `os` instancing failed"
Here, we import the same module twice but as completely different module objects. If you check sys.modules, you can see two names you entered as first parameters to load_module calls. Take a look at the documentation for details.
UPDATE:
To make the main difference of this approach obvious, I want to make this clearer: When you import the same module this way, you will have both versions globally accessible for every other module you import in runtime, which is exactly what the questioner needs as I understood.
Below is another example to emphasize this point.
These two statements do exactly the same thing:
import my_socket_module as socket_imported
socket_imported = imp.load_module('my_socket_module',
*imp.find_module('my_socket_module')
)
On second line, we repeat 'my_socket_module' string twice and that is how import statement works; but these two strings are, in fact, used for two different reasons.
Second occurrence as we passed it to find_module is used as the file name that will be found on the system. The first occurrence of the string as we passed it to load_module method is used as system-wide identifier of the loaded module.
So, we can use different names for these which means we can make it work exactly like we copied the python source file for the module and loaded it.
socket = imp.load_module('socket_original', *imp.find_module('my_socket_module'))
socket_monkey = imp.load_module('socket_patched',*imp.find_module('my_socket_module'))
def alternative_implementation(blah, blah):
return 'Happiness'
socket_monkey.original_function = alternative_implementation
import my_sub_module
Then in my_sub_module, I can import 'socket_patched' which does not exist on system! Here we are in my_sub_module.py.
import socket_patched
socket_patched.original_function('foo', 'bar')
# This call brings us 'Happiness'
This is pretty disgusting, but this might suffice:
import sys
# if socket was already imported, get rid of it and save a copy
save = sys.modules.pop('socket', None)
# import socket again (it's not in sys.modules, so it will be reimported)
import socket as mysock
if save is None:
# if we didn't have a saved copy, remove my version of 'socket'
del sys.modules['socket']
else:
# if we did have a saved copy overwrite my socket with the original
sys.modules['socket'] = save
Here's some code that creates a new module with the functions and variables of the old:
def copymodule(old):
new = type(old)(old.__name__, old.__doc__)
new.__dict__.update(old.__dict__)
return new
Note that this does a fairly shallow copy of the module. The dictionary is newly created, so basic monkey patching will work, but any mutables in the original module will be shared between the two.
Edit: According to the comment, a deep copy is needed. I tried messing around with monkey-patching the copy module to support deep copies of modules, but that didn't work. Next I tried importing the module twice, but since modules are cached in sys.modules, that gave me the same module twice. Finally, the solution I hit upon was removing the modules from sys.modules after importing it the first time, then importing it again.
from imp import find_module, load_module
from sys import modules
def loadtwice(name, path=None):
"""Import two copies of a module.
The name and path arguments are as for `find_module` in the `imp` module.
Note that future imports of the module will return the same object as
the second of the two returned by this function.
"""
startingmods = modules.copy()
foundmod = find_module(name, path)
mod1 = load_module(name, *foundmod)
newmods = set(modules) - set(startingmods)
for m in newmods:
del modules[m]
mod2 = load_module(name, *foundmod)
return mod1, mod2
Physically copy the socket module to socket_monkey and go from there? I don't feel you need any "clever" work-around... but I might well be over simplifying!
I'm working on pypreprocessor which is a preprocessor that takes c-style directives and I've been able to make it work like a traditional preprocessor (it's self-consuming and executes postprocessed code on-the-fly) except that it breaks library imports.
The problem is: The preprocessor runs through the file, processes it, outputs to a temporary file, and exec() the temporary file. Libraries that are imported need to be handled a little different, because they aren't executed, but rather they are loaded and made accessible to the caller module.
What I need to be able to do is: Interrupt the import (since the preprocessor is being run in the middle of the import), load the postprocessed code as a tempModule, and replace the original import with the tempModule to trick the calling script with the import into believing that the tempModule is the original module.
I have searched everywhere and so far and have no solution.
This Stack Overflow question is the closest I've seen so far to providing an answer:
Override namespace in Python
Here's what I have.
# Remove the bytecode file created by the first import
os.remove(moduleName + '.pyc')
# Remove the first import
del sys.modules[moduleName]
# Import the postprocessed module
tmpModule = __import__(tmpModuleName)
# Set first module's reference to point to the preprocessed module
sys.modules[moduleName] = tmpModule
moduleName is the name of the original module, and tmpModuleName is the name of the postprocessed code file.
The strange part is this solution still runs completely normal as if the first module completed loaded normally; unless you remove the last line, then you get a module not found error.
Hopefully someone on Stack Overflow know a lot more about imports than I do, because this one has me stumped.
Note: I will only award a solution, or, if this is not possible in Python; the best, most detailed explanation of why this is not impossible.
Update: For anybody who is interested, here is the working code.
if imp.lock_held() is True:
del sys.modules[moduleName]
sys.modules[tmpModuleName] = __import__(tmpModuleName)
sys.modules[moduleName] = __import__(tmpModuleName)
The 'imp.lock_held' part detects whether the module is being loaded as a library. The following lines do the rest.
Does this answer your question? The second import does the trick.
Mod_1.py
def test_function():
print "Test Function -- Mod 1"
Mod_2.py
def test_function():
print "Test Function -- Mod 2"
Test.py
#!/usr/bin/python
import sys
import Mod_1
Mod_1.test_function()
del sys.modules['Mod_1']
sys.modules['Mod_1'] = __import__('Mod_2')
import Mod_1
Mod_1.test_function()
To define a different import behavior or to totally subvert the import process you will need to write import hooks. See PEP 302.
For example,
import sys
class MyImporter(object):
def find_module(self, module_name, package_path):
# Return a loader
return self
def load_module(self, module_name):
# Return a module
return self
sys.meta_path.append(MyImporter())
import now_you_can_import_any_name
print now_you_can_import_any_name
It outputs:
<__main__.MyImporter object at 0x009F85F0>
So basically it returns a new module (which can be any object), in this case itself. You may use it to alter the import behavior by returning processe_xxx on import of xxx.
IMO: Python doesn't need a preprocessor. Whatever you are accomplishing can be accomplished in Python itself due to it very dynamic nature, for example, taking the case of the debug example, what is wrong with having at top of file
debug = 1
and later
if debug:
print "wow"
?
In Python 2 there is the imputil module that seems to provide the functionality you are looking for, but has been removed in python 3. It's not very well documented but contains an example section that shows how you can replace the standard import functions.
For Python 3 there is the importlib module (introduced in Python 3.1) that contains functions and classes to modify the import functionality in all kinds of ways. It should be suitable to hook your preprocessor into the import system.