I'm working on a project that dynamically collects classes from another module and creates new classes based off the ones it found. For example, if the source module has a "AdaptiveFilter" class, a new class called "AdaptiveFilterNode" is created that contains an "AdaptiveFilter" instance. I've been adding these generated classes to a dynamically created module (module_from_spec()) and created a MetaPathFinder and Loader to extend the import system. This works perfectly fine when importing the dynamic module, e.g. import nodes.auto_gen or from nodes.auto_gen import AdaptiveFilterNode, but fails when trying to directly import a class, e.g. import nodes.auto_gen.AdaptiveFilterNode. I need to be able to directly import a class because the generated classes need to be pickleable, which requires a direct import. I believe the problem is in the Finder that I made:
class MyNodeFinder(MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
if fullname == 'nodes.auto_gen':
auto_gen_spec = = ModuleSpec('nodes.auto_gen',
MyLoader())
return auto_gen_spec
return None
From my limited understanding on the import system, there's nothing in the finder or the module spec that tells the import system there's classes contained in the nodes.auto_gen module until the module is already loaded. How can I create a finder/loader that replicates the standard import system for a dynamic module with dynamic classes?
EDIT: Full example, module "dynamic_creator":
dynamic_creator/__init__.py
from importlib.machinery import ModuleSpec
from importlib.util import module_from_spec
import sys
from importlib.abc import MetaPathFinder
from importlib.abc import Loader
__version__ = '0.0.0'
class _MyWrapperClass():
wrapped_func = None
def __init__(self):
super().__init__()
def __call__(self, *args, **kwargs):
return self.wrapped_func(*args, **kwargs)
class MyFinder(MetaPathFinder):
def find_spec(self, fullname, path=None, target=None):
if fullname == 'dynamic_creator.auto_gen':
auto_node_spec = ModuleSpec('dynamic_creator.auto_gen',
MyLoader())
return auto_node_spec
return None
class MyLoader(Loader):
def create_module(self, spec):
return module_from_spec(ModuleSpec('dynamic_creator.auto_gen', None))
def exec_module(self, module):
def create_type(func):
t = type(func.__name__, (_MyWrapperClass, ), {})
t.wrapped_func = func
# Wait, you can do this?
t.__module__ = 'dynamic_creator.auto_gen'
return t
funcs_to_wrap = [
sum,
abs,
pow
]
for func in funcs_to_wrap:
setattr(module, func.__name__, create_type(func))
my_finder = MyFinder()
sys.meta_path.append(my_finder)
Test script:
import dynamic_creator.auto_gen # Works
from dynamic_creator.auto_gen import abs # Works
import dynamic_creator.auto_gen.pow as my_pow # Fails
EDIT: The reported error is as follows:
No module named 'dynamic_creator.auto_gen.pow'; 'dynamic_creator.auto_gen' is not a package
Related
I am having difficulty with a circular import problem. I have 2 class files like so:
--/service
service_module.py
settings.py
service_module imports other various files used throughout the project and acts as a container for various functions throughout the project. I want to assert in my settings.py file that it is properly passed an instance of the service_module parent class. Removing the assert statement in settings.py fixes the issue and I am able to properly call methods in the service_module class. For code completion and error checking it makes my life easier to assert.
I have always struggled with understanding python imports but is this the right direction to handle my particular case?
service_module.py
from PyQt5.QtCore import QObject
from sqlalchemy import *
from sqlalchemy.orm import scoped_session, Session, sessionmaker
from service.logger import logger
from sqlalchemy.orm.exc import NoResultFound
from database.tables import *
from database.load_db import load_db
from service.settings import settings
from service.web_listener import web_listener
from service.character_manager import character_manager
class Service_Module(QObject):
def __init__(self):
super(Service_Module, self).__init__()
load_database = load_db()
self.sc_session: scoped_session = load_database.get_scoped_session()
tb_scopes.make_default_scopes(service_module=self)
self.logger = logger(service_module=self)
self.settings = settings(service_module=self)
self.characters = character_manager(service_module=self)
self.callback_listener: web_listener = web_listener(service_module=self)
self.callback_listener.start()
assert isinstance(self.sc_session, scoped_session)
assert isinstance(self.logger, logger)
assert isinstance(self.settings, settings)
assert isinstance(self.callback_listener, web_listener)
settings.py
from service.service_module import *
class settings(QObject):
def __init__(self, service_module):
super(settings, self).__init__()
self.service = service_module
assert isinstance(self.service, Service_Module) ##raises NameError: name 'Service_Module' is not defined
Edit:
So changing to this solves my issue although I feel like it's kind of hacky and somehow incorrect.
from service.service_module import *
import service.service_module
class settings(QObject):
def __init__(self, service_module):
super(settings, self).__init__()
self.service = service_module
assert isinstance(self.service, service.service_module.Service_Module)
You need to delay the importing of one (or both) of the files, to break the circular import. You can do this by moving the import statement from file-scope (where it is executed as soon as the module is imported) into the execution scope of a function or method, where is not executed until that method is called. Ie)
class setting(QObject):
def __init__(self, service_module):
from service.service_module import Service_Module # <-- import is here
assert isinstance(self.service, Service_Module)
Of course, this may affect other usages of the imported symbols from that module into this file, so you may need to specify the import in more than one place.
I have class called 'my_class' placed in 'my_module'. And I need to import this class. I tried to do it like this:
import importlib
result = importlib.import_module('my_module.my_class')
but it says:
ImportError: No module named 'my_module.my_class'; 'my_module' is not a package
So. As I can see it works only for modules, but can't handle classes. How can I import a class from a module?
It is expecting my_module to be a package containing a module named 'my_class'. If you need to import a class, or an attribute in general, dynamically, just use getattr after you import the module:
cls = getattr(import_module('my_module'), 'my_class')
Also, yes, it does only work with modules. Remember importlib.import_module is a wrapper of the internal importlib.__import__ function. It doesn't offer the same amount of functionality as the full import statement which, coupled with from, performs an attribute look-up on the imported module.
import importlib
import logging
logger = logging.getLogger(__name__)
def factory(module_class_string, super_cls: type = None, **kwargs):
"""
:param module_class_string: full name of the class to create an object of
:param super_cls: expected super class for validity, None if bypass
:param kwargs: parameters to pass
:return:
"""
module_name, class_name = module_class_string.rsplit(".", 1)
module = importlib.import_module(module_name)
assert hasattr(module, class_name), "class {} is not in {}".format(class_name, module_name)
logger.debug('reading class {} from module {}'.format(class_name, module_name))
cls = getattr(module, class_name)
if super_cls is not None:
assert issubclass(cls, super_cls), "class {} should inherit from {}".format(class_name, super_cls.__name__)
logger.debug('initialising {} with params {}'.format(class_name, kwargs))
obj = cls(**kwargs)
return obj
I am trying to write a function which will take class name a argument and import that class and perform some task.
def search(**kwargs):
"""
:return:
"""
try:
model = __import__('girvi.models', globals(), locals(), kwargs['model_name'], -1)
# Do some task
return results
except Exception:
raise Exception('Model not found')
But thing is model has the class which is in kwargs['model_name'] imported successfully but how do i access it. Please can someone help me.
I would try the following:
import importlib
import sys
def import_class(class_name, module_name = "girvi.models"):
module = importlib.import_module(module_name)
class_object = getattr(module, class_name) # does not work for nested classes
return class_object
According to __import__ one should rather use importlib.import_module.
I am trying to produce a Mock of matplotlib so that I can compile my docs using ReadTheDocs, but have run into a problem.
In my code, I import matplotlib using from matplotlib.pyplot import *.
I am using the following code for my Mocks (as suggested by the ReadTheDocs FAQ):
class Mock(object):
def __init__(self, *args, **kwargs):
pass
def __call__(self, *args, **kwargs):
return Mock()
#classmethod
def __getattr__(cls, name):
if name in ('__file__', '__path__'):
return '/dev/null'
elif name[0] == name[0].upper():
return type(name, (), {})
else:
return Mock()
MOCK_MODULES = ['numpy', 'scipy', 'matplotlib', 'matplotlib.pyplot']
for mod_name in MOCK_MODULES:
sys.modules[mod_name] = Mock()
However, when running from matplotlib.pyplot import * I get an error saying that TypeError: 'type' object does not support indexing.
Is there a way that I can change my Mock so that it allows me to import matplotlib using the from x import * style? I don't need any particular functions to be made available, I just need it to be able to be imported so that ReadTheDocs can import the code properly.
In case of importing via * you need to define the __all__ list in the module. The same goes with your class: just add the __all__ attribute to the class and it should work fine:
class Mock(object):
__all__ = []
Interesting usecase today: I need to migrate a module in our codebase following code changes. The old mynamespace.Document will disappear and I want to ensure smooth migration by replacing this package by a code object that will dynamically import the correct path and migrate the corresponding objects.
In short:
# instanciate a dynamic package, but do not load
# statically submodules
mynamespace.Document = SomeObject()
assert 'submodule' not in mynamespace.Document.__dict__
# and later on, when importing it, the submodule
# is built if not already available in __dict__
from namespace.Document.submodule import klass
c = klass()
A few things to note:
I am not talking only of migrating code. A simple huge sed would in a sense be enough to change the code in order to migrate some imports, and I would not need a dynamic module. I am talking of objects. A website, holding some live/stored objects will need migration. Those objects will be loaded assuming that mynamespace.Document.submodule.klass exists, and that's the reason for the dynamic module. I need to provide the site with something to load.
We cannot, or do not want to change the way objects are unpickled/loaded. For simplicity, let's just say that we want to make sure that the idiom from mynamespace.Document.submodule import klass has to work. I cannot use instead from mynamespace import Document as container; klass = getattr(getattr(container, 'submodule'), 'klass')
What I tried:
import sys
from types import ModuleType
class VerboseModule(ModuleType):
def __init__(self, name, doc=None):
super(VerboseModule, self).__init__(name, doc)
sys.modules[name] = self
def __repr__(self):
return "<%s %s>" % (self.__class__.__name__, self.__name__)
def __getattribute__(self, name):
if name not in ('__name__', '__repr__', '__class__'):
print "fetching attribute %s for %s" % (name, self)
return super(VerboseModule, self).__getattribute__(name)
class DynamicModule(VerboseModule):
"""
This module generates a dummy class when asked for a component
"""
def __getattr__(self, name):
class Dummy(object):
pass
Dummy.__name__ = name
Dummy.__module__ = self
setattr(self, name, Dummy)
return Dummy
class DynamicPackage(VerboseModule):
"""
This package should generate dummy modules
"""
def __getattr__(self, name):
mod = DynamicModule("%s.%s" % (self.__name__, name))
setattr(self, name, mod)
return mod
DynamicModule("foobar")
# (the import prints:)
# fetching attribute __path__ for <DynamicModule foobar>
# fetching attribute DynamicModuleWorks for <DynamicModule foobar>
# fetching attribute DynamicModuleWorks for <DynamicModule foobar>
from foobar import DynamicModuleWorks
print DynamicModuleWorks
DynamicPackage('document')
# fetching attribute __path__ for <DynamicPackage document>
from document.submodule import ButDynamicPackageDoesNotWork
# Traceback (most recent call last):
# File "dynamicmodule.py", line 40, in <module>
# from document.submodule import ButDynamicPackageDoesNotWork
#ImportError: No module named submodule
As you can see the Dynamic Package does not work. I do not understand what is happening because document is not even asked for a ButDynamicPackageDoesNotWork attribute.
Can anyone clarify what is happening; and if/how I can fix this?
The problem is that python will bypass the entry in for document in sys.modules and load the file for submodule directly. Of course this doesn't exist.
demonstration:
>>> import multiprocessing
>>> multiprocessing.heap = None
>>> import multiprocessing.heap
>>> multiprocessing.heap
<module 'multiprocessing.heap' from '/usr/lib/python2.6/multiprocessing/heap.pyc'>
We would expect that heap is still None because python can just pull it out of sys.modules but That doesn't happen. The dotted notation essentially maps directly to {something on python path}/document/submodule.py and an attempt is made to load that directly.
Update
The trick is to override pythons importing system. The following code requires your DynamicModule class.
import sys
class DynamicImporter(object):
"""this class works as both a finder and a loader."""
def __init__(self, lazy_packages):
self.packages = lazy_packages
def load_module(self, fullname):
"""this makes the class a loader. It is given name of a module and expected
to return the module object"""
print "loading {0}".format(fullname)
components = fullname.split('.')
components = ['.'.join(components[:i+1])
for i in range(len(components))]
for component in components:
if component not in sys.modules:
DynamicModule(component)
print "{0} created".format(component)
return sys.modules[fullname]
def find_module(self, fullname, path=None):
"""This makes the class a finder. It is given the name of a module as well as
the package that contains it (if applicable). It is expected to return a
loader for that module if it knows of one or None in which case other methods
will be tried"""
if fullname.split('.')[0] in self.packages:
print "found {0}".format(fullname)
return self
else:
return None
# This is a list of finder objects which is empty by defaule
# It is tried before anything else when a request to import a module is encountered.
sys.meta_path=[DynamicImporter('foo')]
from foo.bar import ThisShouldWork