I am trying to implement a small library for Python 3.5 but keep struggling with how to correctly handle the structuring of the packages/modules and how to get the imports to work.
I keep running into the problem where python complains of being unable to import some name with an error like
ImportError: cannot import name 'SubClass1'
This seems to happen when "SubClass1" needs to import some other module but that other module also needs to know about SubClass1 (a cyclic import).
I need the cyclic import in my library because the base class has a factory method that creates the proper subclass instances (there are also other situations where cyclic imports are needed, e.g. checking the type of a function argument needs the import of where that type is defined, but that module may itself need the class where that check is done: another cyclic dependency!)
Here is example code:
Root directory contains the subdirectory dir1. The directory dir1 contains and empty file init.py, a file baseclass.py and a file subclass1.py.
The file ./dir1/subclass1.py contains:
from . baseclass import BaseClass
class SubClass1(BaseClass):
pass
The file ./dir1/baseclass.py contains:
from . subclass1 import SubClass1
class BaseClass(object):
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
# .. which gets eventually returned by this factory method
return ret
The file ./test1.py contains:
from dir1.subclass1 import SubClass1
sc1 = SubClass1()
This results in the following error:
Traceback (most recent call last):
File "test1.py", line 1, in <module>
from dir1.subclass1 import SubClass1
File "/data/johann/tmp/python1/dir1/subclass1.py", line 1, in <module>
from . baseclass import BaseClass
File "/data/johann/tmp/python1/dir1/baseclass.py", line 1, in <module>
from . subclass1 import SubClass1
ImportError: cannot import name 'SubClass1'
What is the standard/best way to solve this problem, ideally in a way that is backwards compatible to python 2.x and python 3 up to version 3.2?
I have read elsewhere that importing the module instead of something from a module may help here but I do not know how to just import the module (e.g. subclass1) in a relative way because "import . subclass1" or similar does not work.
Your issue is caused by a circular import. The baseclass module is trying to import SubClass1 from the subclass1 module, but subclass is trying to import BaseClass right back. You get NameError because the classes haven't been defined yet when the import statements are running.
There are a few ways to solve the issue.
One option would be to change your style of import. Instead of importing the classes by name, just import the modules and look up the names as attributes later on.
from . import baseclass
class SubClass1(baseclass.BaseClass):
pass
And:
from . import subclass1
class BaseClass:
def make(self,somearg):
# ...
ret = subclass1.SubClass1()
Because SubClass1 needs to be able to use BaseClass immediately at definition time, this code may still fail if the baseclass module is imported before subclass1. So it's not ideal
Another option would be to change baseclass to do its import below the definition of BaseClass. This way the subclass module will be able to import the name when it needs to:
class BaseClass:
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
from .subclass1 import SubClass1
This is not ideal because the normal place to put imports is at the top of the file. Putting them elsewhere makes the code more confusing. You may want to put a comment up at the top of the file explaining why you're delaying the import if you go this route.
Another option may be to combine your two modules into a single file. Python doesn't require each class to have its own module like some other languages do. When you have tightly coupled classes (like the ones in your example), it makes a lot of sense to put them all in one place. This lets you avoid the whole issue, since you don't need any imports at all.
Finally, there are some more complicated solutions, like dependency injection. Rather than the base class needing to know about the subclasses, each subclass could register itself by calling some function and passing a reference to itself. For example:
# no imports of subclasses!
def BaseClass:
subclasses = []
def make(self, somearg):
for sub in self.subclasses:
if sub.accepts(somearg):
return sub()
raise ValueError("no subclass accepts value {!r}".format(somearg))
#classmethod
def register(cls, sub):
cls.subclasses.append(sub)
return sub # return the class so it can be used as a decorator!
And in subclass.py
from .baseclass import BaseClass
#BaseClass.register
class SubClass1(BaseClass):
#classmethod
def accepts(cls, somearg):
# put logic for picking this subclass here!
return True
This style of programming is a bit more complicated, but it can be nice since it's easier to extend than a version where BaseClass needs to know about all of the subclasses up front. There are a variety of ways you can implement this style of code, using a register function is just one of them. One nice thing about it is that it doesn't strictly require inheritance (so you could register a class that doesn't actually inherit from BaseClass if you wanted to). If you are only dealing with actual inheriting subclasses, you might want to consider using a metaclass that does all the registration of subclasses for you automatically.
Related
This question already has answers here:
What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"?
(17 answers)
Closed 6 months ago.
I know the issue of circular imports in python has come up many times before and I have read these discussions. The comment that is made repeatedly in these discussions is that a circular import is a sign of a bad design and the code should be reorganised to avoid the circular import.
Could someone tell me how to avoid a circular import in this situation?: I have two classes and I want each class to have a constructor (method) which takes an instance of the other class and returns an instance of the class.
More specifically, one class is mutable and one is immutable. The immutable class is needed
for hashing, comparing and so on. The mutable class is needed to do things too. This is similar to sets and frozensets or to lists and tuples.
I could put both class definitions in the same module. Are there any other suggestions?
A toy example would be class A which has an attribute which is a list and class B which has an attribute which is a tuple. Then class A has a method which takes an instance of class B and returns an instance of class A (by converting the tuple to a list) and similarly class B has a method which takes an instance of class A and returns an instance of class B (by converting the list to a tuple).
Consider the following example python package where a.py and b.py depend on each other:
/package
__init__.py
a.py
b.py
Types of circular import problems
Circular import dependencies typically fall into two categories depending
on what you're trying to import and where you're using it inside each
module. (And whether you're using python 2 or 3).
1. Errors importing modules with circular imports
In some cases, just importing a module with a circular import dependency
can result in errors even if you're not referencing anything from the
imported module.
There are several standard ways to import a module in python
import package.a # (1) Absolute import
import package.a as a_mod # (2) Absolute import bound to different name
from package import a # (3) Alternate absolute import
import a # (4) Implicit relative import (deprecated, python 2 only)
from . import a # (5) Explicit relative import
Unfortunately, only the 1st and 4th options actually work when you
have circular dependencies (the rest all raise ImportError
or AttributeError). In general, you shouldn't be using the
4th syntax, since it only works in python2 and runs the risk of
clashing with other 3rd party modules. So really, only the first
syntax is guaranteed to work.
EDIT: The ImportError and AttributeError issues only occur in
python 2. In python 3 the import machinery has been rewritten and all
of these import statements (with the exception of 4) will work, even with
circular dependencies. While the solutions in this section may help refactoring python 3 code, they are mainly intended
for people using python 2.
Absolute Import
Just use the first import syntax above. The downside to this method is
that the import names can get super long for large packages.
In a.py
import package.b
In b.py
import package.a
Defer import until later
I've seen this method used in lots of packages, but it still feels
hacky to me, and I dislike that I can't look at the top of a module
and see all its dependencies, I have to go searching through all the
functions as well.
In a.py
def func():
from package import b
In b.py
def func():
from package import a
Put all imports in a central module
This also works, but has the same problem as the first method, where
all the package and submodule calls get super long. It also has two
major flaws -- it forces all the submodules to be imported, even if
you're only using one or two, and you still can't look at any of the
submodules and quickly see their dependencies at the top, you have to
go sifting through functions.
In __init__.py
from . import a
from . import b
In a.py
import package
def func():
package.b.some_object()
In b.py
import package
def func():
package.a.some_object()
2. Errors using imported objects with circular dependencies
Now, while you may be able to import a module with a circular import
dependency, you won't be able to import any objects defined in the module
or actually be able to reference that imported module anywhere
in the top level of the module where you're importing it. You can,
however, use the imported module inside functions and code blocks that don't
get run on import.
For example, this will work:
package/a.py
import package.b
def func_a():
return "a"
package/b.py
import package.a
def func_b():
# Notice how package.a is only referenced *inside* a function
# and not the top level of the module.
return package.a.func_a() + "b"
But this won't work
package/a.py
import package.b
class A(object):
pass
package/b.py
import package.a
# package.a is referenced at the top level of the module
class B(package.a.A):
pass
You'll get an exception
AttributeError: module 'package' has no attribute 'a'
Generally, in most valid cases of circular dependencies, it's possible
to refactor or reorganize the code to prevent these errors and move
module references inside a code block.
Only import the module, don't import from the module:
Consider a.py:
import b
class A:
def bar(self):
return b.B()
and b.py:
import a
class B:
def bar(self):
return a.A()
This works perfectly fine.
We do a combination of absolute imports and functions for better reading and shorter access strings.
Advantage: Shorter access strings compared to pure absolute imports
Disadvantage: a bit more overhead due to extra function call
main/sub/a.py
import main.sub.b
b_mod = lambda: main.sub.b
class A():
def __init__(self):
print('in class "A":', b_mod().B.__name__)
main/sub/b.py
import main.sub.a
a_mod = lambda: main.sub.a
class B():
def __init__(self):
print('in class "B":', a_mod().A.__name__)
I have a python source file with a class defined in it, and a class from another module imported into it. Essentially, this structure:
from parent import SuperClass
from other import ClassA
class ClassB(SuperClass):
def __init__(self): pass
What I want to do is look in this module for all the classes defined in there, and only to find ClassB (and to overlook ClassA). Both ClassA and ClassB extend SuperClass.
The reason for this is that I have a directory of plugins which are loaded at runtime, and I get a full list of the plugin classes by introspecting on each .py file and loading the classes which extend SuperClass. In this particular case, ClassB uses the plugin ClassA to do some work for it, so is dependent upon it (ClassA, meanwhile, is not dependent on ClassB). The problem is that when I load the plugins from the directory, I get 2 instances of ClassA, as it gets one from ClassA's file, and one from ClassB's file.
For packages there is the approach:
__all__ = ['module_a', 'module-b']
to explicitly list the modules that you can import, but this lives in the __init__.py file, and each of the plugins is a .py file not a directory in its own right.
The question, then, is: can I limit access to the classes in a .py file, or do I have to make each one of them a directory with its own init file? Or, is there some other clever way that I could distinguish between these two classes?
You meant "for packages there is the approach...". Actually, that works for every module (__init__.py is a module, just with special semantics). Use __all__ inside the plugin modules and that's it.
But remember: __all__ only limits what you import using from xxxx import *; you can still access the rest of the module, and there's no way to avoid that using the standard Python import mechanism.
If you're using some kind of active introspection technique (eg. exploring the namespace in the module and then importing classes from it), you could check if the class comes from the same file as the module itself.
You could also implement your own import mechanism (using importlib, for example), but that may be overkill...
Edit: for the "check if the class come from the same module":
Say that I have two modules, mod1.py:
class A(object):
pass
and mod2.py:
from mod1 import A
class B(object):
pass
Now, if I do:
from mod2 import *
I've imported both A and B. But...
>>> A
<class 'mod1.A'>
>>> B
<class 'mod2.B'>
as you see, the classes carry information about where did they originate. And actually you can check it right away:
>>> A.__module__
'mod1'
>>> B.__module__
'mod2'
Using that information you can discriminate them easily.
New to python, apologies if this is nub stuff:
Writing a small module (maybe a package, eventually?). It's gotten too big for single file, so I'm moving my larger classes to individual files. All of my classes extend a base class I use to house some rudimentary functions.
Here's a generic form of my problem:
runtime.py:
import baseclass
import subclass
#do stuff
baseclass.py
class BaseClass():
def __init__(self):
pass
def sharedMethod(self):
pass
# shared functionality
subclass.py
class SubClass(baseclass.BaseClass):
def __init__(self):
BaseClass.__init__(self)
The traceback reads:
Traceback (most recent call last):
File "/path/to/runtime.py", line 2, in <module>
import baseclass, subclass
File "path/to/subclass.py", line 2, in <module>
class Subclass(baseclass.BassClass):
NameError: name 'baseclass' is not defined
If I import baseclass into subclass.py it works fine, but this seems to imply that every class which extends baseclass needs to import it, and it musn't be the case that the appropriate way to do this is import baseclass over and over again.
I'd like to be able to import classes from separate files in the order that they build upon each other, I just don't know how to do it. Advice?
Note: All the answers were helpful/correct. I just chose, for the "correct" one, the answer that was most thorough/most likely to be useful to everyone. Cheers, thanks. :)
What you said is exactly right. You have to import baseclass in any file where you want to use it. The right way is indeed to import it "over and over" if you have many files that use it.
But that doesn't mean that every class that extends baseclass needs to import it. It just means that every module that uses baseclass needs to import it. There is no need to have just one class in a given file, and there's no particular reason to do things that way. It's perfectly fine to have import baseclass in one file and then, in that file, define several new classes all inherited from baseclass. That is a good way to do things. Don't split up each class into its own file unless each class really is big enough and complex enough to warrant that.
In addition, doing import baseclass in many files doesn't cost anything in terms of resources. The actual baseclass.py file is only run once, and the same module is re-used on later imports, so you don't pay any extra penality in terms of performance for importing it many times.
import is the way that Python files access stuff from other Python modules. You just can't use things from one file in another file without importing them.
Incidentally, why are you putting that __name__ = "SubClass" business into your modules?
In Python, modules behave like seperate programs. So, if you import baseclass in runtime.py, it will not be imported into subclass.py. subclass.py must import baseclass itself.
I'm considering a package implementation set up like this:
wordproc
__init__.py
_generic.py
gedit.py
oofice.py
word.py
_generic.py would have a class like this:
class WordProc (object):
def __init__ (self):
pass
def createNewDoc (self):
print "createNewDoc unimplemented in current interface"
def getWordCount (self):
print "getWordCount unimplemented in current interface"
etc...
These could print out as shown, or raise errors. App-specific modules would just be copies of _generic.py with the WordProc classes deriving from _generic.WordProc. In this way, functionality could be implemented iteratively over time, with messages about unimplemented things simply raising alerts.
I'm imagining that __init__.py could look for the following things (listed in order) to figure out which module to use:
a wordproc module variable
a settings file in the path
a wordproc environment variable
a function that attempts to determine the environment
a default in __init__.py (probably _generic.py)
I think 3 could be a function in each app's module, or these could go into folders with particularly named environment test scripts (e.g. env.py), and __init__.py could loop over them.
I'd like then in any libraries that want to use wordproc to simply be able to do this:
import wordproc as wp
wp.createNewDoc()
etc...
What I don't know is how to have wp resolve to the proper class in the proper module as determined by __init__.py. It doesn't make sense to do this:
import wordproc.gedit as wp
This destroys the point of having __init__.py determine which module in wordproc to use. I need something like class inheritance, but on the module level.
You can achieve your desired effect by writing __init__.py like this:
Import the appropriate module first. See python docs on importlib.import_module or __import__ for help on dynamic imports.
Instantiate the class from which you want to export methods
Assign the instance methods to locals()
# import appropriate module as mod depending on settings, environment
# using importlib.import_module, or __import__
__all__ = []
_instance = mod.WordProc()
for attr in dir(_instance):
if not attr.startswith('_') and callable(getattr(_instance, attr)):
locals()[attr] = getattr(_instance, attr)
I have a Utilities module which defines a few functions which are repeatedly used and am also adding in some constants. I'm running into trouble importing these constants though...
Let's say I'm working in class A, and I have a class in my constants also named A
from Utils.Constants import A as DistinctA
class A(object):
.... Implementation ....
some_var = DistinctA.SOME_CONSTANT
class Utils(object):
class Constants(object):
class A(object):
SOME_CONSTANT = "Constant"
I'm probably making this too much like Java, so if so just yell / smack my knuckles with a ruler.
When I attempt to import that class, I get an error that there is no module named Constants. What's this python newbie missing?
The identifier after 'from' must point to a module; you can't refer to a class. While I'm not qualified to say whether your nested classes are 'pythonic', I have never seen it done like that before. I'd be more inclined to create a constants.py module that contains the A class. Then you could do this:
from constants import A as DistinctA
If you really want those constants to live inside utils, you could make utils a package:
utils/
utils/__init__.py
utils/constants.py
Then you can do:
from utils.constants import A as DistinctA