I have two Python classes in two separate modules. The modules reference each other, so I must use the import XXX syntax rather than from XXX import YYY. I can't figure out how to access a class defiend in one module in the other, importing module though:
### testa.py
import testb
class TestA():
...
### testb.py
import testa
class TestB(testa.TestA): # doesn't work
...
How do I do this?
You can solve this by putting import testb after the definition of TestA. However, you might need to rethink your module structure since circular imports are difficult to deal with.
Related
This question already has answers here:
What can I do about "ImportError: Cannot import name X" or "AttributeError: ... (most likely due to a circular import)"?
(17 answers)
Closed 6 months ago.
I know the issue of circular imports in python has come up many times before and I have read these discussions. The comment that is made repeatedly in these discussions is that a circular import is a sign of a bad design and the code should be reorganised to avoid the circular import.
Could someone tell me how to avoid a circular import in this situation?: I have two classes and I want each class to have a constructor (method) which takes an instance of the other class and returns an instance of the class.
More specifically, one class is mutable and one is immutable. The immutable class is needed
for hashing, comparing and so on. The mutable class is needed to do things too. This is similar to sets and frozensets or to lists and tuples.
I could put both class definitions in the same module. Are there any other suggestions?
A toy example would be class A which has an attribute which is a list and class B which has an attribute which is a tuple. Then class A has a method which takes an instance of class B and returns an instance of class A (by converting the tuple to a list) and similarly class B has a method which takes an instance of class A and returns an instance of class B (by converting the list to a tuple).
Consider the following example python package where a.py and b.py depend on each other:
/package
__init__.py
a.py
b.py
Types of circular import problems
Circular import dependencies typically fall into two categories depending
on what you're trying to import and where you're using it inside each
module. (And whether you're using python 2 or 3).
1. Errors importing modules with circular imports
In some cases, just importing a module with a circular import dependency
can result in errors even if you're not referencing anything from the
imported module.
There are several standard ways to import a module in python
import package.a # (1) Absolute import
import package.a as a_mod # (2) Absolute import bound to different name
from package import a # (3) Alternate absolute import
import a # (4) Implicit relative import (deprecated, python 2 only)
from . import a # (5) Explicit relative import
Unfortunately, only the 1st and 4th options actually work when you
have circular dependencies (the rest all raise ImportError
or AttributeError). In general, you shouldn't be using the
4th syntax, since it only works in python2 and runs the risk of
clashing with other 3rd party modules. So really, only the first
syntax is guaranteed to work.
EDIT: The ImportError and AttributeError issues only occur in
python 2. In python 3 the import machinery has been rewritten and all
of these import statements (with the exception of 4) will work, even with
circular dependencies. While the solutions in this section may help refactoring python 3 code, they are mainly intended
for people using python 2.
Absolute Import
Just use the first import syntax above. The downside to this method is
that the import names can get super long for large packages.
In a.py
import package.b
In b.py
import package.a
Defer import until later
I've seen this method used in lots of packages, but it still feels
hacky to me, and I dislike that I can't look at the top of a module
and see all its dependencies, I have to go searching through all the
functions as well.
In a.py
def func():
from package import b
In b.py
def func():
from package import a
Put all imports in a central module
This also works, but has the same problem as the first method, where
all the package and submodule calls get super long. It also has two
major flaws -- it forces all the submodules to be imported, even if
you're only using one or two, and you still can't look at any of the
submodules and quickly see their dependencies at the top, you have to
go sifting through functions.
In __init__.py
from . import a
from . import b
In a.py
import package
def func():
package.b.some_object()
In b.py
import package
def func():
package.a.some_object()
2. Errors using imported objects with circular dependencies
Now, while you may be able to import a module with a circular import
dependency, you won't be able to import any objects defined in the module
or actually be able to reference that imported module anywhere
in the top level of the module where you're importing it. You can,
however, use the imported module inside functions and code blocks that don't
get run on import.
For example, this will work:
package/a.py
import package.b
def func_a():
return "a"
package/b.py
import package.a
def func_b():
# Notice how package.a is only referenced *inside* a function
# and not the top level of the module.
return package.a.func_a() + "b"
But this won't work
package/a.py
import package.b
class A(object):
pass
package/b.py
import package.a
# package.a is referenced at the top level of the module
class B(package.a.A):
pass
You'll get an exception
AttributeError: module 'package' has no attribute 'a'
Generally, in most valid cases of circular dependencies, it's possible
to refactor or reorganize the code to prevent these errors and move
module references inside a code block.
Only import the module, don't import from the module:
Consider a.py:
import b
class A:
def bar(self):
return b.B()
and b.py:
import a
class B:
def bar(self):
return a.A()
This works perfectly fine.
We do a combination of absolute imports and functions for better reading and shorter access strings.
Advantage: Shorter access strings compared to pure absolute imports
Disadvantage: a bit more overhead due to extra function call
main/sub/a.py
import main.sub.b
b_mod = lambda: main.sub.b
class A():
def __init__(self):
print('in class "A":', b_mod().B.__name__)
main/sub/b.py
import main.sub.a
a_mod = lambda: main.sub.a
class B():
def __init__(self):
print('in class "B":', a_mod().A.__name__)
i have following project structure: 1
and here is content of files:
# run.py
from module.submodule.base import DefaultObject
d = DefaultObject()
# module/sumbodule/base.py
from module.submodule.modulea import A
class BaseObject(object):
pass
class DefaultObject(BaseObject):
def return_something(self):
return A()
# module/submodule/modulea.py
from module.submodule.moduleb import B
class A(object):
def return_something(self):
return B()
# module/submodule/moduleb.py
from module.submodule.base import BaseObject
class B(BaseObject):
pass
and when I try to run python3 run.py I get ImportError: cannot import name 'BaseObject
I don't understand why I am able to import class B in modulea.py, but I am not able to class BaseObject in moduleb.py
What is correct way to make imports in such situation?
You have a circular import - base imports modulea which imports moduleb which imports base. Python doesn't support circular imports so it cannot technically work, and even for languages that technically supports them, circular dependencies are a very bad thing anyway.
Your solutions here are either to regroup interdependant objects (classes, functions etc) in a same module - note that Python is not Java and doesn't require "one module per class" (it's even actually an antipattern in Python) - or to move DefaultObject to it's own module.
I have a python source file with a class defined in it, and a class from another module imported into it. Essentially, this structure:
from parent import SuperClass
from other import ClassA
class ClassB(SuperClass):
def __init__(self): pass
What I want to do is look in this module for all the classes defined in there, and only to find ClassB (and to overlook ClassA). Both ClassA and ClassB extend SuperClass.
The reason for this is that I have a directory of plugins which are loaded at runtime, and I get a full list of the plugin classes by introspecting on each .py file and loading the classes which extend SuperClass. In this particular case, ClassB uses the plugin ClassA to do some work for it, so is dependent upon it (ClassA, meanwhile, is not dependent on ClassB). The problem is that when I load the plugins from the directory, I get 2 instances of ClassA, as it gets one from ClassA's file, and one from ClassB's file.
For packages there is the approach:
__all__ = ['module_a', 'module-b']
to explicitly list the modules that you can import, but this lives in the __init__.py file, and each of the plugins is a .py file not a directory in its own right.
The question, then, is: can I limit access to the classes in a .py file, or do I have to make each one of them a directory with its own init file? Or, is there some other clever way that I could distinguish between these two classes?
You meant "for packages there is the approach...". Actually, that works for every module (__init__.py is a module, just with special semantics). Use __all__ inside the plugin modules and that's it.
But remember: __all__ only limits what you import using from xxxx import *; you can still access the rest of the module, and there's no way to avoid that using the standard Python import mechanism.
If you're using some kind of active introspection technique (eg. exploring the namespace in the module and then importing classes from it), you could check if the class comes from the same file as the module itself.
You could also implement your own import mechanism (using importlib, for example), but that may be overkill...
Edit: for the "check if the class come from the same module":
Say that I have two modules, mod1.py:
class A(object):
pass
and mod2.py:
from mod1 import A
class B(object):
pass
Now, if I do:
from mod2 import *
I've imported both A and B. But...
>>> A
<class 'mod1.A'>
>>> B
<class 'mod2.B'>
as you see, the classes carry information about where did they originate. And actually you can check it right away:
>>> A.__module__
'mod1'
>>> B.__module__
'mod2'
Using that information you can discriminate them easily.
Here's a python module. foo is in sys.path.
foo\
__init__.py
bar\
__init__.py
base.py
class Base(object)
derived.py
import foo.bar.base as base
class Derived(base.Base)
I've got nothing fancy going on yet. If I want to instantiate the Derived class from the derived module, I can do that easily enough:
import foo.bar.derived as derived
print(derived.Derived())
However, I'd like to just import the bar module and call bar.Derived(), because I plan to have lots of classes within lots of different modules, and I don't want to deal with all these tentacular import paths. My understanding is that I can simply import Derived into the namespace of the bar module, by modifying my project like so:
foo\
__init__.py
bar\
__init__.py
from foo.bar.derived import Derived
base.py
class Base(object)
derived.py
import foo.bar.base as base
class Derived(base.Base)
Now I should be able to do the following:
import foo.bar as bar
print(bar.Derived())
But I get an AttributeError complaining that the foo module has no submodule called bar:
test.py (1): import foo.bar
foo\bar\__init__.py (1): from foo.bar.derived import Derived
foo\bar\derived.py (1): import foo.bar.base as base
AttributeError: 'module' object has no attribute 'bar'
In fact, my original test code (at top) doesn't work either! As soon as I try to import foo.bar, I get errors.
What I can gleam from this error is that the import statement in __init__.py causes derived.py to be executed before bar is fully loaded, and therefore it can't import the module (also from bar) which contains its own base class. I'm coming from the C++ world, where ultra-nested namespaces aren't as integral and a simple forward declaration would negate this problem, but I've been led to believe that what I'm looking for is possible and at least a somewhat acceptably Pythonic solution. What am I doing wrong? What's the correct way to make classes from a submodule available in the parent module's namespace?
If you're working with Python 2.5 or later, try using explicit relative imports (http://www.python.org/dev/peps/pep-0328/#guido-s-decision):
test.py (1): import foo.bar
foo\bar\__init__.py (1): from .derived import Derived
foo\bar\derived.py (1): from . import base
(Note that if you are indeed working with Python 2.5 or 2.6, you'll need to include from __future__ import absolute_import in your modules.)
in derived.py, use this:
EDIT: as JAB pointed out, implicit relative imports are deprecated, to the following isn't recommended (although it does work still in Python 2.7 - with no deprecation errors!).
import base # this is all you need - it's in the current directory
Instead, use:
from . import base #
(or)
from foo.bar import base
instead of:
import foo.bar.base as base
This will solve both your errors (since they're from the same issue). Your import doesn't work since there is no base function or class inside the foo.bar.base module.
I have a Utilities module which defines a few functions which are repeatedly used and am also adding in some constants. I'm running into trouble importing these constants though...
Let's say I'm working in class A, and I have a class in my constants also named A
from Utils.Constants import A as DistinctA
class A(object):
.... Implementation ....
some_var = DistinctA.SOME_CONSTANT
class Utils(object):
class Constants(object):
class A(object):
SOME_CONSTANT = "Constant"
I'm probably making this too much like Java, so if so just yell / smack my knuckles with a ruler.
When I attempt to import that class, I get an error that there is no module named Constants. What's this python newbie missing?
The identifier after 'from' must point to a module; you can't refer to a class. While I'm not qualified to say whether your nested classes are 'pythonic', I have never seen it done like that before. I'd be more inclined to create a constants.py module that contains the A class. Then you could do this:
from constants import A as DistinctA
If you really want those constants to live inside utils, you could make utils a package:
utils/
utils/__init__.py
utils/constants.py
Then you can do:
from utils.constants import A as DistinctA