I'm implementing a function that returns an object from the appropriate subclass. If I move SubClass from base.py, no subclasses appear for __subclasses__. Are they required to be in the same file?
Perhaps the fact that I'm never importing directly subclass.py hides the subclass from python? What can I do? I have even checked the attribute __mro__ and get_subclass points to the right class.
# project/main.py
from project.src.base import get_subclass
obj = get_subclass(cls,name) # Returns an object of a subclass of cls
# project/src/subclass.py
from project.src.base import BaseClass
class SubClass(BaseClass):
pass
# project/src/base.py
def get_subclass(cls,name):
subclss = cls.__subclasses__ # This is returning an empty list
pass
class BaseClass(object):
pass
Python only runs code of modules that are imported. If you move code to a different module but never import it, Python does not know about its contents.
You have to import the files containing the subclasses you want accessible.
# project/src/__init__.py
import project.src.base # executes the ``BaseClass`` definition
import project.src.subclass # executes the ``SubClass`` definition
Note that it does not really matter where you import these - they must be imported before you need SubClass to appear in __subclasses__, though.
Related
I am trying to implement a small library for Python 3.5 but keep struggling with how to correctly handle the structuring of the packages/modules and how to get the imports to work.
I keep running into the problem where python complains of being unable to import some name with an error like
ImportError: cannot import name 'SubClass1'
This seems to happen when "SubClass1" needs to import some other module but that other module also needs to know about SubClass1 (a cyclic import).
I need the cyclic import in my library because the base class has a factory method that creates the proper subclass instances (there are also other situations where cyclic imports are needed, e.g. checking the type of a function argument needs the import of where that type is defined, but that module may itself need the class where that check is done: another cyclic dependency!)
Here is example code:
Root directory contains the subdirectory dir1. The directory dir1 contains and empty file init.py, a file baseclass.py and a file subclass1.py.
The file ./dir1/subclass1.py contains:
from . baseclass import BaseClass
class SubClass1(BaseClass):
pass
The file ./dir1/baseclass.py contains:
from . subclass1 import SubClass1
class BaseClass(object):
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
# .. which gets eventually returned by this factory method
return ret
The file ./test1.py contains:
from dir1.subclass1 import SubClass1
sc1 = SubClass1()
This results in the following error:
Traceback (most recent call last):
File "test1.py", line 1, in <module>
from dir1.subclass1 import SubClass1
File "/data/johann/tmp/python1/dir1/subclass1.py", line 1, in <module>
from . baseclass import BaseClass
File "/data/johann/tmp/python1/dir1/baseclass.py", line 1, in <module>
from . subclass1 import SubClass1
ImportError: cannot import name 'SubClass1'
What is the standard/best way to solve this problem, ideally in a way that is backwards compatible to python 2.x and python 3 up to version 3.2?
I have read elsewhere that importing the module instead of something from a module may help here but I do not know how to just import the module (e.g. subclass1) in a relative way because "import . subclass1" or similar does not work.
Your issue is caused by a circular import. The baseclass module is trying to import SubClass1 from the subclass1 module, but subclass is trying to import BaseClass right back. You get NameError because the classes haven't been defined yet when the import statements are running.
There are a few ways to solve the issue.
One option would be to change your style of import. Instead of importing the classes by name, just import the modules and look up the names as attributes later on.
from . import baseclass
class SubClass1(baseclass.BaseClass):
pass
And:
from . import subclass1
class BaseClass:
def make(self,somearg):
# ...
ret = subclass1.SubClass1()
Because SubClass1 needs to be able to use BaseClass immediately at definition time, this code may still fail if the baseclass module is imported before subclass1. So it's not ideal
Another option would be to change baseclass to do its import below the definition of BaseClass. This way the subclass module will be able to import the name when it needs to:
class BaseClass:
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
from .subclass1 import SubClass1
This is not ideal because the normal place to put imports is at the top of the file. Putting them elsewhere makes the code more confusing. You may want to put a comment up at the top of the file explaining why you're delaying the import if you go this route.
Another option may be to combine your two modules into a single file. Python doesn't require each class to have its own module like some other languages do. When you have tightly coupled classes (like the ones in your example), it makes a lot of sense to put them all in one place. This lets you avoid the whole issue, since you don't need any imports at all.
Finally, there are some more complicated solutions, like dependency injection. Rather than the base class needing to know about the subclasses, each subclass could register itself by calling some function and passing a reference to itself. For example:
# no imports of subclasses!
def BaseClass:
subclasses = []
def make(self, somearg):
for sub in self.subclasses:
if sub.accepts(somearg):
return sub()
raise ValueError("no subclass accepts value {!r}".format(somearg))
#classmethod
def register(cls, sub):
cls.subclasses.append(sub)
return sub # return the class so it can be used as a decorator!
And in subclass.py
from .baseclass import BaseClass
#BaseClass.register
class SubClass1(BaseClass):
#classmethod
def accepts(cls, somearg):
# put logic for picking this subclass here!
return True
This style of programming is a bit more complicated, but it can be nice since it's easier to extend than a version where BaseClass needs to know about all of the subclasses up front. There are a variety of ways you can implement this style of code, using a register function is just one of them. One nice thing about it is that it doesn't strictly require inheritance (so you could register a class that doesn't actually inherit from BaseClass if you wanted to). If you are only dealing with actual inheriting subclasses, you might want to consider using a metaclass that does all the registration of subclasses for you automatically.
I have two modules called Dfs and Graph.
In my Graph module, I have a class Graph and a method called ReadGraph.
In my Dfs module, I have a method that calls on ReadGraph but I get the following error message when I type: Dfs.ProcessGraph(testcase.txt,verbose=True)
Error message:
NameError: name 'testcase' is not defined
Could someone explain how to fix this?
Thanks.
From my Dfs.py module:
import sys
from Graph import *
class Dfs( object ):
def ProcessGraph(file_name, verbose):
g=ReadGraph(file_name)
From my Graph.py module:
class Graph( object ):
def ReadGraph( file_name ):
Remove your class declaration from Graph.py. When you import all from a file you get all top level objects. In this case it's the Graph class itself, not its methods.
Also you need to pass string 'testcase.txt' not testcase.txt.
You have multiple problems here:
If you from Graph import * (which is bad practice to start with), you bring Graph into your name space. However, ReadGraph is inside Graph, so to access it you need Graph.ReadGraph.
Having done that, you try to call Dfs.ProcessGraph(testcase.txt,verbose=True). The first argument is interpreted as "pass the txt attribute of the object referenced by name testcase, which doesn't exist. Instead, you mean "testcase.txt" (quoted to make it a string).
Having done all of that, you get e.g. TypeError: unbound method ProcessGraph() must be called with Dfs instance as first argument (got str instance instead). When you call an instance method, the first argument, self by convention, is the instance itself. You have two choices; either a) make e.g. ProcessGraph a #staticmethod, and access it Graph.ReadGraph; or b) move it outside the class, then you can access it directly like you tried to in the first place. As you don't seem to have any class or instance attributes, it's not clear why you are bothering with the classes at all.
What it should probably look like:
import sys
from Graph import read_graph
def process_graph(file_name, verbose):
g = read_graph(file_name)
Graph.py module (note absence of class Graph):
def read_graph(file_name):
...
(Generally, I suggest you read PEP 8).
your code should be : Dfs.ProcessGraph('testcase.txt',verbose=True)
and not Dfs.ProcessGraph(testcase.txt,verbose=True)
'testcase.txt' # is a string and should be between quotes
also check if it is in the same directory where your code live else point to it
plus in DFs you should instantiate Graph :
from Graph.Graph import *
g = Graph()
grf = g.ReadGraph('filename')
EDIT: to be more precise
in Graph module:
class Graph(object):
def __init__(self):
pass # for the example now
def read_graph(self, file_name):
return file_name
in Dfs Module:
from Graph import *
class Dfs(object):
def __init__(self):
pass # for the example now
def ProcessGraph(file_name, verbose):
g = Graph()
file_name = Graph.read_graph(file_name)
I have a large python code with many modules and classes. I have a special class, whose single instance is needed everywhere throughout the code (it's a threaded application, and that instance of a class also holds Thread Local Storage, locks, etc). It's a bit uncomfortable to always "populate" that instance in every imported module. I know, using globals is not the best practice, but anyway: is there any "import hook" in python, so I can do with hooking on it to have my instance available in every modules without extra work? It should work for normal imports, "from mod import ..." too, and for import constructs too. If this is not possible, can you suggest a better solution? Certenly it's not fun to pass that instance to the constructors of every classes, etc ... Inheritance also does not help, since I have modules without classes, and also I need a single instance, not the class itself ...
class master():
def import_module(self, name):
mod = __import__(name)
mod.m = self
return mod
[...]
m = master()
Currently I am thinking something like that: but then I have to use m.import_module() to import modules, then other modules will have instance of master class with name of "m" available, so I can use m.import_module() too, etc. But then I have to give up to use "normal" import statements, and I should write this:
example_mod = m.module_import("example_mod")
instead of just this:
import example_mod
(but for sure I can do with this too, to assign "m" to example_mod.m then)
Certainly it's not fun to pass that instance to the constructors of
every classes
You don't have to do this. Set up your global class in a module like config and import it
# /myapp/enviroment/__init__.py
class ThatSingleInstanceClass: pass
# create the singleton object directly or have a function init the module
singleton = ThatSingleInstanceClass()
# /myapp/somewhere.py
# all you need to use the object is importing it
from myapp.enviroment import singleton
class SomeClass:
def __init__(self): # no need to pass that object
print "Always the same object:", singleton
What's wrong with having each module import the needed object? Explicit is better than implicit.
I'm trying to 'intercept' all calls to a specific module, and reroute them to another object. I'd like to do this so that I can have a fairly simple plugin architecture.
For example, in main.py
import renderer
renderer.draw('circle')
In renderer.py
specificRenderer = OpenGLRenderer()
#Then, i'd like to route all calls from main.py so that
#specificRenderer.methodName(methodArgs) is called
# i.e. the above example would call specificRenderer.draw('circle')
This means that any function can just import renderer and use it, without worrying about the details. It also means that I can completely change the renderer just by creating another object and assigning it to the 'specificRenderer' value in renderer.py
Any ideas?
In renderer.py:
import sys
if __name__ != "__main__":
sys.modules[__name__] = OpenGLRenderer()
The module name is now mapped to the OpenGLRenderer instance, and import renderer in other modules will get the same instance.
Actually, you don't even need the separate module. You can just do:
import sys
sys.modules["renderer"] = OpenGLRenderer()
import renderer # gives current module access to the "module"
... first thing in your main module. Imports of renderer in other modules, once again, will refer to the same instance.
Are you sure you really want to do this in the first place? It isn't really how people expect modules to behave.
The simplest way to do that is to have main.py do
from renderer import renderer
instead, then just name specificRenderer renderer.
My answer is very similar to #kindall's although I got the idea elsewhere. It goes a step further in the sense that it replaces the module object that's usually put in the sys.modules list with an instance of a class of your own design. At a minimum such a class would need to look something like this:
File renderer.py:
class _renderer(object):
def __init__(self, specificRenderer):
self.specificRenderer = specificRenderer
def __getattr__(self, name):
return getattr(self.specificRenderer, name)
if __name__ != '__main__':
import sys
# from some_module import OpenGLRenderer
sys.modules[__name__] = _renderer(OpenGLRenderer())
The __getattr__() method simply forwards most attribute accesses on to the real renderer object. The advantage to this level of indirection is that with it you can add your own attributes to the private _renderer class and access them through the renderer object imported just as though they were part of an OpenGLRenderer object. If you give them the same name as something already in an OpenGLRenderer object, they will be called instead, are free to forward, log, ignore, and/or modify the call before passing it along -- which can sometimes be very handy.
Class instances placed in sys.modules are effectively singletons, so if the module is imported in other scripts in the application, they will all share the single instance created by the first one.
If you don't mind that import renderer results an object rather than a module, then see kindall's brilliant solution.
If you want to make #property work (i.e. each time you fetch renderer.mytime, you want the function corresponding to OpenGLRenderer.mytime get called) and you want to keep renderer as a module, then it's impossible. Example:
import time
class OpenGLRenderer(object):
#property
def mytime(self):
return time.time()
If you don't care about properties, i.e. it's OK for you that mytime gets called only once (at module load time), and it will keep returning the same timestamp, then it's possible to do it by copying all symbols from the object to the module:
# renderer.py
specificRenderer = OpenGLRenderer()
for name in dir(specificRenderer):
globals()[name] = getattr(specificRenderer, name)
However, this is a one-time copy. If you add methods or other attributes to specificRenderer later dynamically, or change some attributes later, then they won't be automatically copied to the renderer module. This can be fixed, however, by some ugly __setattr__ hacking.
Edit: This answer does not do what the OP wants; it doesn't instantiate an object and then let calls to a module be redirected to that same object. This answer is about changing which rendering module is being used.
Easiest might be to import the OpenGLRenderer in the main.py program like this:
import OpenGLRenderer as renderer
That's code in just one place, and in the rest of your module OpenGLRenderer can be referred to as renderer.
If you have several modules like main.py, you could have your renderer.py file be just the same line:
import OpenGLRenderer as renderer
and then other modules can use
from renderer import renderer
If OpenGLRenderer doesn't quite quack right yet, you can monkeypatch it to work as you need in the renderer.py module.
Suppose you have the following layout for a python package
./a
./a/__init__.py
./a/_b.py
inside __init__.py you have
from _b import *
and inside _b.py you have
class B(object): pass
If you import from interactive prompt
>>> import a
>>> a.B
<class 'a._b.B'>
>>>
How can I completely hide the existence of _b ?
The problem I am trying to solve is the following: I want a facade package importing "hidden" modules and classes. The classes available from the facade (in my case a) are kept stable and guaranteed for the future. I want, however, freedom to relocate classes "under the hood", hence the hidden modules. This is all nice, but if some client code pickles an object provided by the facade, this pickled data will refer to the hidden module nesting, not to the facade nesting. In other words, if I reposition the B class in a module _c.py, client codes will not be able to unpickle because the pickled classes are referring to a._b.B, which has been moved. If they referred to a.B, I could relocate the B class as much as I want under the hood, without ruining pickled data.
try:
B.__module__= 'a'
Incidentally you probably want an absolute import:
from a._b import *
as relative imports without the new explicit dot syntax are going away (see PEP 328).
ETA re comment:
I would have to set the module explicitly for every class
Yes, I don't think there's a way around that but you could at least automate it, at the end of __init__:
for value in globals().values():
if inspect.isclass(value) and value.__module__.startswith('a.'):
value.__module__= 'a'
(For new-style classes only you could get away with isinstance(value, type) instead of inspect. If the module doesn't have to run as __main__ you could use __name__ instead of hard-coding 'a'.)
You could set the __module__ variable for Class B
class B(object): pass
B.__module__ = 'a'
For classes, functions, and methods, this attribute contains the name of the module in which the object was defined.
Or define it once in your __init__.py:
from a._b import B # change this line, when required, e.g. from a._c import B
B.__module__ = 'a'