I'm considering a package implementation set up like this:
wordproc
__init__.py
_generic.py
gedit.py
oofice.py
word.py
_generic.py would have a class like this:
class WordProc (object):
def __init__ (self):
pass
def createNewDoc (self):
print "createNewDoc unimplemented in current interface"
def getWordCount (self):
print "getWordCount unimplemented in current interface"
etc...
These could print out as shown, or raise errors. App-specific modules would just be copies of _generic.py with the WordProc classes deriving from _generic.WordProc. In this way, functionality could be implemented iteratively over time, with messages about unimplemented things simply raising alerts.
I'm imagining that __init__.py could look for the following things (listed in order) to figure out which module to use:
a wordproc module variable
a settings file in the path
a wordproc environment variable
a function that attempts to determine the environment
a default in __init__.py (probably _generic.py)
I think 3 could be a function in each app's module, or these could go into folders with particularly named environment test scripts (e.g. env.py), and __init__.py could loop over them.
I'd like then in any libraries that want to use wordproc to simply be able to do this:
import wordproc as wp
wp.createNewDoc()
etc...
What I don't know is how to have wp resolve to the proper class in the proper module as determined by __init__.py. It doesn't make sense to do this:
import wordproc.gedit as wp
This destroys the point of having __init__.py determine which module in wordproc to use. I need something like class inheritance, but on the module level.
You can achieve your desired effect by writing __init__.py like this:
Import the appropriate module first. See python docs on importlib.import_module or __import__ for help on dynamic imports.
Instantiate the class from which you want to export methods
Assign the instance methods to locals()
# import appropriate module as mod depending on settings, environment
# using importlib.import_module, or __import__
__all__ = []
_instance = mod.WordProc()
for attr in dir(_instance):
if not attr.startswith('_') and callable(getattr(_instance, attr)):
locals()[attr] = getattr(_instance, attr)
Related
I am trying to implement a small library for Python 3.5 but keep struggling with how to correctly handle the structuring of the packages/modules and how to get the imports to work.
I keep running into the problem where python complains of being unable to import some name with an error like
ImportError: cannot import name 'SubClass1'
This seems to happen when "SubClass1" needs to import some other module but that other module also needs to know about SubClass1 (a cyclic import).
I need the cyclic import in my library because the base class has a factory method that creates the proper subclass instances (there are also other situations where cyclic imports are needed, e.g. checking the type of a function argument needs the import of where that type is defined, but that module may itself need the class where that check is done: another cyclic dependency!)
Here is example code:
Root directory contains the subdirectory dir1. The directory dir1 contains and empty file init.py, a file baseclass.py and a file subclass1.py.
The file ./dir1/subclass1.py contains:
from . baseclass import BaseClass
class SubClass1(BaseClass):
pass
The file ./dir1/baseclass.py contains:
from . subclass1 import SubClass1
class BaseClass(object):
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
# .. which gets eventually returned by this factory method
return ret
The file ./test1.py contains:
from dir1.subclass1 import SubClass1
sc1 = SubClass1()
This results in the following error:
Traceback (most recent call last):
File "test1.py", line 1, in <module>
from dir1.subclass1 import SubClass1
File "/data/johann/tmp/python1/dir1/subclass1.py", line 1, in <module>
from . baseclass import BaseClass
File "/data/johann/tmp/python1/dir1/baseclass.py", line 1, in <module>
from . subclass1 import SubClass1
ImportError: cannot import name 'SubClass1'
What is the standard/best way to solve this problem, ideally in a way that is backwards compatible to python 2.x and python 3 up to version 3.2?
I have read elsewhere that importing the module instead of something from a module may help here but I do not know how to just import the module (e.g. subclass1) in a relative way because "import . subclass1" or similar does not work.
Your issue is caused by a circular import. The baseclass module is trying to import SubClass1 from the subclass1 module, but subclass is trying to import BaseClass right back. You get NameError because the classes haven't been defined yet when the import statements are running.
There are a few ways to solve the issue.
One option would be to change your style of import. Instead of importing the classes by name, just import the modules and look up the names as attributes later on.
from . import baseclass
class SubClass1(baseclass.BaseClass):
pass
And:
from . import subclass1
class BaseClass:
def make(self,somearg):
# ...
ret = subclass1.SubClass1()
Because SubClass1 needs to be able to use BaseClass immediately at definition time, this code may still fail if the baseclass module is imported before subclass1. So it's not ideal
Another option would be to change baseclass to do its import below the definition of BaseClass. This way the subclass module will be able to import the name when it needs to:
class BaseClass:
def make(self,somearg):
# .. some logic to decide which subclass to create
ret = SubClass1()
from .subclass1 import SubClass1
This is not ideal because the normal place to put imports is at the top of the file. Putting them elsewhere makes the code more confusing. You may want to put a comment up at the top of the file explaining why you're delaying the import if you go this route.
Another option may be to combine your two modules into a single file. Python doesn't require each class to have its own module like some other languages do. When you have tightly coupled classes (like the ones in your example), it makes a lot of sense to put them all in one place. This lets you avoid the whole issue, since you don't need any imports at all.
Finally, there are some more complicated solutions, like dependency injection. Rather than the base class needing to know about the subclasses, each subclass could register itself by calling some function and passing a reference to itself. For example:
# no imports of subclasses!
def BaseClass:
subclasses = []
def make(self, somearg):
for sub in self.subclasses:
if sub.accepts(somearg):
return sub()
raise ValueError("no subclass accepts value {!r}".format(somearg))
#classmethod
def register(cls, sub):
cls.subclasses.append(sub)
return sub # return the class so it can be used as a decorator!
And in subclass.py
from .baseclass import BaseClass
#BaseClass.register
class SubClass1(BaseClass):
#classmethod
def accepts(cls, somearg):
# put logic for picking this subclass here!
return True
This style of programming is a bit more complicated, but it can be nice since it's easier to extend than a version where BaseClass needs to know about all of the subclasses up front. There are a variety of ways you can implement this style of code, using a register function is just one of them. One nice thing about it is that it doesn't strictly require inheritance (so you could register a class that doesn't actually inherit from BaseClass if you wanted to). If you are only dealing with actual inheriting subclasses, you might want to consider using a metaclass that does all the registration of subclasses for you automatically.
I have a python source file with a class defined in it, and a class from another module imported into it. Essentially, this structure:
from parent import SuperClass
from other import ClassA
class ClassB(SuperClass):
def __init__(self): pass
What I want to do is look in this module for all the classes defined in there, and only to find ClassB (and to overlook ClassA). Both ClassA and ClassB extend SuperClass.
The reason for this is that I have a directory of plugins which are loaded at runtime, and I get a full list of the plugin classes by introspecting on each .py file and loading the classes which extend SuperClass. In this particular case, ClassB uses the plugin ClassA to do some work for it, so is dependent upon it (ClassA, meanwhile, is not dependent on ClassB). The problem is that when I load the plugins from the directory, I get 2 instances of ClassA, as it gets one from ClassA's file, and one from ClassB's file.
For packages there is the approach:
__all__ = ['module_a', 'module-b']
to explicitly list the modules that you can import, but this lives in the __init__.py file, and each of the plugins is a .py file not a directory in its own right.
The question, then, is: can I limit access to the classes in a .py file, or do I have to make each one of them a directory with its own init file? Or, is there some other clever way that I could distinguish between these two classes?
You meant "for packages there is the approach...". Actually, that works for every module (__init__.py is a module, just with special semantics). Use __all__ inside the plugin modules and that's it.
But remember: __all__ only limits what you import using from xxxx import *; you can still access the rest of the module, and there's no way to avoid that using the standard Python import mechanism.
If you're using some kind of active introspection technique (eg. exploring the namespace in the module and then importing classes from it), you could check if the class comes from the same file as the module itself.
You could also implement your own import mechanism (using importlib, for example), but that may be overkill...
Edit: for the "check if the class come from the same module":
Say that I have two modules, mod1.py:
class A(object):
pass
and mod2.py:
from mod1 import A
class B(object):
pass
Now, if I do:
from mod2 import *
I've imported both A and B. But...
>>> A
<class 'mod1.A'>
>>> B
<class 'mod2.B'>
as you see, the classes carry information about where did they originate. And actually you can check it right away:
>>> A.__module__
'mod1'
>>> B.__module__
'mod2'
Using that information you can discriminate them easily.
If I were to use a class in a module how would I make it "top level" in it's instance?
Structure:
/package
__init__.py
/subPackage
__init__.py
module.py
subModule.py
/theScript.py
python theScript.py
Source of theScript:
import package.subPackage.module
package.subPackage.module.method()
Source of /package/subPackage/module.py:
class module:
def method(self): pass
moduleInstance = module()
I guess what I am asking is how would I make it so that I don't have to do package.subPackage.module.moduleInstance.method() and could package.subPackage.module.method()
I know I can just remove the class and instance but I prefer the class because it makes it easier to subclass later if somebody wants to without needing to modify our source directly but if I ultimately have to I will just use methods instead of class/method in module.
Here is how the random module in the standard lib solved this problem:
_inst = Random()
seed = _inst.seed
random = _inst.random
uniform = _inst.uniform
triangular = _inst.triangular
...
Seems a reasonable solution to me. Of course there is the drawback that you have to manually keep the method lists in sync, but the worst thing that will happen if you forget to add a method name is an error message that tells you exactly what is missing.
I have an application consisting of a base app that brings in several modules. The base app reads a parameter file into a configuration hash, and I want to share it across all my modules.
Currently, I am passing a 'parent' object down to modules, and then those modules are doing stuff like self.parent.config to obtain the configuration.
However, as there are several levels to the module hierarchy, I find myself doing things like self.parent.parent.config, which is starting to look bad.
What are some better patterns for sharing a config object across an application and it's modules? I am thinking about having a 'config' module which basically creates a global config variable that can be set by the base app, then imported and accessed by other modules, but I am not sure if using globals like that is a bad practice for other reasons.
I am reasonably new to Python so be nice =)
You could just:
import config
and have a global config module
excerpts from my comments:
You can always add special rules for odd situations by just saying oddValue if isOddSituation() else config.normalValue.
If you want to have configuration modules be hierarchically subclassable (like my other answer describes), then you can represent a config as a class, or you can use the copy module and make a shallow copy and modify it, or you can use a "config dictionary", e.g.:
import config as baseConfig
config = dict(baseConfig, overriddenValue=etc)
It doesn't really matter too much which scope you're in.
Answering old question:
Just use dependency injection as suggested by #Reed Copsey here. E.g.
class MyClass:
def __init__(myConfig):
self.myConfig = myConfig
...
def foo():
self.myConfig.getConfig(key)
...
self.myConfig.setConfig(key,val)
...
...
# myConfig is your configuration management Module/Class
obj = SomeClass(myConfig)
I think by 'module', you are actually referring to a 'class/object'. An object is an instance of a class, for example:
class MyClass(object):
def __init__(self, ...):
...
...
myObject = MyClass()
A module is a .py file you import, like so:
import mymodule
It seems unlikely that all the classes you instantiate would want to have access to a global configuration. However if you really need everything in your application to have access to some global parameters, you can put them in your own config module:
myParam1 = 1
myParam2 = 2
and then from any module or any object or anywhere really, as long as you did import config, you could just say print(config.myParam1)
Alternatively if you want a large hierarchy of objects to all share access to the same property, you don't need to refer to it via manually setting a self.parent. As long as you use inheritance, you can do stuff like:
class Parent(object):
def __init__(self, theConfig):
self.theConfig = theConfig
class Child(Parent):
...
def method(self,...):
print(self.theConfig)
Take a look at this. It could help you:
https://gist.github.com/dgarana/c052a3287629dd7c0b0c9d7921081e9d
I have a Utilities module which defines a few functions which are repeatedly used and am also adding in some constants. I'm running into trouble importing these constants though...
Let's say I'm working in class A, and I have a class in my constants also named A
from Utils.Constants import A as DistinctA
class A(object):
.... Implementation ....
some_var = DistinctA.SOME_CONSTANT
class Utils(object):
class Constants(object):
class A(object):
SOME_CONSTANT = "Constant"
I'm probably making this too much like Java, so if so just yell / smack my knuckles with a ruler.
When I attempt to import that class, I get an error that there is no module named Constants. What's this python newbie missing?
The identifier after 'from' must point to a module; you can't refer to a class. While I'm not qualified to say whether your nested classes are 'pythonic', I have never seen it done like that before. I'd be more inclined to create a constants.py module that contains the A class. Then you could do this:
from constants import A as DistinctA
If you really want those constants to live inside utils, you could make utils a package:
utils/
utils/__init__.py
utils/constants.py
Then you can do:
from utils.constants import A as DistinctA