Suppose you have the following layout for a python package
./a
./a/__init__.py
./a/_b.py
inside __init__.py you have
from _b import *
and inside _b.py you have
class B(object): pass
If you import from interactive prompt
>>> import a
>>> a.B
<class 'a._b.B'>
>>>
How can I completely hide the existence of _b ?
The problem I am trying to solve is the following: I want a facade package importing "hidden" modules and classes. The classes available from the facade (in my case a) are kept stable and guaranteed for the future. I want, however, freedom to relocate classes "under the hood", hence the hidden modules. This is all nice, but if some client code pickles an object provided by the facade, this pickled data will refer to the hidden module nesting, not to the facade nesting. In other words, if I reposition the B class in a module _c.py, client codes will not be able to unpickle because the pickled classes are referring to a._b.B, which has been moved. If they referred to a.B, I could relocate the B class as much as I want under the hood, without ruining pickled data.
try:
B.__module__= 'a'
Incidentally you probably want an absolute import:
from a._b import *
as relative imports without the new explicit dot syntax are going away (see PEP 328).
ETA re comment:
I would have to set the module explicitly for every class
Yes, I don't think there's a way around that but you could at least automate it, at the end of __init__:
for value in globals().values():
if inspect.isclass(value) and value.__module__.startswith('a.'):
value.__module__= 'a'
(For new-style classes only you could get away with isinstance(value, type) instead of inspect. If the module doesn't have to run as __main__ you could use __name__ instead of hard-coding 'a'.)
You could set the __module__ variable for Class B
class B(object): pass
B.__module__ = 'a'
For classes, functions, and methods, this attribute contains the name of the module in which the object was defined.
Or define it once in your __init__.py:
from a._b import B # change this line, when required, e.g. from a._c import B
B.__module__ = 'a'
Related
I would like to convert a singleton-object programmatically into a Python module so that I can use the methods of this singleton-object directly by importing them via the module instead of accessing them as object attributes. By "programmatically" I mean that I do not want to have to copy-paste the class methods explicitly into a module file. I need some sort of a workaround that allows me to import the object methods into to global scope of another module.
I would really appreciate if someone could help me on this one.
Here is a basic example that should illustrate my problem:
mymodule.py
class MyClass:
"""This is my custom class"""
def my_method(self):
return "myValue"
singleton = MyClass()
main_as_is.py
from mymodule import MyClass
myobject = MyClass()
print(myobject.my_method())
main_to_be.py
from mymodule import my_method # or from mymodule.singleton import my_method
print(my_method())
You can use the same strategy that the standard random module uses. All the functions in that module are actually methods of a "private" instance of the Random class. That's convenient for most common uses of the module, although sometimes it's useful to create your own instances of Random so that you can have multiple independent random streams.
I've adapted your code to illustrate that technique. I named the class and its instance with a single leading underscore, since that's the usual convention in Python to signify a private name, but bear in mind it's simply a convention, Python doesn't do anything to enforce this privacy.
mymodule.py
class _MyClass:
""" This is my custom class """
def my_method(self):
return "myValue"
_myclass = _MyClass()
my_method = _myclass.my_method
main_to_be.py
from mymodule import my_method
print(my_method())
output
myValue
BTW, the from mymodule import method1, method2 syntax is ok if you only import a small number of names, or it's clear from the name which module it's from (like math module functions and constants), and you don't import from many modules. Otherwise it's better to use this sort of syntax
import mymodule as mm
# Call a method from the module
mm.method1()
That way it's obvious which names are local, and which ones are imported and where they're imported from. Sure, it's a little more typing, but it makes the code a whole lot more readable. And it eliminates the possibility of name collisions.
FWIW, here's a way to automate adding all of the _myclass methods without explicitly listing them (but remember "explicit is better than implicit"). At the end of "mymodule.py", in place of my_method = _myclass.my_method, add this:
globals().update({k: getattr(_myclass, k) for k in _MyClass.__dict__
if not k.startswith('__')})
I'm not comfortable with recommending this, since it directly injects items into the globals() dict. Note that that code will add all class attributes, not just methods.
In your question you talk about singleton objects. We don't normally use singletons in Python, and many programmers in various OOP languages consider them to be an anti-pattern. See https://stackoverflow.com/questions/12755539/why-is-singleton-considered-an-anti-pattern for details. For this application there is absolutely no need at all to use a singleton. If you only want a single instance of _MyClass then simply don't create another instance of it, just use the instance that mymodule creates for you. But if your boss insists that you must use a singleton, please see the example code here.
I am struggling to figure out how to handle importing dependencies that are used in multiple files.
Let's say I want to import an external API for example, and two classes depend on this import. Putting the import into the 'index' file, as an attempt to make it global does not work. I can import it in each class file fine, but that seems to be a violation of DRY, as well as setting myself up for failure later on.
So is there a way to import once, in a single file that is globally accessible? What I experimented with was creating an index.py, foo.py (for the foo class) and bar.py (for the bar class):
Index:
from example import API
import foo
import bar
foo()
bar()
foo.py:
class foo:
... (try to put the example API to use)
bar.py: (same as foo.py really, just here to make the case for using the same dependency in two different places)
This failed to work, as the classes appeared to not be able to access exampleAPI. What is the correct way to do this, or am I looking at it wrong? Thanks!
In general, you should import each module you need in each of your own modules that needs to use it. You don't need to worry about duplication, since each module has its own global namespace. Furthermore, modules are cached (in the sys.modules dictionary) so you don't need to worry about extra work being done to load the module multiple times.
That said, there can be some exceptions. For instance, if the specific source of an API is considered "private" information (e.g. because it's an implementation detail or because it might be configurable and not always come from the same place all the time), it might make sense to import it into some namespace where all other users will look for it.
On the other hand, your example suggests you may be splitting up your code more than you should. Unlike some other languages (such as Java), in Python it's neither required nor recommended for each class to live in its own file. Instead, you should divide your code up into modules dictated by how closely they interact with each other. Closely related classes should be part of the same module, while pieces that don't interact at all might make more sense in separate modules (especially if some other code might need one part but not the other). It may not be inappropriate for your whole program to be in a single module! Obviously, some of this is a matter of style and taste, so there's not a single best option for every programmer in every situation.
For your example code, if you want to keep separate modules, I'd suggest something like this:
index.py:
from foo import Foo # no need to import API here if you're not using it directly
from bar import Bar
foo = Foo() # create an instance of the foo class
result = foo.some_method() # call methods on it
bar = Bar(foo) # you can also pass your instances around to other classes
foo.py:
from example import API
class Foo:
def some_method(self):
return API.whatever() # use the API in some way
bar.py:
from example import API # don't worry about importing the API more than ocne
class Bar:
def __init__(self, foo):
self.foo = foo
def blah(self):
API.something_else(self.foo.some_method())
Note that I changed some names around. Python's convention is to use CapitalizedNames for classes, and lowercase_names_with_underscores (sometimes known as "snake case") for modules, variables and methods. Your original code seemed to have some confusion between the modules name foo and bar and the classes within them with the same names. Using different styles for the different names can help avoiding that confusion.
I have a python source file with a class defined in it, and a class from another module imported into it. Essentially, this structure:
from parent import SuperClass
from other import ClassA
class ClassB(SuperClass):
def __init__(self): pass
What I want to do is look in this module for all the classes defined in there, and only to find ClassB (and to overlook ClassA). Both ClassA and ClassB extend SuperClass.
The reason for this is that I have a directory of plugins which are loaded at runtime, and I get a full list of the plugin classes by introspecting on each .py file and loading the classes which extend SuperClass. In this particular case, ClassB uses the plugin ClassA to do some work for it, so is dependent upon it (ClassA, meanwhile, is not dependent on ClassB). The problem is that when I load the plugins from the directory, I get 2 instances of ClassA, as it gets one from ClassA's file, and one from ClassB's file.
For packages there is the approach:
__all__ = ['module_a', 'module-b']
to explicitly list the modules that you can import, but this lives in the __init__.py file, and each of the plugins is a .py file not a directory in its own right.
The question, then, is: can I limit access to the classes in a .py file, or do I have to make each one of them a directory with its own init file? Or, is there some other clever way that I could distinguish between these two classes?
You meant "for packages there is the approach...". Actually, that works for every module (__init__.py is a module, just with special semantics). Use __all__ inside the plugin modules and that's it.
But remember: __all__ only limits what you import using from xxxx import *; you can still access the rest of the module, and there's no way to avoid that using the standard Python import mechanism.
If you're using some kind of active introspection technique (eg. exploring the namespace in the module and then importing classes from it), you could check if the class comes from the same file as the module itself.
You could also implement your own import mechanism (using importlib, for example), but that may be overkill...
Edit: for the "check if the class come from the same module":
Say that I have two modules, mod1.py:
class A(object):
pass
and mod2.py:
from mod1 import A
class B(object):
pass
Now, if I do:
from mod2 import *
I've imported both A and B. But...
>>> A
<class 'mod1.A'>
>>> B
<class 'mod2.B'>
as you see, the classes carry information about where did they originate. And actually you can check it right away:
>>> A.__module__
'mod1'
>>> B.__module__
'mod2'
Using that information you can discriminate them easily.
I suspect this is one of those questions that will result in "Why do you want to do that?" but here goes:
I need to find a way to fake the module of a class defined in __main__; ie:
Lets say we have a.py:
class A(object):
name = "a"
pass
And b.py:
import inspect
import a
class B(object):
__name__ = a.__name__
name = "b"
pass
print inspect.getmodule(B)
Calling b.py results in:
<module '__main__' from 'b.py'>
So, how do I get the module of class A taken to be b, without moving it into b ?
You can set the class's __module__ attribute, but there are caveats:
class A(object):
__module__ = 'b'
The main caveat is that this will cause getmodule to return None if the module b has not imported at the time you call getmodule(A). I think it's also possible for problems to arise if you import b in some nonstandard way, like importing it via an implicit relative import instead of as part of a package (if it's part of a package).
Conceptually, though, what you're doing is somewhat dubious. It's risky to set A's module to be some other module, because that module might not be known (i.e., imported) when the real module is, and in fact it's possible the "disguise" module might not even exist.
I thought about this for a while and can't think of a better title, sorry.
I'm new'ish to Python, and (like many other's it seems) I just can't get my head around import.
I think I understand 'modules' and 'packages', classes and attributes and all that. It's one specific behavior I need clarified.
Say I have a file, foo.py. It has one line it:
x = 1
If, in another file, I `import foo", I can reference x. And, wonderfully, in another file I can import foo and now those two files can share x. Leaving classes out of the discussion for simplicity, I believe this is the pythonic way to share attributes between files.
Here's the question: Is is fair to say, when I import foo, that foo.py itself is, (for lack of a better metaphor), secretly instantiated by the interpreter?
I realize if I define a class in a module, it follow traditional rules and only become instantiated if I explicitly do so. But, the python interpreter (via the import statement) instantiating an instance of my module in the global namespace is the only way to explain the attribute sharing behavior.
Is this true? Semi-true? Or am I wandering with the Sleestaks in the Land of the Lost?
When you import a module:
if the module has not been previously imported, the file is parsed in to a module object which is added to sys.modules with a key that is the import path from the pythonpath to your module
that module object (or some member thereof) is aliased in the importing namespace, the alias and object being referenced being determined by the specific form of import you used
So when you import foo, the interpreter checks sys.modules for something registered with the name foo. If it finds it, it provides a label foo in the local namespace for the foo module. If it doesn't, it searches down the pythonpath until it finds a foo module, parses that to a module object, adds that object to sys.modules, and adds a label in the local namespace for that module object.
import foo as foof does the same thing, only the local namespace label created is foof. from foo import x follows the same process up to the point of creating a label and reference in the local namespace, instead providing a label x in the namespace for the attribute x from the foo module. from foo import x as foox just combines the 2 ideas.
With classes, you can actually poke around this whole system by crawling up and down the tree using the __module__ attribute.
The import creates an instance of a "module" object. It is worth knowing that this is created only the first time the module is imported. The following times it is imported you are getting a reference to the original. You can create your own module objects on the fly with a bit of instrospection.
import glob # Import any python module
moduleType = type(glob)
onTheFly = moduleType("OnTheFly", "Docstring for this module")
Although there isn't much benefit to creating these.
Yes, indeed its true. If you execute import foo a module object foo is instatiated and the contents of your file e.g a class bar is added as a member of that object.