Python3
Tried to found an answer but failed. First I'll present the snippet, then I'll explain why I wanted to do it this way and what I wanted to achieve. Maybe it'll look like this approach is "the bad one".
Hence this semi-double topic, cause first I'd like to know why this snippet isn't working and second - I'd like to know if this approach is right.
So:
class Namespace:
def some_function():
pass
class SomeClass:
fcnt = some_function
This won't work due to:
NameError: name 'some_function' is not defined
What I want to achieve is code and file structure readability.
Above example is a snippet which I use (not this one, but it looks like this) in Pyramid project.
My project tree looks like this:
my_project
├── models
│ ├── __init__.py
│ └── some_model.py
├── schemas
│ ├── __init__.py
│ ├── some_schema.py
│ └── some_other_schema.py
...
├── views
│ ├── __init__.py
│ └── some_view.py
└── __init__.py
What I wanted to achieve is clean schema/model/view importing.
In some_schema.py file resides class SomeSchema, in some_other_schema.py class SomeOtherSchema.
With above snippet I can make:
from my_project.schemas.some_schema import Schema
and use it like Schema.SomeSchema()
I've got a little bit lost with packages and imports. How one could make a clean structure (one schema per file) and still be able to use Schema namespace? (In C++ I'd just put each of those classes in Schema namespace, that's why I did this in snippet above. But! What works in C++ maybe shouldn't be used in python, right?).
Thanks for answer in advance.
EDIT:
Ok, I've done some testing (I thought that I've done it, but looks like not..).
using from my_project.schemas.some_schema import Schema with another from my_project.schemas.some_other_schema import Schema causes in the second import shadowing first one. So if after first import I'd be able to use x = Schema.SomeSchema() than after second import I'd be unable to do this, because class Schema gets overriden. Right, so as Erik said - classes aren't namespaces. GOT IT!
in my very first snippet yes, I should've used fnct = Namespace.some_function. What's wierd - it works. I have the same statement in my pyramid code, with one difference. some_function has #colander.deferred decorator. In fact it looks like this:
class Schema:
#colander.deferred
def deferred_some_function(node, kw):
something = kw.get("something", [])
return deform.widget.SelectWidget(values=something,
multiple=True)
class SomeSchema(colander.MappingSchema):
somethings = colander.SchemaNode(colander.Set(),
widget=Schema.deferred_some_function)
And I get NameError: name 'Schema' is not defined
Getting back to package format. With this:
### another/file.py
from foo.bar.schema import SomeSchema
# do something with SomeSchema:
smth = SomeSchema()
smth.fcnt()
I have to make one module foo/bar/schema.py in which I'd have to put all my SomeXSchema classes. An if I have lots of them, then there's the unreadabilty glitch which I wanted to get rid off by splitting SomeXSchema - one per file. Can I accomplish this somehow? I want to call this class for example: User. And here's the THING. Maybe I do it wrong? I'd like to have class named User in schema namespace and class named User in model namespace. Shouldn't I? Maybe I ought to use prefix? Like class SchemaUser and class ModelUser ? I wanted to avoid it by the use of modules/packages.
If I'd use : import foo.bar.schema then I'd have to use it like x = foo.bar.schema.User() right? There is no way to use it like x = schema.User() ? Sorry, I just got stuck, my brain got fixed. Maybe I need a little break to take a fresh look?
ANOTHER EDIT (FOR POINT 3 ONLY)
I did some more research. The answer here would be to make it like this:
## file: myproject/schemas/__init__.py
from .some_schema import SomeSchema
from .some_other_schema import SomeOtherSchema
then usage would be like this:
## some file using it
import myproject.schemas as schema
s1 = schema.SomeSchema()
s2 = schema.SomeOtherSchema()
Would it be lege artis?
If anyone thinks that topic should be changed - go ahead, give me something more meaningful, I'd appreciate it.
Your are swimming upstream by trying to do what you are trying to do.
Classes are meant for defining new data types not as a means to group related parts of code together. Modules are perfectly suited for that, and I presume you know that well because of the "(vs proper package structure)" part in the question title.
Modules can also be imported as objects, so to achieve what you want:
### foo/bar/schema.py
def some_function():
pass
class SomeSchema:
fcnt = some_function
### another/file.py
from foo.bar import schema
# do something with SomeSchema:
smth = schema.SomeSchema()
smth.fcnt()
...although it's also typical to import classes directly into the scope like this (i.e. being able to refer to SomeSchema after the import as opposed to schema.SomeSchema):
### another/file.py
from foo.bar.schema import SomeSchema
# do something with SomeSchema:
smth = SomeSchema()
smth.fcnt()
(Also note that module names should be lowercase as suggested by PEP8 and only class names should use PascalCase)
This, by the way, applies to programming in general, not just Python. There are a few languages such as Java and C# which require that functions be declared inside of classes as statics because they disallow writing of code outside of classes for some weird reason, but even these languages have modules/proper namespaces for structuring your code; i.e. classes are not normally put inside other classes (they sometimes are, but for wholly different reasons/goals than yours).
So basically "class" means a "type" or "a set of objects having similar behavior"; once you ignore that principle/definition, you're writing bad code by definition.
PS. if you are using Python 2.x, you should be inheriting your classes from object so as to get new-style classes.
PPS. in any case, even technically speaking, what you are trying to do won't work cleanly in Python:
class fake_namespace:
def some_function():
pass
class RealClass:
some_function # <-- that name is not even visibile here;
# you'd have to use fake_namespace.some_function instead
...and this is the reason for the exception I reported I was getting: NameError: name 'some_function' is not defined.
EDIT AS PER YOUR EDITS:
I'm not really sure why you're making it so complicated; also some of your statements are false:
If I'd use : import foo.bar.schema then I'd have to use it like x = foo.bar.schema.User right?
No. Please learn how Python modules work.
I'd like to have class named User in Schema namespace and class named User in Model namespace. Shouldn't I? Maybe I ought to use prefix? Like class SchemaUser and class ModelUser
please note that namespaces a.k.a. modules should be lowercase not PascalCase.
An if I have lots of them, then there's the unreadabilty glitch which I wanted to get rid off by splitting SomeXSchema - one per file. Can I accomplish this somehow?
Yes; you can put your classes in individual submodules, e.g. schema1/class1.py, schema/class2.py etc; then you can "collect" them into schema/__init__.py so that you could import them directly from schema:
# schema/__init__.py
from .class1 import Class1
from .class2 import Class2
__all__ = [Class1, Class2] # optional
General note: you can name your schema modules differently, e.g. schema1, schema2, etc; then you could just use them like this:
from somewhere import schema1
from somewhere_else import schema2
s1_user = schema1.User()
s2_user = schema2.User()
# etc
For more information on how Python modules work, refer to http://docs.python.org/2/tutorial/modules.html
Name and binding
You can read Python naming and binding and understand how Python namespace works.
A scope defines the visibility of a name within a block. If a local variable is defined in a block, its scope includes that block. If the definition occurs in a function block, the scope extends to any blocks contained within the defining one, unless a contained block introduces a different binding for the name. The scope of names defined in a class block is limited to the class block; it does not extend to the code blocks of methods this includes generator expressions since they are implemented using a function scope.
BTW, use globals() and locals() can help debug for variable binding.
The User Problem
You can try this instead:
from model import User as modelUser
from foo.bar.schema import User as schemaUser
Related
Python Double-Underscore methods are hiding everywhere and behind everything in Python! I am curious about how this is specifically working with the interpreter.
import some_module as sm
From my current understanding:
Import searches for requested module
It binds result to the local assignment (if given)
It utilizes the __init__.py . . . ???
There seems to be something going on that is larger than my scope of understanding. I understand we use __init__() for class initialization. It is functioning as a constructor for our class.
I do not understand how calling import is then utilizing the __init__.py.
What exactly is happening when we run import?
How is __init__.py different from other dunder methods?
Can we manipulate this dunder method (if we really wanted to?)
import some_module is going to look for one of two things. It's either going to look for a some_module.py in the search path or a some_module/__init__.py. Only one of those should exist. The only thing __init__.py means when it comes to modules is "this is the module that represents this folder". So consider this folder structure.
foo/
__init__.py
module1.py
bar.py
Then the three modules available are foo (which corresponds to foo/__init__.py), foo.module1 (which corresponds to foo/module1.py), and bar (which corresponds to bar.py). By convention, foo/__init__.py will usually import important names from module1.py and reexport some of them for convenience, but this is by no means a requirement.
I've got a really complex singleton object. I've decided to modify it, so it'll be a separate module with module--wide global variables that would store data.
Are there some pitfalls of this approach? I just feel, like that's a little bit hacky, and that there may be some problems I cannot see now.
Maybe someone did this or have some opinion :) Thanks in advance for help.
Regards.
// Minimal, Complete, and Verifiable example:
"""
This is __init__.py of the module, that could be used as a singleton:
I need to set and get value of IMPORTANT_VARIABLE from different places in my code.
Folder structure:
--singleton_module
|
-__init__.py
Example of usage:
import singleton_module as my_singleton
my_singleton.set_important_variable(3)
print(my_singleton.get_important_variable())
"""
IMPORTANT_VARIABLE = 0
def set_important_variable(value):
global IMPORTANT_VARIABLE
IMPORTANT_VARIABLE = value
def get_important_variable():
return IMPORTANT_VARIABLE
Technically, Python modules ARE singletons, so from this point of view there's no particular issue (except the usual issues with singletons that is) with your code. I'd just spell the varibale in all_lower (ALL_UPPER denotes a pseudo-constant) and prefix it with either a single ("protected") or double ("really private") leading underscore to make clear it's not part of the public API (standard Python naming convention).
Now whether singletons are a good idea is another debate but that's not the point here...
e.g that in one potential situation I may lost data, or that module could be imported in different places of code two times, so it would not be a singleton if imported inside scope of function or something like that.
A module is only instanciated once per process (the first time it's imported), then subsquent imports will directly get if from sys.modules. The only case where you could have two distinct instances of the same module is when the module is imported by two different path, which can only happens if you have a somewhat broken sys.path ie something like this:
src/
foo/
__init.py
bar/
__init__.py
baaz/
__init__.py
mymodule.py
with both "src" and "foo" in sys.path, then importing mymodule once as from foo.bar.baaz import mymodule and a second time as from bar.baaz import mymodule
Needless to say that it's a degenerate case, but it can happens and lead to hard to diagnose bugs. Note that when you have this case, you do have quite a few other things that breaks, like identity testing anything from mymodule.
Also, I am not sure how would using object instead of module increase security
It doesn't.
And I am just asking, if that's not a bad practice, maybe someone did this and found some problems. This is probably not a popular pattern
Well, quite on the contrary you'll often find advises on using modules as singletons instead of using classes with only staticmethods, classmethods and class attributes (another way of implementing a singleton in Python). This most often concerns stateless classes used as namespaces while your example does have a state, but this doesn't make much practical difference.
Now what you won't get are all the nice OO features like computed attributes, inheritance, magicmethods etc, but I assume you already understood this.
As far as I'm concerned, depending on the context, I might rather use a plain class but only expose one single instance of the class as the module's API ie:
# mymodule.py
__all__ = ["mysingleton"]
class __MySingletonLike(object):
def __init__(self):
self._variable = 42
#property
def variable(self):
return self._variable
#variable.setter
def variable(self, value):
check_value(value) # imaginary validation
self._variable = value
mysingleton = __MySingleton()
but that's only when I have special concerns about the class (implementation reuse, proper testability, other special features requiring a class etc).
So I have a small project of my own in Python which I want to regularly use. It is, at the moment, composed of two classes, let's call them class Aclass and Bclass.
The amount of code in both classes combined is large. So large that I do not want to have to have them in one file, so I create one file for each. I put the first into the file aclass.py, the second into bclass.py so my structure now looks something like this:
my_project/
aclass.py
bclass.py
__init__.py
What I want is to be able to call my project from python and have the classes ready, like so:
import my_project
a_instance = my_project.Aclass()
b_instance = my_project.Bclass()
At the moment, what I have to do is
import my_project
a_instance = my_project.aclass.Aclass()
Which, if you ask me, is redundant. In the question:
Avoid double typing class names in python
I saw an advice to put
from aclass import Aclass
from bclass import Bclass
into the __init__.py file, but this causes confusion, because if I do that, I can create an instance of my class either by writing
import my_project
a_instance = my_project.Aclass()
or by writing
import my_project
a_instance = my_project.aclass.Aclass()
I find this solution ugly, because not only does it "double" the functions (creating both aclass.Aclass() and Aclass(), but it also means that any instance of the class Aclass will now be, in Python's eyes,
<class my_project.aclass.Aclass at ******************>
instead of
<class my_project.Aclass at ******************>
which it is in the natural project structure.
Thus, my question is this:
Is there a solution to my problem in which Aclass will be located directly inside the my_project package without writing the class into the __init__ file?
No; you only have two choices, either:
Put everything in one file, and have only a single route to each object at the cost of an unwieldy codebase; or
Have multiple files and use the __init__.py to define what's available at the top level of your library, accepting that this will mean multiple routes to an object.
For any project beyond a handful of classes and functions, the latter is clearly preferable; it is widely used in the standard library and beyond. For example, with the json library, where the __init__.py includes from .decoder import JSONDecoder:
>>> import json
>>> json.JSONDecoder
<class 'json.decoder.JSONDecoder'>
>>> json.JSONDecoder is json.decoder.JSONDecoder
True
In the overwhelming majority of cases, any user of your code will use the shorter name rather than the longer name but, as they point to the same object, it doesn't really matter if they mix and match, and you will rarely have any occasion to see what "Python thinks" the fully-qualifed class name is.
I'm having some real headaches right now trying to figure out how to import stuff properly. I had my application structured like so:
main.py
util_functions.py
widgets/
- __init__.py
- chooser.py
- controller.py
I would always run my applications from the root directory, so most of my imports would be something like this
from util_functions import *
from widgets.chooser import *
from widgets.controller import *
# ...
And my widgets/__init__.py was setup like this:
from widgets.chooser import Chooser
from widgets.controller import MainPanel, Switch, Lever
__all__ = [
'Chooser', 'MainPanel', 'Switch', 'Lever',
]
It was working all fine, except that widgets/controller.py was getting kind of lengthy, and I wanted it to split it up into multiple files:
main.py
util_functions.py
widgets/
- __init__.py
- chooser.py
- controller/
- __init__.py
- mainpanel.py
- switch.py
- lever.py
One of issues is that the Switch and Lever classes have static members where each class needs to access the other one. Using imports with the from ___ import ___ syntax that created circular imports. So when I tried to run my re-factored application, everything broke at the imports.
My question is this: How can I fix my imports so I can have this nice project structure? I cannot remove the static dependencies of Switch and Lever on each other.
This is covered in the official Python FAQ under How can I have modules that mutually import each other.
As the FAQ makes clear, there's no silvery bullet that magically fixes the problem. The options described in the FAQ (with a little more detail than is in the FAQ) are:
Never put anything at the top level except classes, functions, and variables initialized with constants or builtins, never from spam import anything, and then the circular import problems usually don't arise. Clean and simple, but there are cases where you can't follow those rules.
Refactor the modules to move the imports into the middle of the module, where each module defines the things that need to be exported before importing the other module. This can means splitting classes into two parts, an "interface" class that can go above the line, and an "implementation" subclass that goes below the line.
Refactor the modules in a similar way, but move the "export" code (with the "interface" classes) into a separate module, instead of moving them above the imports. Then each implementation module can import all of the interface modules. This has the same effect as the previous one, with the advantage that your code is idiomatic, and more readable by both humans and automated tools that expect imports at the top of a module, but the disadvantage that you have more modules.
As the FAQ notes, "These solutions are not mutually exclusive." In particular, you can try to move as much top-level code as possible into function bodies, replace as many from spam import … statements with import spam as is reasonable… and then, if you still have circular dependencies, resolve them by refactoring into import-free export code above the line or in a separate module.
With the generalities out of the way, let's look at your specific problem.
Your switch.Switch and lever.Lever classes have "static members where each class needs to access the other one". I assume by this you mean they have class attributes that are initialized using class attributes or class or static methods from the other class?
Following the first solution, you could change things so that these values are initialized after import time. Let's assume your code looked like this:
class Lever:
switch_stuff = Switch.do_stuff()
# ...
You could change that to:
class Lever:
#classmethod
def init_class(cls):
cls.switch_stuff = Switch.do_stuff()
Now, in the __init__.py, right after this:
from lever import Lever
from switch import Switch
… you add:
Lever.init_class()
Switch.init_class()
That's the trick: you're resolving the ambiguous initialization order by making the initialization explicit, and picking an explicit order.
Alternatively, following the second or third solution, you could split Lever up into Lever and LeverImpl. Then you do this (whether as separate lever.py and leverimpl.py files, or as one file with the imports in the middle):
class Lever:
#classmethod
def get_switch_stuff(cls):
return cls.switch_stuff
from switch import Swift
class LeverImpl(Lever):
switch_stuff = Switch.do_stuff()
Now you don't need any kind of init_class method. Of course you do need to change the attribute to a method—but if you don't like that, with a bit of work, you can always change it into a "class #property" (either by writing a custom descriptor, or by using #property in a metaclass).
Note that you don't actually need to fix both classes to resolve the circularity, just one. In theory, it's cleaner to fix both, but in practice, if the fixes are ugly, it may be better to just fix the one that's less ugly to fix and leave the dependency in the opposite direction alone.
mod1.py
import mod2
class Universe:
def __init__(self):
pass
def answer(self):
return 42
u = Universe()
mod2.show_answer(u)
mod2.py
#import mod1 -- not necessary
def show_answer(thing):
print thing.answer()
Coming from a C++ background I had the feeling it was necessary to import the module containing the Universe class definition before the show_answer function would work. I.e. everything had to be declared before it could be used.
Am I right in thinking this isn't necessary? This is duck typing, right? So if an import isn't required to see the methods of a class, I'd at least need it for the class definition itself and the top level functions of a module?
In one script I've written, I even went as far as writing a base class to declare an interface with a set of methods, and then deriving concrete classes to inherit that interface, but I think I get it now - that's just wrong in Python, and whether an object has a particular method is checked at runtime at the point where the call is made?
I realise Python is so much more dynamic than C++, it's taken me a while to see how little code you actually need to write!
I think I know the answer to this question, but I just wanted to get clarification and make sure I was on the right track.
UPDATE: Thanks for all the answers, I think I should clarify my question now:
Does mod2.show_answer() need an import (of any description) to know that thing has a method called answer(), or is that determined dynamically at runtime?
In this case you're right: show_answer() is given an object, of which it calls the method "answer". As long as the object given to show_answer() has such a method, it doesn't matter where the object comes from.
If, however, you wanted to create an instance of Universe inside mod2, you'd have to import mod1, because Universe is not in the mod2 namespace, even after mod2 has been imported by mod1.
import is all about names -- mostly "bare names" that are bound at top level (AKA global level, AKA module-level names) in a certain module, say mod2. When you've done import mod2, you get the mod2 namespace as an available name (top-level in your own module, if you're doing the import itself as top level, as is most common; but a local import within a function would make mod2 a local variable of that function, etc); and therefore you can use mod2.foobar to access the name foobar that's bound at top level in mod2. If you have no need to access such names, then you have no need to import mod2 in your own module.
Think of import being more like the linker.
With "import mod2" you are simply telling python that it can find the function in the file mod2.py
Actually, in this case, importing mod1 in mod2.py should not work.
Would it not create a circular reference?
In fact, according to this explanation , the circular import will not work the way you want it to work: if you uncomment import mod1, the second module will still not know about the Universe.
I think this is quite reasonable. If both of your files need access to the type of some specific object, like Universe, you have several choices:
if your program is small, just use one file
if it's big, you need to decide if your files both need to know how Universe is implemented, perhaps passing an object of not-yet-known type to show_answer is fine
if that doesn't work for you, by all means put Universe in a separate module and load it first.
import in Python loads the module into the given namespace. As such, is it as if the def show_answer actually existed in the mod1.py module. Because of this, mod2.py does not need to know of the Universe class and thus you do not need to import mod1 from mod2.py.
I don't know much about C++, so can't directly compare it, but..
import basically loads the other Python script (mod2.py) into the current script (the top level of mod1.py). It's not so much a link, it's closer to an eval
For example, in Python'ish psuedo-code:
eval("mod2.py")
is the same as..
from mod2 import *
..it executes mod2.py, and makes the functions/classes defined accessible in the current script.
Both above snippets would allow you to call show_answer() (well, eval doesn't quite work like that, thus I called it pseudo code!)
import mod2
..is basically the same, but instead of bringing in all the functions into the "top level", it brings them into the mod2 module, so you call show_answer by doing..
mod2.show_answer
Am I right in thinking [the import in mod2.py] isn't necessary?
Absolutely. In fact if you try and import mod1 from mod2 you get a circular dependancy error (since mod2 then tries to import mod1 and so on..)