We Keep Coding

Organizing methods in python classes with hierarchical names

Python officially recognizes namespaces as a "honking great idea" that we should "do more of". One nice thing about namespaces is their hierarchical presentation that organizes code into related parts. Is there an elegant way to organize python class methods into related parts, much as hierarchical namespaces are organized — especially for the purposes of tab-completion?
Some of my python classes cannot be split up into smaller classes, but have many methods attached to them (easily over a hundred). I also find (and my code's users tell me) that the easiest way to find useful methods is to use tab-completion. But with so many methods, this becomes unwieldy, as an enormous list of options is presented — and usually organized alphabetically, which means that closely related methods may be located in completely different parts of this massive list.
Typically, there are very distinct groups of closely related methods. For example, I have one class in which almost all of the methods fall into one of four groups:
io
statistics
transformations
symmetries
And the io group might have read and write subgroups, where there are different options for the file type to read or write, and then some additional methods involved in looking at the metadata for example. To a small extent, I can address this problem using underscores in my method names. For example, I might have methods like
myobject.io_read_from_csv
myobject.io_write_to_csv
This helps with the classification, but is ugly and still leads to unwieldy tab-completion lists. I would prefer it if the first tab-completion list just had the four options listed above, then when one of those options is selected, additional options would be presented with the next tab.
For a slightly more concrete example, here's a partial list of the hierarchy that I have in mind for my class:
myobject.io
myobject.io.read
myobject.io.read.csv
myobject.io.read.h5
myobject.io.read.npy
myobject.io.write
myobject.io.write.csv
myobject.io.write.h5
myobject.io.write.npy
myobject.io.parameters
myobject.io.parameters.from_csv_header
myobject.io.parameters.from_h5_attributes
...
...
myobject.statistics
myobject.statistics.max
myobject.statistics.max_time
myobject.statistics.norm
...
myobject.transformations
myobject.transformations.rotation
myobject.transformations.boost
myobject.transformations.spatial_translation
myobject.transformations.time_translation
myobject.transformations.supertranslation
...
myobject.symmetries
myobject.symmetries.parity
myobject.symmetries.parity.conjugate
myobject.symmetries.parity.symmetric_part
myobject.symmetries.parity.antisymmetric_part
myobject.symmetries.parity.violation
myobject.symmetries.parity.violation_normalized
myobject.symmetries.xreflection
myobject.symmetries.xreflection.conjugate
myobject.symmetries.xreflection.symmetric_part
...
...
...
One way I can imagine solving this problem is to create classes like IO, Statistics, etc., within my main MyClass class whose sole purpose is to store a reference to myobject and provide the methods that it needs. The main class would then have #property methods that just return the instances of those lower-lever classes, for which tab-completion should then work. Does this make sense? Would it work at all to provide tab-completion in ipython, for example? Would this lead to circular-reference problems? Is there a better way?

It looks like my naive suggestion of defining classes within the class does indeed work with ipython's tab-completion and without any circularity problems.
Here's the proof-of-concept code:
class A(object):
class _B(object):
def __init__(self, a):
self._owner = a
def calculate(self, y):
return y * self._owner.x
def __init__(self, x):
self.x = x
self._b = _B(self)
#property
def b(self):
return self._b
(In fact, it would be even simpler if I used self.b = _B(self), and I could skip the property, but I like this because it impedes overwriting b from outside the class. Plus this shows that this more complicated case still works.)
So if I create a = A(1.2), for example, I can hit a.<TAB> and get b as the completion, then a.b.<TAB> suggests calculate as the completion. I haven't run into any problems with this structure in my brief tests so far, and the changes to my code aren't very big — just adding ._owner into a lot of the methods code.

How does __test__ = False magic attribute work for test discovery

So I'm attempting to implement something similar to how unittesting frameworks do the following thing:
class BaseTest(T.TestCase):
# Disables this test from being run
__test__ = False
def test_foo(self): pass
# However this test is picked up because it doesn't directly have __test__ set
class InheritingTest(BaseTest): pass
A thing I find peculiar:
# >> InheritingTest.__test__
# False
Which would indicate to me that it isn't using a metaclass to set __test__ to True on construction of the type.
I tried grepping through the python library find . -name "*.py" | xargs grep '__test__' but did not seem to find anything related to this.
My "guess" approach at solving this problem is to do the following:
def is_class_tested(cls):
return cls.__dict__.get('__test__', True)
However this feels fragile to me... Is there a cleaner / nicer way to do this that works in all cases? Is there ever a chance that a class will not have a __dict__ property?

Testify, the library you are using, does the following in testify/test_discovery.py around line 140:
# it's not a list, it's not a bare module - let's see if it's an honest-to-god TestCaseBase
elif isinstance(test_module, MetaTestCase) and (not '__test__' in test_module.__dict__ or bool(test_module.__test__)):
...
# detect unittest test cases
elif issubclass(test_module, unittest.TestCase) and (not '__test__' in test_module.__dict__ or bool(test_module.__test__)):
So, in other words, it is doing exactly what your "guess" approach does, in a slightly more verbose way: it tests for the presence and value of __test__ in the class __dict__ directly.

Is there ever a chance that a class will not have a __dict__ property? My initial reaction to that is no. That might not be exactly what you were asking because you can have an instance of a class without a __dict__ if it defines __slots__ and does not inherit from a class with a __dict__ already in it. See the __slots__ description in the Python Data Model description
EDIT:
as pointed out by #nneonneo, comments below about the double underscore are not correct becuase there are trailing underscores. The content is left in for historical reasons.
The behavior you claim looks peculiar, I think looks natural. #Owen was right in asking about the code that that expects something different and thankyou for posting the reference to Yelp/Testify. That framework makes extensive use of the double_underscore.
Alex Martelli's SO answer on the double_underscore helps shed light on what might be going on but the very shortest answer is that the double underscore is returning the results above because the dot notation of InheritingTest.__test__ goes through the normal attribute resolution machinery but the Testify framework, by deciding to use the leading double_underscore, has reserved the right to access it at it's class scope even if your subclass overrides it.
Test frameworks are, by their very nature, magical beasts and the use of __test__ hear looks proper on their part. I checked out their docs and the documentation feels sparse so your using __test__ at all feels dangerous (the double underscore kind of signals hands off this "class local" variable anyway).

Have well-defined, narrowly-focused classes ... now how do I get anything done in my program?

I'm coding a poker hand evaluator as my first programming project. I've made it through three classes, each of which accomplishes its narrowly-defined task very well:
HandRange = a string-like object (e.g. "AA"). getHands() returns a list of tuples for each specific hand within the string:
[(Ad,Ac),(Ad,Ah),(Ad,As),(Ac,Ah),(Ac,As),(Ah,As)]
Translation = a dictionary that maps the return list from getHands to values that are useful for a given evaluator (yes, this can probably be refactored into another class).
{'As':52, 'Ad':51, ...}
Evaluator = takes a list from HandRange (as translated by Translator), enumerates all possible hand matchups and provides win % for each.
My question: what should my "domain" class for using all these classes look like, given that I may want to connect to it via either a shell UI or a GUI? Right now, it looks like an assembly line process:
user_input = HandRange()
x = Translation.translateList(user_input)
y = Evaluator.getEquities(x)
This smells funny in that it feels like it's procedural when I ought to be using OO.
In a more general way: if I've spent so much time ensuring that my classes are well defined, narrowly focused, orthogonal, whatever ... how do I actually manage work flow in my program when I need to use all of them in a row?
Thanks,
Mike

Don't make a fetish of object orientation -- Python supports multiple paradigms, after all! Think of your user-defined types, AKA classes, as building blocks that gradually give you a "language" that's closer to your domain rather than to general purpose language / library primitives.
At some point you'll want to code "verbs" (actions) that use your building blocks to perform something (under command from whatever interface you'll supply -- command line, RPC, web, GUI, ...) -- and those may be module-level functions as well as methods within some encompassing class. You'll surely want a class if you need multiple instances, and most likely also if the actions involve updating "state" (instance variables of a class being much nicer than globals) or if inheritance and/or polomorphism come into play; but, there is no a priori reason to prefer classes to functions otherwise.
If you find yourself writing static methods, yearning for a singleton (or Borg) design pattern, writing a class with no state (just methods) -- these are all "code smells" that should prompt you to check whether you really need a class for that subset of your code, or rather whether you may be overcomplicating things and should use a module with functions for that part of your code. (Sometimes after due consideration you'll unearth some different reason for preferring a class, and that's allright too, but the point is, don't just pick a class over a module w/functions "by reflex", without critically thinking about it!).

You could create a Poker class that ties these all together and intialize all of that stuff in the __init__() method:
class Poker(object):
def __init__(self, user_input=HandRange()):
self.user_input = user_input
self.translation = Translation.translateList(user_input)
self.evaluator = Evaluator.getEquities(x)
# and so on...
p = Poker()
# etc, etc...

Is it a good idea to using class as a namespace in Python

I am putting a bunch of related stuff into a class. The main purpose is to organize them into a namespace.
class Direction:
north = 0
east = 1
south = 2
west = 3
#staticmethod
def turn_right(d):
return turn_to_the_right
#staticmethod
def turn_left(d):
return turn_to_the_left
# defined a short alias because direction will be used a lot
D = Direction
d0 = D.north
d1 = D.turn_right(d)
There is not much object concept involved. In C++, I will be using the actual language keyword namespace. There is no such thing in Python. So I am trying to use class for this purpose.
Is this a good idea? Any pitfall with this approach?
I've just answer a related question yesterday. This question is asked in a different way. It is an actual decision I need to make for myself.
Static method vs module function in python - Stack Overflow
Static method vs module function in python

Yes, indeed. You can use Python classes strictly for namespacing as that is one of the special things they can do and do differently than modules. It's a lot easier to define a class as a namespace inline in a file than to generate more files.
You should not do it without commenting your code saying what it's for.
Python classes come in a lot of different forms and purposes and this makes difficulty understanding code you have not seen before.
A Python class used as a namespace is no less a Python class than one that meets the perception of what a class is in other languages. Python does not require a class to be instantiated to be useful. It does not require ivars and does not require methods. It is fairly flexible.
Clases can contain other classes too.
Lots of people have their ideas about what is or isn't Pythonic.
But if they were all worried about something like consistency, they'd push to have things like len() dir() and help() be a method of objects rather than a global function.
Do what works, comment / document it if it isn't usual or obvious usage.

No. Stick it in a module instead.
Python doesn't have namespaces in the same way that C++ does, but modules serve a somewhat similar purpose (that is, grouping "like" classes and functions together, and giving them unique names to avoid clashes).
Edit
I saw the comment you posted to your question. To answer more explicitly, no, in Pythonic code it's not really correct to use a class to emulate a namespace. Modules are there to group related classes, functions, and variables -- use a module instead. A class represents a "thing" that has a behavior (methods) and data (instance variables) -- it's not just a collection of standalone functions and variables.

Yes, it's fine. You can even use property to make methods look like attributes.
If you have a big class, it might be neater to use a module

It depends on the situation; if you can stick a constant in the module and have it make sense, by all means do so, but putting them in the class can make their meaning more obvious, and allow similar constants to have more "abstraction": placing them in the ServerError class makes more sense than having them all prepended with SERVER_ERROR residing freely in the module.
Do what is most intuitive, but try to avoid namespace pollution.

I mostly agree with #uchuga's answer, but I want to emphasize a caveat:
a = "global"
class C:
a = "class"
def f():
print(a)
f()
... will print "global", not "class".

In my opinion, a class is a class, and a Namespace is a namespace. You can use argparse.Namespace like so to create a namespace:
from argparse import Namespace
directions = Namespace(
north = 0,
east = 1,
south = 2,
west = 3,
)
print(directions.north) # 0
print(directions.east) # 1
print(directions.south) # 2
print(directions.west) # 3

Should I use a class in this: Reading a XML file using lxml

This question is in continuation to my previous question, in which I asked about passing around an ElementTree.
I need to read the XML files only and to solve this, I decided to create a global ElementTree and then parse it wherever required.
My question is:
Is this an acceptable practice? I heard global variables are bad. If I don't make it global, I was suggested to make a class. But do I really need to create a class? What benefits would I have from that approach. Note that I would be handling only one ElementTree instance per run, the operations are read-only. If I don't use a class, how and where do I declare that ElementTree so that it available globally? (Note that I would be importing this module)
Please answer this question in the respect that I am a beginner to development, and at this stage I can't figure out whether to use a class or just go with the functional style programming approach.

There are a few reasons that global variables are bad. First, it gets you in the habit of declaring global variables which is not good practice, though in some cases globals make sense -- PI, for instance. Globals also create problems when you on purpose or accidentally re-use the name locally. Or worse, when you think you're using the name locally but in reality you're assigning a new value to the global variable. This particular problem is language dependent, and python handles it differently in different cases.
class A:
def __init__(self):
self.name = 'hi'
x = 3
a = A()
def foo():
a.name = 'Bedevere'
x = 9
foo()
print x, a.name #outputs 3 Bedevere
The benefit of creating a class and passing your class around is you will get a defined, constant behavior, especially since you should be calling class methods, which operate on the class itself.
class Knights:
def __init__(self, name='Bedevere'):
self.name = name
def knight(self):
self.name = 'Sir ' + self.name
def speak(self):
print self.name + ":", "Run away!"
class FerociousRabbit:
def __init__(self):
self.death = "awaits you with sharp pointy teeth!"
def speak(self):
print "Squeeeeeeee!"
def cave(thing):
thing.speak()
if isinstance(thing, Knights):
thing.knight()
def scene():
k = Knights()
k2 = Knights('Launcelot')
b = FerociousRabbit()
for i in (b, k, k2):
cave(i)
This example illustrates a few good principles. First, the strength of python when calling functions - FerociousRabbit and Knights are two different classes but they have the same function speak(). In other languages, in order to do something like this, they would at least have to have the same base class. The reason you would want to do this is it allows you to write a function (cave) that can operate on any class that has a 'speak()' method. You could create any other method and pass it to the cave function:
class Tim:
def speak(self):
print "Death awaits you with sharp pointy teeth!"
So in your case, when dealing with an elementTree, say sometime down the road you need to also start parsing an apache log. Well if you're doing purely functional program you're basically hosed. You can modify and extend your current program, but if you wrote your functions well, you could just add a new class to the mix and (technically) everything will be peachy keen.

Pragmatically, is your code expected to grow? Even though people herald OOP as the right way, I found that sometimes it's better to weigh cost:benefit(s) whenever you refactor a piece of code. If you are looking to grow this, then OOP is a better option in that you can extend and customise any future use case, while saving yourself from unnecessary time wasted in code maintenance. Otherwise, if it ain't broken, don't fix it, IMHO.

I generally find myself regretting it when I give in to the temptation to give a module, for example, a load_file() method that sets a global that the module's other functions can then use to find the file they're supposed to be talking about. It makes testing far more difficult, for example, and as soon as I need two XML files there is a problem. Plus, every single function needs to check whether the file's there and give an error if it's not.
If I want to be functional, I simply therefore have every function take the XML file as an argument.
If I want to be object oriented, I'll have a MyXMLFile class whose methods can just look at self.xmlfile or whatever.
The two approaches are more or less equivalent when there's just one single thing, like a file, to be passed around; but when the number of things in the "state" becomes larger than a few, then I find classes simpler because I can stick all of those things in the class.
(Am I answering your question? I'm still a big vague on what kind of answer you want.)