So, I am implementing a data tree in python that represents mathematic expressions much like this binary expression tree.
Each node represents an operation (+, *, exp(), ...) and each leaf represents a number or variable. Therefore I created a module Expression.py, that contains a parent class Node and child classes for mathematical operations.
Now, as the project becomes more complex, I am starting to implement more and more node types to cover more operations and each child class of Node is starting to have quite a lot methods for tasks as term simplification etc.
So far, I implemented all of these child nodes in the Expression.py file. But it is by now a 500+ lines file and I am not even done yet. I tried to split it up by putting each child class into a single file (Java style) and merging all of them in one package, which would match my understanding of correct structure. But this implementation is giving me problems as the different modules like Addition.py and Multiplication.py still reference each other. E.g. an Addition-object's method might return a Multiplication-object and vice versa.
My question is: How do you structure such a project? How to structure many related child classes that reference each other besides putting them in a single huge file?
If I arrange them in a package, how would I import them properly? And how would I reference them properly?
Edit:
Ok, let me be more specific, this is some sample code:
class Node():
def __init__(self):
pass
def derive(self):
pass
class Sine(Node):
def __init__(self, arg):
self.arg = arg
def derive(self):
return Cosine(self.arg)
class Cosine(Node):
def __init__(self, arg):
self.arg = arg
def derive(self):
return Multiplication(Num(-1), Sine(self.arg))
class Multiplication:
...
As you see the classes Sine and Cosine have a circular dependency, that I cannot (knowingly) split into two seperate files. Though I do not want to put thousands of lines of child classes into one file. This is only sample code. The classes actually consist of way more lines.
As you don't show any code, it is only possible to give a general answer. First of all, between putting everything in one file and each class in a single file, you have the option to put groups of classes in modules. I doubt that all your classes depend on each other. So you need to get a better understanding of the structure of your dependencies. If you understand the structure, Python will offer you the tools to express this structure in code. Mixins, decorators, meta classes, ... are powerful tools to express complex structures without violating DRY. So my advice would be: Try to understand your structure better!
Related
When composing functions from mutliple classes, I have seen both of the following conventions employed: Data1 uses mixins to add methods directly to the child class while Data2 instantiates attribute objects that do the work.
I think they are equivalent and that the choice is just one of style. Is that right? Or is one preferred over the other?
class Data1(Reader, Processor, Writer):
def __init__(self):
Reader.__init__(self)
Processor.__init__(self)
Writer.__init__(self)
def run(self):
self.read()
self.process()
self.write()
or
class Data2:
def __init__(self):
self.reader = Reader()
self.processor = Processor()
self.writer = Writer()
def run(self):
self.reader.read()
self.processor.process()
self.writer.write()
To flush out a specific example, I have a processing pipeline where there are various data products that each need to be read (Reader.read()), processed (Processor.process()), and then the product of the processing step needs to be written to a db (Writer.write()).
To be even more concrete, consider that I have multiple fitness data types:
heart rate data from a csv file that needs to averaged on
1-second intervals and dumped to a heart-rate table in a db
running speed data from a json file that needs to be converted to
mi/hr, then formatted as part of a html report.
weather data from
a web api that needs to be aggregated over day-long periods and then
posted to another web api.
For each of these "data products", there is a logical read, process, write pipeline and I'd like to capture that in an abstract class that can then be used as a consistent template for handling future "data products".
In these examples, Reader.read() would be an abstract class that might read a csv, json, or web API. Processor.process() performs various aggregations. Writer.write() would send the processed data to various places.
Given that, I am unsure of the best structure.
I would like to avoid a religious was because you could find technical reasons to use one or the other, but a rule of thumb is to ask the question is A a B or does A have a B? In former case you should use inheritance, in the latter composition.
For example, a coloured square is a figure and has a colour. So it should inherit from figure and contain a colour. There can be some hints: is one of the subobject has an independant lifecycle (it may exist before being used in the combining object) then it is with no doubt a has a relation. On the other side, if it cannot exist alone (abstract class) then is is with no doubt a **is a* relation.
But that means that without knowing what you call a Reader a Writer and a Processor and what data is, I cannot say which model I would use. But if Reader and Writer were both subclasses of the same ancestor class with independant parent members, I would use composition. If they were specially tailored classes sharing members of a common ancestor, that would be more of a is a operation and I would use inheritance.
The rule is that when possible, you should respect the real object semantics. After all in deep code execution it does not really matter whether you have used inheritance or composition.
BTW, what is discussed above is the general inheritance vs composition question. Strictly speaking, a mixin is a special case because a mixin should maintain no state but only add methods and because of that is often abstract. In Python mixins are implemented through inheritance, but other language may have other implementations. But in Python they are a typical example of something that is not necessarily a is a relation but does use inheritance.
Using superclasses is shorter (if you initialize the bases correctly):
class Data1(Reader, Processor, Writer):
def __init__(self):
super().__init__()
def run(self):
self.read()
self.process()
self.write()
but many people find the composition version to be easier to work with, since you don't need to go hunting through the inheritance tree to find where a method is implemented/overriden.
Wikipedia has a longer article on the topic which is well worth the read: https://en.wikipedia.org/wiki/Composition_over_inheritance
Addendum: your .run() method might be better implemented as
self.write(self.process(self.read()))
and then it would be easier to just make it a function:
def run(reader, processor, writer):
return writer.write(processor.process(reader.read()))
or with e.g. logging:
def run(reader, processor, writer):
for data in reader.read():
log.debug("Read data: %r", data)
for output_chunk in processor.process(data):
log.debug("processed %r and got %r", data, output_chunk)
writer.write(output_chunk)
log.debug("wrote %r", output_chunk)
and call it:
run(Reader(), Processor(), Writer())
assuming the reader is yielding data, this could be much more efficient, and it is very much easier to write unit tests for.
Finally: You do not need Reader as an abstract base class for csv, json, or web API reader classes. People coming from Java/C++ tend to conflate classes and types, and subclasses with subtypes. The Python type of the reader parameter in
def run(reader, processor, writer):
is ∀τ ≤ {read: NONE→DATA}, ie. all subtypes t of an object type with a .read(..) that takes NONE (the type of None) and returns a value of (here unspecified) type DATA. E.g. the standard file object has such a type and could be passed in directly, instead of writing a wrapper-class FileReader with tons of boilerplate. This, incidentally, is why I consider adding under-powered type languages to Python to be a very bad thing, but at this point I realize that I'm digressing ;-)
Python officially recognizes namespaces as a "honking great idea" that we should "do more of". One nice thing about namespaces is their hierarchical presentation that organizes code into related parts. Is there an elegant way to organize python class methods into related parts, much as hierarchical namespaces are organized — especially for the purposes of tab-completion?
Some of my python classes cannot be split up into smaller classes, but have many methods attached to them (easily over a hundred). I also find (and my code's users tell me) that the easiest way to find useful methods is to use tab-completion. But with so many methods, this becomes unwieldy, as an enormous list of options is presented — and usually organized alphabetically, which means that closely related methods may be located in completely different parts of this massive list.
Typically, there are very distinct groups of closely related methods. For example, I have one class in which almost all of the methods fall into one of four groups:
io
statistics
transformations
symmetries
And the io group might have read and write subgroups, where there are different options for the file type to read or write, and then some additional methods involved in looking at the metadata for example. To a small extent, I can address this problem using underscores in my method names. For example, I might have methods like
myobject.io_read_from_csv
myobject.io_write_to_csv
This helps with the classification, but is ugly and still leads to unwieldy tab-completion lists. I would prefer it if the first tab-completion list just had the four options listed above, then when one of those options is selected, additional options would be presented with the next tab.
For a slightly more concrete example, here's a partial list of the hierarchy that I have in mind for my class:
myobject.io
myobject.io.read
myobject.io.read.csv
myobject.io.read.h5
myobject.io.read.npy
myobject.io.write
myobject.io.write.csv
myobject.io.write.h5
myobject.io.write.npy
myobject.io.parameters
myobject.io.parameters.from_csv_header
myobject.io.parameters.from_h5_attributes
...
...
myobject.statistics
myobject.statistics.max
myobject.statistics.max_time
myobject.statistics.norm
...
myobject.transformations
myobject.transformations.rotation
myobject.transformations.boost
myobject.transformations.spatial_translation
myobject.transformations.time_translation
myobject.transformations.supertranslation
...
myobject.symmetries
myobject.symmetries.parity
myobject.symmetries.parity.conjugate
myobject.symmetries.parity.symmetric_part
myobject.symmetries.parity.antisymmetric_part
myobject.symmetries.parity.violation
myobject.symmetries.parity.violation_normalized
myobject.symmetries.xreflection
myobject.symmetries.xreflection.conjugate
myobject.symmetries.xreflection.symmetric_part
...
...
...
One way I can imagine solving this problem is to create classes like IO, Statistics, etc., within my main MyClass class whose sole purpose is to store a reference to myobject and provide the methods that it needs. The main class would then have #property methods that just return the instances of those lower-lever classes, for which tab-completion should then work. Does this make sense? Would it work at all to provide tab-completion in ipython, for example? Would this lead to circular-reference problems? Is there a better way?
It looks like my naive suggestion of defining classes within the class does indeed work with ipython's tab-completion and without any circularity problems.
Here's the proof-of-concept code:
class A(object):
class _B(object):
def __init__(self, a):
self._owner = a
def calculate(self, y):
return y * self._owner.x
def __init__(self, x):
self.x = x
self._b = _B(self)
#property
def b(self):
return self._b
(In fact, it would be even simpler if I used self.b = _B(self), and I could skip the property, but I like this because it impedes overwriting b from outside the class. Plus this shows that this more complicated case still works.)
So if I create a = A(1.2), for example, I can hit a.<TAB> and get b as the completion, then a.b.<TAB> suggests calculate as the completion. I haven't run into any problems with this structure in my brief tests so far, and the changes to my code aren't very big — just adding ._owner into a lot of the methods code.
I'm using ast python library and want to traverse through my ast nodes.
Visitor pattern is supported in the library pretty well, but if I use it, I will have to implement methods for visiting items of concrete classes, e.g. def visit_Load. In my case, concrete classes are not so important - I'd like to know whether the node is an operator or an expr according to the structure given here.
Of course, I can add generic_visit method and then check all the conditions here, but that looks like a wrong way to use the pattern.
Is there any other pretty way to implement this idea without massive code duplication?
This probably isn't pretty, but you can dynamically create all the methods by introspection:
def visit_expr(self, node):
"""Do something in here!"""
self.generic_visit(node)
ExprVisitor = type('ExprVisitor', (ast.NodeVisitor,), {
'visit_' % cls.__name__: visit_expr for cls in ast.expr.__subclasses__()})
Of course, you can keep doing this for whatever ast node types you need to deal with. . .
My code base is in Python. Let's say I have a fairly generic class called Report. It takes a large number of parameters
class Report(object):
def __init__(self, title, data_source, columns, format, ...many more...)
And there are many many instantiation of the Report. These instantiation are not entirely unrelated. Many reports share similar set of parameters, differ only with minor variation, like having the same data_source and columns but with a different title.
Rather than duplicating the parameters, some programming construct is applied to make expression this structure easier. And I'm trying to find some help to sort my head to identify some idiom or design pattern for this.
If a subcategory of report need some extra processing code, subclass seems to be a good choice. Say we have a subcategory of ExpenseReport.
class ExpenseReport(Report):
def __init__(self, title, ... a small number of parameters ...)
# some parameters are fixed, while others are specific to this instance
super(ExpenseReport,self).__init__(
title,
EXPENSE_DATA_SOURCE,
EXPENSE_COLUMNS,
EXPENSE_FORMAT,
... a small number of parameters...)
def processing(self):
... extra processing specific to ExpenseReport ...
But in a lot of cases, the subcategory merely fix some parameters without any extra processing. It could easily be done with partial function.
ExpenseReport = functools.partial(Report,
data_source = EXPENSE_DATA_SOURCE,
columns = EXPENSE_COLUMNS,
format = EXPENSE_FORMAT,
)
And in some case, there isn't even any difference. We simply need 2 copies of the same object to be used in different environment, like to be embedded in different page.
expense_report = Report("Total Expense", EXPENSE_DATA_SOURCE, ...)
page1.add(expense_report)
...
page2.add(clone(expense_report))
And in my code base, an ugly technique is used. Because we need 2 separate instances for each page, and because we don't want to duplicate the code with long list of parameter that creates report, we just clone (deepcopy in Python) the report for page 2. Not only is the need of cloning not apparent, neglecting to clone the object and instead sharing one instance creates a lot of hidden problem and subtle bugs in our system.
Is there any guidance in this situation? Subclass, partial function or other idiom? My desire is for this construct to be light and transparent. I'm slight wary of subclassing because it is likely to result in a jungle of subclass. And it induces programmer to add special processing code like what I have in ExpenseReport. If there is a need I rather analyze the code to see if it can be generalized and push to the Report layer. So that Report becomes more expressive without needing special processing in lower layers.
Additional Info
We do use keyword parameter. The problem is more in how to manage and organize the instantiation. We have a large number of instantiation with common patterns:
expense_report = Report("Expense", data_source=EXPENSE, ..other common pattern..)
expense_report_usd = Report("USD Expense", data_source=EXPENSE, format=USD, ..other common pattern..)
expense_report_euro = Report("Euro Expense", data_source=EXPENSE, format=EURO, ..other common pattern..)
...
lot more reports
...
page1.add(expense_report_usd)
page2.add(expense_report_usd) # oops, page1 and page2 shared the same instance?!
...
lots of pages
...
Why don't you just use keyword arguments and collect them all into a dict:
class Report(object):
def __init__(self, **params):
self.params = params
...
I see no reason why you shouldn't just use a partial function.
If your main problem is common arguments in class constructos, possible solution is to write something like:
common_arguments = dict(arg=value, another_arg=anoter_value, ...)
expense_report = Report("Expense", data_source=EXPENSE, **common_arguments)
args_for_shared_usd_instance = dict(title="USD Expense", data_source=EXPENSE, format=USD)
args_for_shared_usd_instance.update(common_arguments)
expense_report_usd = Report(**args_for_shared_usd_instance)
page1.add(Report(**args_for_shared_usd_instance))
page2.add(Report(**args_for_shared_usd_instance))
Better naming, can make it convenient. Maybe there is better design solution.
I found some information myself.
I. curry -- associating parameters with a function « Python recipes « ActiveState Code
http://code.activestate.com/recipes/52549-curry-associating-parameters-with-a-function/
See the entire dicussion. Nick Perkins' comment on 'Lightweight' subclasses is similar to what I've described.
II. PEP 309 -- Partial Function Application
http://www.python.org/dev/peps/pep-0309/
The question is quite old, but this might still help someone who stumbles onto it...
I made a small library called classical to simplify class inheritance cases like this (Python 3 only).
Simple example:
from classical.descriptors import ArgumentedSubclass
class CustomReport(Report):
Expense = ArgumentedSubclass(data_source=EXPENSE, **OTHER_EXPENSE_KWARGS)
Usd = ArgumentedSubclass(format=USD)
Euro = ArgumentedSubclass(format=EURO)
PatternN = ArgumentedSubclass(**PATTERN_N_KWARGS)
PatternM = ArgumentedSubclass(**PATTERN_M_KWARGS)
# Now you can chain these in any combination (and with additional arguments):
my_report_1 = CustomReport.Expense.Usd(**kwargs)
my_report_2 = CustomReport.Expense.Euro(**kwargs)
my_report_3 = CustomReport.Expense.PatternM.PatternN(**kwargs)
In this example it's not really necessary to separate Report and CustomReport classes, but might be a good idea to keep the original class "clean".
Hope this helps :)
I'm coding a poker hand evaluator as my first programming project. I've made it through three classes, each of which accomplishes its narrowly-defined task very well:
HandRange = a string-like object (e.g. "AA"). getHands() returns a list of tuples for each specific hand within the string:
[(Ad,Ac),(Ad,Ah),(Ad,As),(Ac,Ah),(Ac,As),(Ah,As)]
Translation = a dictionary that maps the return list from getHands to values that are useful for a given evaluator (yes, this can probably be refactored into another class).
{'As':52, 'Ad':51, ...}
Evaluator = takes a list from HandRange (as translated by Translator), enumerates all possible hand matchups and provides win % for each.
My question: what should my "domain" class for using all these classes look like, given that I may want to connect to it via either a shell UI or a GUI? Right now, it looks like an assembly line process:
user_input = HandRange()
x = Translation.translateList(user_input)
y = Evaluator.getEquities(x)
This smells funny in that it feels like it's procedural when I ought to be using OO.
In a more general way: if I've spent so much time ensuring that my classes are well defined, narrowly focused, orthogonal, whatever ... how do I actually manage work flow in my program when I need to use all of them in a row?
Thanks,
Mike
Don't make a fetish of object orientation -- Python supports multiple paradigms, after all! Think of your user-defined types, AKA classes, as building blocks that gradually give you a "language" that's closer to your domain rather than to general purpose language / library primitives.
At some point you'll want to code "verbs" (actions) that use your building blocks to perform something (under command from whatever interface you'll supply -- command line, RPC, web, GUI, ...) -- and those may be module-level functions as well as methods within some encompassing class. You'll surely want a class if you need multiple instances, and most likely also if the actions involve updating "state" (instance variables of a class being much nicer than globals) or if inheritance and/or polomorphism come into play; but, there is no a priori reason to prefer classes to functions otherwise.
If you find yourself writing static methods, yearning for a singleton (or Borg) design pattern, writing a class with no state (just methods) -- these are all "code smells" that should prompt you to check whether you really need a class for that subset of your code, or rather whether you may be overcomplicating things and should use a module with functions for that part of your code. (Sometimes after due consideration you'll unearth some different reason for preferring a class, and that's allright too, but the point is, don't just pick a class over a module w/functions "by reflex", without critically thinking about it!).
You could create a Poker class that ties these all together and intialize all of that stuff in the __init__() method:
class Poker(object):
def __init__(self, user_input=HandRange()):
self.user_input = user_input
self.translation = Translation.translateList(user_input)
self.evaluator = Evaluator.getEquities(x)
# and so on...
p = Poker()
# etc, etc...