I have been programming in python for about two years; mostly data stuff (pandas, mpl, numpy), but also automation scripts and small web apps. I'm trying to become a better programmer and increase my python knowledge and one of the things that bothers me is that I have never used a class (outside of copying random flask code for small web apps). I generally understand what they are, but I can't seem to wrap my head around why I would need them over a simple function.
To add specificity to my question: I write tons of automated reports which always involve pulling data from multiple data sources (mongo, sql, postgres, apis), performing a lot or a little data munging and formatting, writing the data to csv/excel/html, send it out in an email. The scripts range from ~250 lines to ~600 lines. Would there be any reason for me to use classes to do this and why?
Classes are the pillar of Object Oriented Programming. OOP is highly concerned with code organization, reusability, and encapsulation.
First, a disclaimer: OOP is partially in contrast to Functional Programming, which is a different paradigm used a lot in Python. Not everyone who programs in Python (or surely most languages) uses OOP. You can do a lot in Java 8 that isn't very Object Oriented. If you don't want to use OOP, then don't. If you're just writing one-off scripts to process data that you'll never use again, then keep writing the way you are.
However, there are a lot of reasons to use OOP.
Some reasons:
Organization:
OOP defines well known and standard ways of describing and defining both data and procedure in code. Both data and procedure can be stored at varying levels of definition (in different classes), and there are standard ways about talking about these definitions. That is, if you use OOP in a standard way, it will help your later self and others understand, edit, and use your code. Also, instead of using a complex, arbitrary data storage mechanism (dicts of dicts or lists or dicts or lists of dicts of sets, or whatever), you can name pieces of data structures and conveniently refer to them.
State: OOP helps you define and keep track of state. For instance, in a classic example, if you're creating a program that processes students (for instance, a grade program), you can keep all the info you need about them in one spot (name, age, gender, grade level, courses, grades, teachers, peers, diet, special needs, etc.), and this data is persisted as long as the object is alive, and is easily accessible. In contrast, in pure functional programming, state is never mutated in place.
Encapsulation:
With encapsulation, procedure and data are stored together. Methods (an OOP term for functions) are defined right alongside the data that they operate on and produce. In a language like Java that allows for access control, or in Python, depending upon how you describe your public API, this means that methods and data can be hidden from the user. What this means is that if you need or want to change code, you can do whatever you want to the implementation of the code, but keep the public APIs the same.
Inheritance:
Inheritance allows you to define data and procedure in one place (in one class), and then override or extend that functionality later. For instance, in Python, I often see people creating subclasses of the dict class in order to add additional functionality. A common change is overriding the method that throws an exception when a key is requested from a dictionary that doesn't exist to give a default value based on an unknown key. This allows you to extend your own code now or later, allow others to extend your code, and allows you to extend other people's code.
Reusability: All of these reasons and others allow for greater reusability of code. Object oriented code allows you to write solid (tested) code once, and then reuse over and over. If you need to tweak something for your specific use case, you can inherit from an existing class and overwrite the existing behavior. If you need to change something, you can change it all while maintaining the existing public method signatures, and no one is the wiser (hopefully).
Again, there are several reasons not to use OOP, and you don't need to. But luckily with a language like Python, you can use just a little bit or a lot, it's up to you.
An example of the student use case (no guarantee on code quality, just an example):
Object Oriented
class Student(object):
def __init__(self, name, age, gender, level, grades=None):
self.name = name
self.age = age
self.gender = gender
self.level = level
self.grades = grades or {}
def setGrade(self, course, grade):
self.grades[course] = grade
def getGrade(self, course):
return self.grades[course]
def getGPA(self):
return sum(self.grades.values())/len(self.grades)
# Define some students
john = Student("John", 12, "male", 6, {"math":3.3})
jane = Student("Jane", 12, "female", 6, {"math":3.5})
# Now we can get to the grades easily
print(john.getGPA())
print(jane.getGPA())
Standard Dict
def calculateGPA(gradeDict):
return sum(gradeDict.values())/len(gradeDict)
students = {}
# We can set the keys to variables so we might minimize typos
name, age, gender, level, grades = "name", "age", "gender", "level", "grades"
john, jane = "john", "jane"
math = "math"
students[john] = {}
students[john][age] = 12
students[john][gender] = "male"
students[john][level] = 6
students[john][grades] = {math:3.3}
students[jane] = {}
students[jane][age] = 12
students[jane][gender] = "female"
students[jane][level] = 6
students[jane][grades] = {math:3.5}
# At this point, we need to remember who the students are and where the grades are stored. Not a huge deal, but avoided by OOP.
print(calculateGPA(students[john][grades]))
print(calculateGPA(students[jane][grades]))
Whenever you need to maintain a state of your functions and it cannot be accomplished with generators (functions which yield rather than return). Generators maintain their own state.
If you want to override any of the standard operators, you need a class.
Whenever you have a use for a Visitor pattern, you'll need classes. Every other design pattern can be accomplished more effectively and cleanly with generators, context managers (which are also better implemented as generators than as classes) and POD types (dictionaries, lists and tuples, etc.).
If you want to write "pythonic" code, you should prefer context managers and generators over classes. It will be cleaner.
If you want to extend functionality, you will almost always be able to accomplish it with containment rather than inheritance.
As every rule, this has an exception. If you want to encapsulate functionality quickly (ie, write test code rather than library-level reusable code), you can encapsulate the state in a class. It will be simple and won't need to be reusable.
If you need a C++ style destructor (RIIA), you definitely do NOT want to use classes. You want context managers.
I think you do it right. Classes are reasonable when you need to simulate some business logic or difficult real-life processes with difficult relations.
As example:
Several functions with share state
More than one copy of the same state variables
To extend the behavior of an existing functionality
I also suggest you to watch this classic video
dantiston gives a great answer on why OOP can be useful. However, it is worth noting that OOP is not necessary a better choice most cases it is used. OOP has the advantage of combining data and methods together. In terms of application, I would say that use OOP only if all the functions/methods are dealing and only dealing with a particular set of data and nothing else.
Consider a functional programming refactoring of dentiston's example:
def dictMean( nums ):
return sum(nums.values())/len(nums)
# It's good to include automatic tests for production code, to ensure that updates don't break old codes
assert( dictMean({'math':3.3,'science':3.5})==3.4 )
john = {'name':'John', 'age':12, 'gender':'male', 'level':6, 'grades':{'math':3.3}}
# setGrade
john['grades']['science']=3.5
# getGrade
print(john['grades']['math'])
# getGPA
print(dictMean(john['grades']))
At a first look, it seems like all the 3 methods exclusively deal with GPA, until you realize that Student.getGPA() can be generalized as a function to compute mean of a dict, and re-used on other problems, and the other 2 methods reinvent what dict can already do.
The functional implementation gains:
Simplicity. No boilerplate class or selfs.
Easily add automatic test code right after each
function for easy maintenance.
Easily split into several programs as your code scales.
Reusability for purposes other than computing GPA.
The functional implementation loses:
Typing in 'name', 'age', 'gender' in dict key each time is not very DRY (don't repeat yourself). It's possible to avoid that by changing dict to a list. Sure, a list is less clear than a dict, but this is a none issue if you include an automatic test code below anyway.
Issues this example doesn't cover:
OOP inheritance can be supplanted by function callback.
Calling an OOP class has to create an instance of it first. This can be boring when you don't have data in __init__(self).
A class defines a real world entity. If you are working on something that exists individually and has its own logic that is separate from others, you should create a class for it. For example, a class that encapsulates database connectivity.
If this not the case, no need to create class
It depends on your idea and design. If you are a good designer, then OOPs will come out naturally in the form of various design patterns.
For simple script-level processing, OOPs can be overhead.
Simply consider the basic benefits of OOPs like reusability and extendability and make sure if they are needed or not.
OOPs make complex things simpler and simpler things complex.
Simply keep the things simple in either way using OOPs or not using OOPs. Whichever is simpler, use that.
Related
I built a pygame game a few years back. It worked, but wasn't the best coding style and had a lot of classic code smells. I've recently picked it back up and am trying to refactor it with more discipline this time.
One big code smell was that I had a huge class, GameObject, that inherited from pygame.sprite.DirtySprite which had a lot of code related to:
various ways of moving a sprite
various ways of animating a sprite
various ways of exploding a sprite
etc.
The crazier I though of ways for sprites to behave, the code duplication was adding up and changes were getting more difficult. So, I started breaking out functionality into lots of smaller classes and then passing them in at object creation:
class GameObject(DirtySprite):
def __init__(initial_position, mover_cls, imager_cls, exploder_cls):
self.mover = mover(self, initial_position)
self.imager = imager(self)
self.exploder = exploder(self)
...
spaceship = GameObject(pos, CrazyMover, SpaceshipImager, BasicExploder)
As I factored out more and more code into these helper classes, the code was definitely better, more flexible and had less duplication. However, for each type of helper classes, the number of parameters got longer and longer. Creating sprites became a chore and the code was ugly. So, during another refactor I created a bunch of really small classes to do the composition:
class GameObjectAbstract(MoverAbstract, ImagerAbstract, \
ExploderAbstract, DirtySprite):
def __init__(self, initial_position):
...
...
class CrazySpaceship(CrazyMover, SpaceshipImager, BasicExploder, GameObjectAbstract):
pass # Many times, all the behavior comes from super classes
...
spaceship = CrazySpaceship(pos)
I like this approach better. Is this a common approach? It seems to have the same benefits of having all the logic broken out in small classes, but creating the objects is much cleaner.
However, this approach isn't as dynamic. I cannot, for example, decide on a new mashup at run-time. However, this wasn't something I was really doing. While I do a lot of mashups, it seems OK that they are statically defined using class statements.
Am I missing anything when it comes to future maintainability and reuse? I hear that composition is better than inheritance, but this feels like I'm using inheritance to do composition - so I feel like this is OK.
Is there a different pattern that I should be using?
That is ok, if you can separate the behaviors well enough -
Just that it is not "composition" at all - it is multiple inheritance, using what we call "mixin classes": a mixin class is roughly a class that provides an specific behavior that can be combined with other classes.
If you are using Python's super correctly, thatcouldbe the best approach. (If you are managing to create your game objects basically just defining the class name and the mixin classes it uses, that is actually a very good approach)
By the way, if you ever want to create new classes at runtime with this method, it is also possible - just use a call to type to create a new class, instead of a class statement:
class CrazySpaceship(CrazyMover, SpaceshipImager, BasicExploder, GameObjectAbstract):
pass # Many times, all the behavior comes from super classes
Is just equivalent in Python to:
CrazySpaceShip = type('CrazySpaceShip', (CrazyMover, SpaceshipImager, BasicExploder, GameObjectAbstract), {})
And the tuple you used as second parameter can be any sequence built at runtime.
Im writing code for research purposes, in which I search through a bulk of files and rank them according to their relevance. I call the entire process quickSearching, and it is composed of two serial stages - first I search the file and retrieve a list a candidates files, then I score those candidates and rank them.
So a quicksearch is simply a serial combination of a search method and a score method.
Im planning to implement various searching and scoring methodologies, and I would like to test all possible combinations and evaluate them to see which is the winning combo.
Since the number of combos will grow very fast, It is important to write the code in a good structure and design. I thought about the following designs (Im writing the code in python):
A quickSearcher class that will receive pointers to a searcher and scorer functions
A quickSearcher class that will receive a searcher object and a scorer object
A quickSearcher calss that will inherit from a searcher class and an scorer class
since Im basically an EE engineer, Im not sure how to select between the options and if this a common problem in CS with trivial pattern design.The design i'm looking will hopefully:
Be very code-volume efficient, since some of the searching and scoring methods differ in simply a different value of a parameter or two.
Be very modular and logical errors prone.
Be easy to navigate through
Any other consideration I should take?
This is my first design question so it might not be valid or missing important info, please notify me if it is.
Classes are often overused, especially by programmers coming from languages like Java and C# where they are compulsory. I recommend watching the presentation Stop Writing Classes.
When deciding whether to create a class it is useful to ask yourself the following questions:
1) Will the class need to have multiple methods?
If the class only has a single method (apart from __init__) then you may as well make it a function instead. If it needs to preserve state between calls then use a generator. If it needs to be created in one place with some parameters then called elsewhere you can use a closure (a function that returns another function) or functools.partial.
2) will it need to share state between methods?
If the class does not need to share state between methods then it may be better replaced with either a set of independent functions or smaller classes (or some combination).
If the answer to both questions is yes then go ahead and create a class.
For your example I think option 1 is the way to go. The searcher and scorer objects sound like they if they are classes they will only have a single method, probably called something like execute or run. Make them functions instead.
Depending on your use case, quickSorter itself may be better off as a function or generator as well, so no need for any classes at all.
BTW there is no distinction in Python between a function and a pointer to a function.
A fairly small question: does anyone know about a pre-made suite of Python unit tests that just check if a class conforms to one of the standard Python data structure interfaces (e.g., lists, sets, dictionaries, queues, etc). It's not overly hard to write them, but I'd hate to bother doing so if someone has already done this. It seems like very basic functionality that someone has probably done already.
The use case is that I am using a factory pattern to create data structures due to different restrictions related to platforms. As such, I need to be able to test that the resulting created objects still conform to the standard interfaces on the surface. Also, I should note that by "conform" I mean that the tests should check not just that the interface functions exist, but also check that they work (e.g., can set and retrieve a value in a map, for instance). Python 2.7 tests would be preferred.
First, "the standard Python data structure interfaces" are not lists, sets, dictionaries, queues, etc. Those are specific implementations of the interfaces. (And queue isn't even a data structure in the sense you're thinking of—its salient features are that its operations are atomic, and put and get optionally synchronize on a Condition, and so on.)
Anyway, the interfaces are defined in five different not-quite-compatible ways.
The Built-in Types section of the documentation describes what it means to be an iterator type, a sequence type, etc. However, these are not nearly as rigorous as you'd expect for reference documentation (at least if you're used to, say, C++ or Java).
I'm not aware of any tests for such a thing, so I think you'd have to build them from scratch.
The collections module contains Collections Abstract Base Classes that define the interfaces, and provide a way to register "virtual subclasses" via the abc module. So, you can declare "I am a mapping" by inheriting from collections.Mapping, or calling collections.Mapping.register. But that doesn't actually prove that you are a mapping, just that you're claiming to be. (If you inherit from Mapping, it also acts as a mixin that helps you complete the interface by implementing, e.g., __contains__ on top of __getitem__.)
If you want to test the ABC meaning, defuz's answer is very close, and with a little more work I think he or someone else can complete it.
The CPython C API defines an Abstract Objects Layer. While this is not actually authoritative for the language, it's obviously intended that the C-API protocols and the language-level interfaces are supposed to match. And, unlike the latter, the former are rigorously defined. And of course the source code from CPython 2.7, and maybe other implementations like PyPy, may help.
There are tests for this that come with CPython, but really, they're for testing that calling PyMapping_GetItem from C properly calls your mymapping.__getitem__ in Python, which is really at a tangent to what you want to test, so I don't think it will help much.
The actual concrete classes have additional interface on top of the protocols, that you may want to test, but that's harder to describe. In particular, the way the __new__ and __init__ methods work is often important. Implementing the Mapping protocol means someone can construct an empty Foo instance and add items to it with foo[key] = value, but it doesn't mean someone can construct Foo(key=value), or Foo({key: value}) or Foo([(key, value)]).
And for this case, there are existing tests that come with all of the standard Python implementations. CPython comes with a very extensive test suite that includes things like test_dict.py. PyPy runs all the (Python-level) CPython tests, and some extra ones besides.
You will obviously have to modify these tests to run on an arbitrary class instead of one hardcoded into the tests, and you may also have to modify them to handle whichever definition you pick. Plus, they probably test more than you asked for. You just want to know if a class conforms to the protocol, not whether its methods do the right thing, right? But still, I think they're a good starting point.
Finally, the C API defines a Concrete Objects Layer that, although it's not authoritative, matches the previous definition and is more rigorously defined.
Unfortunately, the tests for this one are definitely not going to be very useful to you, because they're checking things like whether PyDict_Check and PyDict_GetItem work on your class, which they will not for any mapping defined in pure Python.
If you do build something complete for any of these definitions, I would strongly suggest putting it on PyPI, and posting about it to python-list, so you get feedback (and bug reports).
There are abstract base classes in standart module collections based on ABC module.
You have to inherit your classes from these classes to be sure that your classes correspond to the standard behavior:
import collections
class MyDict(collections.Mapping):
...
Also, your can test already existed class that does not obviously inherit the abstract class:
class MyPerfectDict(object):
... realization ...
def is_inherit(cls, abstract):
try:
class Test(abstract, cls): pass
test = Test()
except TypeError:
return False
else:
return True
is_inherit(MyPerfectDict, Mapping) # False
is_inherit(dict, Mapping) # True
Note: I'm not talking about preventing the rebinding of a variable. I'm talking about preventing the modification of the memory that the variable refers to, and of any memory that can be reached from there by following the nested containers.
I have a large data structure, and I want to expose it to other modules, on a read-only basis. The only way to do that in Python is to deep-copy the particular pieces I'd like to expose - prohibitively expensive in my case.
I am sure this is a very common problem, and it seems like a constant reference would be the perfect solution. But I must be missing something. Perhaps constant references are hard to implement in Python. Perhaps they don't quite do what I think they do.
Any insights would be appreciated.
While the answers are helpful, I haven't seen a single reason why const would be either hard to implement or unworkable in Python. I guess "un-Pythonic" would also count as a valid reason, but is it really? Python does do scrambling of private instance variables (starting with __) to avoid accidental bugs, and const doesn't seem to be that different in spirit.
EDIT: I just offered a very modest bounty. I am looking for a bit more detail about why Python ended up without const. I suspect the reason is that it's really hard to implement to work perfectly; I would like to understand why it's so hard.
It's the same as with private methods: as consenting adults authors of code should agree on an interface without need of force. Because really really enforcing the contract is hard, and doing it the half-assed way leads to hackish code in abundance.
Use get-only descriptors, and state clearly in your documentation that these data is meant to be read only. After all, a determined coder could probably find a way to use your code in different ways you thought of anyways.
In PEP 351, Barry Warsaw proposed a protocol for "freezing" any mutable data structure, analogous to the way that frozenset makes an immutable set. Frozen data structures would be hashable and so capable being used as keys in dictionaries.
The proposal was discussed on python-dev, with Raymond Hettinger's criticism the most detailed.
It's not quite what you're after, but it's the closest I can find, and should give you some idea of the thinking of the Python developers on this subject.
There are many design questions about any language, the answer to most of which is "just because". It's pretty clear that constants like this would go against the ideology of Python.
You can make a read-only class attribute, though, using descriptors. It's not trivial, but it's not very hard. The way it works is that you can make properties (things that look like attributes but call a method on access) using the property decorator; if you make a getter but not a setter property then you will get a read-only attribute. The reason for the metaclass programming is that since __init__ receives a fully-formed instance of the class, you actually can't set the attributes to what you want at this stage! Instead, you have to set them on creation of the class, which means you need a metaclass.
Code from this recipe:
# simple read only attributes with meta-class programming
# method factory for an attribute get method
def getmethod(attrname):
def _getmethod(self):
return self.__readonly__[attrname]
return _getmethod
class metaClass(type):
def __new__(cls,classname,bases,classdict):
readonly = classdict.get('__readonly__',{})
for name,default in readonly.items():
classdict[name] = property(getmethod(name))
return type.__new__(cls,classname,bases,classdict)
class ROClass(object):
__metaclass__ = metaClass
__readonly__ = {'a':1,'b':'text'}
if __name__ == '__main__':
def test1():
t = ROClass()
print t.a
print t.b
def test2():
t = ROClass()
t.a = 2
test1()
While one programmer writing code is a consenting adult, two programmers working on the same code seldom are consenting adults. More so if they do not value the beauty of the code but them deadlines or research funds.
For such adults there is some type safety, provided by Enthought's Traits.
You could look into Constant and ReadOnly traits.
For some additional thoughts, there is a similar question posed about Java here:
Why is there no Constant feature in Java?
When asking why Python has decided against constant references, I think it's helpful to think of how they would be implemented in the language. Should Python have some sort of special declaration, const, to create variable references that can't be changed? Why not allow variables to be declared a float/int/whatever then...these would surely help prevent programming bugs as well. While we're at it, adding class and method modifiers like protected/private/public/etc. would help enforce compile-type checking against illegal uses of these classes. ...pretty soon, we've lost the beauty, simplicity, and elegance that is Python, and we're writing code in some sort of bastard child of C++/Java.
Python also currently passes everything by reference. This would be some sort of special pass-by-reference-but-flag-it-to-prevent-modification...a pretty special case (and as the Tao of Python indicates, just "un-Pythonic").
As mentioned before, without actually changing the language, this type of behaviour can be implemented via classes & descriptors. It may not prevent modification from a determined hacker, but we are consenting adults. Python didn't necessarily decide against providing this as an included module ("batteries included") - there was just never enough demand for it.
OK I've got 2 really big classes > 1k lines each that I currently have split up into multiple ones. They then get recombined using multiple inheritance. Now I'm wondering, if there is any cleaner/better more pythonic way of doing this. Completely factoring them out would result in endless amounts of self.otherself.do_something calls, which I don't think is the way it should be done.
To make things clear here's what it currently looks like:
from gui_events import GUIEvents # event handlers
from gui_helpers import GUIHelpers # helper methods that don't directly modify the GUI
# GUI.py
class GUI(gtk.Window, GUIEvents, GUIHelpers):
# general stuff here stuff here
One problem that is result of this is Pylint complaining giving me trillions of "init not called" / "undefined attribute" / "attribute accessed before definition" warnings.
EDIT:
You may want to take a look at the code, to make yourself a picture about what the whole thing actually is.
http://github.com/BonsaiDen/Atarashii/tree/next/atarashii/usr/share/pyshared/atarashii/
Please note, I'm really trying anything to keep this thing as DRY as possible, I'm using pylint to detect code duplication, the only thing it complains about are the imports.
If you want to use multiple inheritance to combine everything into one big class (it might make sense to do this), then you can refactor each of the parent classes so that every method and property is either private (starts with '__') or has a short 2-3 character prefix unique to that class. For example, all the methods and properties in your GUIEvents class could start with ge_, everything in GUIHelpers could start with gh_. By doing this, you'll get achieve some of the clarity of using separate sub-class instances (self.ge.doSomething() vs self.ge_doSomething()) and you'll avoid conflicting member names, which is the main risk when combining such large classes into one.
Start by finding classes that model real world concepts that your application needs to work with. Those are natural candidates for classes.
Try to avoid multiple inheritance as much as possible; it's rarely useful and always somewhat confusing. Instead, look to use functional composition ("HAS-A" relationships) to give rich attributes to your objects made of other objects.
Remember to make each method do one small, specific thing; this necessarily entails breaking up methods that do too many things into smaller pieces.
Refactor cases where you find many such methods are duplicating each other's functionality; this is another way to find natural collections of functionality that deserve to be in a distinct class.
I think this is more of a general OO-design problem than Python problem. Python pretty much gives you all the classic OOP tools, conveniently packaged. You'd have to describe the problem in more detail (e.g. what do the GUIEvents and GUIHelpers classes contain?)
One Python-specific aspect to consider is the following: Python supports multiple programming paradigms, and often the best solution is not OOP. This may be the case here. But again, you'll have to throw in more details to get a meaningful answer.
Your code may be substantially improved by implementing a Model-View-Controller design. Depending on how your GUI and tool are setup, you may also benefit from "widgetizing" portions of your GUI, so that rather than having one giant Model-View-Controller, you have a main Model-View-Controller that manages a bunch of smaller Model-View-Controllers, each for distinct portions of your GUI. This would allow you to break up your tool and GUI into many classes, and you may be able to reuse portions of it, reducing the total amount of code you need to maintain.
While python does support multiple programming paradigms, for GUI tools, the best solution will nearly always be an Object-Oriented design.
One possibility is to assign imported functions to class attributes:
In file a_part_1.py:
def add(self, n):
self.n += n
def __init__(self, n):
self.n = n
And in main class file:
import a_part_1
class A:
__init__ = a_part_1.__init__
add = a_part_1.add
Or if you don't want to update main file when new methods are added:
class A: pass
import a_part_1
for k, v in a_part_1.__dict__.items():
if callable(v):
setattr(A,k,v)