Related
I have a class that looks like the following
class A:
communicate = set()
def __init__(self):
pass
...
def _some_func(self):
...some logic...
self.communicate.add(some_var)
The communicate variable is shared among the instances of the class. I use it to provide a convenient way for the instances of this class to communicate with one another (they have some mild orchestration needed and I don't want to force the calling object to serve as an intermediary of this communication). However, I realized this causes problems when I run my tests. If I try to test multiple aspects of my code, since the python interpreter is the same throughout all the tests, I won't get a "fresh" A class for the tests, and as such the communicate set will be the aggregate of all objects I add to that set (in normal usage this is exactly what I want, but for testing I don't want interactions between my tests). Furthermore, down the line this will also cause problems in my code execution if I want to loop over my whole process multiple times (because I won't have a way of resetting this class variable).
I know I can fix this issue where it occurs by having the creator of the A objects do something like
A.communicate = set()
before it creates and uses any instances of A. However, I don't really love this because it forces my caller to know some details about the communication pathways of the A objects, and I don't want that coupling. Is there a better way for me to to reset the communicate A class variable? Perhaps some method I could call on the class instead of an instance itself like (A.new_batch()) that would perform this resetting? Or is there a better way I'm not familiar with?
Edit:
I added a class method like
class A:
communicate = set()
def __init__(self):
pass
...
#classmethod
def new_batch(cls):
cls.communicate = set()
def _some_func(self):
...some logic...
self.communicate.add(some_var)
and this works with the caller running A.new_batch(). Is this the way it should be constructed and called, or is there a better practice here?
Using "new" style classes (I'm in python 3.2) is there a way to split a class over multiple files? I've got a large class (which really should be a single class from an object-oriented design perspective, considering coupling, etc, but it'd be nice to split over a few files just for ease of editing the class.
If your problem really is just working with a large class in an editor, the first solution I'd actually look for is a better way to break down the problem. The second solution would be a better editor, preferably one with code folding.
That said, there are a couple of ways you might break up a class into multiple files. Python lets you use a folder as a module by putting an __init__.py in it, which can then import things from other files. We'll use this capability in each solution. Make a folder called, say, bigclass first.
In the folder put the various .py files that will eventually comprise your class. Each should contain functions and variable definitions for the eventual class, not classes. In __init__.py in the same folder write the following to join them all together.
class Bigclass(object):
from classdef1 import foo, bar, baz, quux
from classdef2 import thing1, thing2
from classdef3 import magic, moremagic
# unfortunately, "from classdefn import *" is an error or warning
num = 42 # add more members here if you like
This has the advantage that you end up with a single class derived directly from object, which will look nice in your inheritance graphs.
You could use multiple inheritance to combine the various parts of your class. In your individual modules you would write a class definition for Bigclass with parts of the class. Then in your __init__.py write:
import classdef1, classdef2, classdef3
class Bigclass(classdef1.Bigclass, classdef2.Bigclass, classdef3.Bigclass):
num = 42 # add more members if desired
If the multiple inheritance becomes an issue, you can use single inheritance: just have each class inherit from another one in chain fashion. Assuming you don't define anything in more than one class, the order doesn't matter. For example, classdef2.py would be like:
import classdef1
class Bigclass(classdef1.Bigclass):
# more member defs here
classdef3 would import Bigclass from classdef2 and add to it, and so on. Your __init__.py would just import the last one:
from classdef42 import Bigclass
I'd generally prefer #1 because it's more explicit about what members you're importing from which files but any of these solutions could work for you.
To use the class in any of these scenarios you can just import it, using the folder name as the module name: from bigclass import Bigclass
You can do this with decorators like so:
class Car(object):
def start(self):
print 'Car has started'
def extends(klass):
def decorator(func):
setattr(klass, func.__name__, func)
return func
return decorator
#this can go in a different module/file
#extends(Car)
def do_start(self):
self.start()
#so can this
car = Car()
car.do_start()
#=> Car has started
Class definitions containing hundreds of lines do occur "in the wild" (I have seen some in popular open-source Python-based frameworks), but I believe that if you ponder what the methods are doing, it will be possible to reduce the length of most classes to a manageable point. Some examples:
Look for places where mostly the same code occurs more than once. Break that code out into its own method and call it from each place with arguments.
"Private" methods that do not use any of the object state can be brought out of the class as stand-alone functions.
Methods that should be called only under certain conditions may indicate a need to place those methods in a subclass.
To directly address your question, it is possible to split up the definition of a class. One way is to "monkey-patch" the class by defining it and then adding outside functions to it as methods. Another is to use the built-in type function to create the class "by hand", supplying its name, any base classes, and its methods and attributes in a dictionary. But I do not recommend doing this just because the definition would be long otherwise. That sort of cure is worse than the disease in my opinion.
I've previously toyed around with something similar. My usecase was a class hierarchy of nodes in an abstract syntax tree, and then I wanted to put all e.g. prettyprinting functions in a separate prettyprint.py file but still have them as methods in the classes.
One thing I tried was to use a decorator that puts the decorated function as an attribute on a specified class. In my case this would mean that prettyprint.py contains lots of def prettyprint(self) all decorated with different #inclass(...)
A problem with this is that one must make sure that the sub files are always imported, and that they depend on the main class, which makes for a circular dependency, which may be messy.
def inclass(kls):
"""
Decorator that adds the decorated function
as a method in specified class
"""
def _(func):
setattr(kls,func.__name__, func)
return func
return _
## exampe usage
class C:
def __init__(self, d):
self.d = d
# this would be in a separate file.
#inclass(C)
def meth(self, a):
"""Some method"""
print "attribute: %s - argument: %s" % (self.d, a)
i = C(10)
print i.meth.__doc__
i.meth(20)
I've not used it, but this package called partial claims to add support for partial classes.
It seems like there's a few other ways you could implement this yourself as well.
You could implement separate parts of the class as mixins in seperate files, then import them all somewhere and subclass them.
Alternatively, you could implement each of the methods of your class somewhere then in a central file import them and assign them as attributes on a class, to create the whole object. Like so:
a.py:
def AFunc( self, something ):
# Do something
pass
b.py:
def BFunc( self, something ):
# Do something else
pass
c.py:
import a, b
class C:
AFunc = a.AFunc
BFunc = b.BFunc
You could even go so far as to automate this process if you really wanted - loop through all the functions provided by modules a and b and then add them as attributes on C. Though that might be total overkill.
There might be other (possibly better) ways to go about it, but those are the 2 that popped into mind.
I would like to add that the pythonic way of doing this is through multiple inheritance, not necessarily using mixins. Instance attributes can be added using super().__init__(*args, **kwargs) in __init__ calls to pass arguments to baseclasses (see ‘super considered super’ presentation by Raymond Hettinger 1). This also enables dependency injection and kind of forces you to think about organization of base classes (it works best if only one baseclass sets an attribute in __init__ and all classes using the attribute inherit from it).
This does usually require you having control over the base classes (or they being written for this pattern).
Another option is using descriptors returning functions through __get__ to add functionality to classes in a decoupled way.
You could also look at __init_subclass__ to add e.g. methods to classes during class generation (i think added in python 3.6, but check)
First I'd like to say that something this complicated it probably not a good idea just to make finding your place in the class easier - it would be best to add comments, highlight sections etc. However, I see two ways you could do this:
Write the class in several files, then read them in as text, concatenate them and exec the resulting string.
Create a separate class in each file, then inherit them all into a master class as mixins. However, if you're subclassing another class already this could lead to MRO problems. You could get around this by creating a metaclass for your master class which manually resolves the MRO, but this could get messy.
The easiest would be the first option.
First off, I don't see how splitting the class into multiple files makes editing any easier. A decent IDE should be able to find any method easily whether in one file or multiple; if you're not using a decent IDE, splitting the class means the maintainer has to guess which file a given method is in, which sounds harder rather than easier.
More fundamentally, this class - so large that you want a special language feature just to support its weight - sounds fundamentally broken. How many lines of code are we talking about? Almost certainly, it would be a better idea to do one of:
Refactor duplicated code into fewer, more general primitives
Define a base class and extend it with subclasses as Karoly Horvath suggests in comments (this is the closest thing to the 'partial classes' that you're asking for that I would endorse)
Define a few separate classes to encapsulate different parts of this
class's functionality, and compose this class of instances of those
smaller ones.
I met the same situation - I want to slipt my class to 2 files.
the reason is that - I want part 1 for GUI layout, only layout
and another file keeps all function.
like c#'s Partial class. one for XAML and another one for functions.
I really don't know how to word this problem, so I'll try to explain it with an example.
Let's say I have three GUI classes:
Base Surface class
Detailed Surface Class
Sprite Class
All of them are independent of each other, no inheritance among them.
Now I have a function "drag()" that makes a surface/sprite dragable, and I want to implement this function as a method for all three of them.
Since it's the exact same code for all implementations I find it annoying, cumbersome and bad practice to rewrite the code.
The only thing I came up with so far was to make a saperate class for it and inherit this class. But that also doesn't seem to be the way to go.
I'd be very thankfull for some advice.
EDIT
Another example with a slightly different setup - I have the following classes:
BaseSurface
Dragable
Resizable
EventHandler
Only the first one is independent, the others depend on the first (must be inherited).
The end user should, without any effort, be able to choose between a simple BaseSurface, one with that implements dragable, one with resizable, one with eventHandler, and any combination. By "without any effort" I mean the end user should not have to make e custom Class and inherit the desired classes plus call the appropriate methods (init, update, ...) that some classes share.
So what I could do is make a class for every possible combination, eg.
"BaseSurfaceDrag", "BaseSurfaceDragResize", ...
which will get messy really quickly. Whats a different and better approach to this?
This is exactly the kind of case that you should use a parent class for. In both cases it looks like your parent class (logically) should be something like:
class Drawable(object):
def drag(self, *args, **kwargs):
"""Drag and drop behavior"""
# Your code goes here
Then each of your other classes inherits from that
class BaseSurface(Drawable):
# stuff
class DetailedSurface(Drawable):
# stuff
class Sprite(Drawable):
# stuff
In the second case what you have are interfaces, so you could logically do something like:
class DragInterface(object):
"""Implements a `drag` method"""
def drag(self):
"""Drag and drop behavior"""
# Your code goes here
class ResizeInterface(object):
"""Implements a `resize` method"""
def resize(self):
"""Drag and drop resize"""
# Code
class EventHandlerInterface(object):
"""Handles events"""
def handle(self, evt):
# Code
class MyNewSurface(BaseSurface, DragInterface, ResizeInterface):
"""Draggable, resizeable surface"""
# Implement here
Using "new" style classes (I'm in python 3.2) is there a way to split a class over multiple files? I've got a large class (which really should be a single class from an object-oriented design perspective, considering coupling, etc, but it'd be nice to split over a few files just for ease of editing the class.
If your problem really is just working with a large class in an editor, the first solution I'd actually look for is a better way to break down the problem. The second solution would be a better editor, preferably one with code folding.
That said, there are a couple of ways you might break up a class into multiple files. Python lets you use a folder as a module by putting an __init__.py in it, which can then import things from other files. We'll use this capability in each solution. Make a folder called, say, bigclass first.
In the folder put the various .py files that will eventually comprise your class. Each should contain functions and variable definitions for the eventual class, not classes. In __init__.py in the same folder write the following to join them all together.
class Bigclass(object):
from classdef1 import foo, bar, baz, quux
from classdef2 import thing1, thing2
from classdef3 import magic, moremagic
# unfortunately, "from classdefn import *" is an error or warning
num = 42 # add more members here if you like
This has the advantage that you end up with a single class derived directly from object, which will look nice in your inheritance graphs.
You could use multiple inheritance to combine the various parts of your class. In your individual modules you would write a class definition for Bigclass with parts of the class. Then in your __init__.py write:
import classdef1, classdef2, classdef3
class Bigclass(classdef1.Bigclass, classdef2.Bigclass, classdef3.Bigclass):
num = 42 # add more members if desired
If the multiple inheritance becomes an issue, you can use single inheritance: just have each class inherit from another one in chain fashion. Assuming you don't define anything in more than one class, the order doesn't matter. For example, classdef2.py would be like:
import classdef1
class Bigclass(classdef1.Bigclass):
# more member defs here
classdef3 would import Bigclass from classdef2 and add to it, and so on. Your __init__.py would just import the last one:
from classdef42 import Bigclass
I'd generally prefer #1 because it's more explicit about what members you're importing from which files but any of these solutions could work for you.
To use the class in any of these scenarios you can just import it, using the folder name as the module name: from bigclass import Bigclass
You can do this with decorators like so:
class Car(object):
def start(self):
print 'Car has started'
def extends(klass):
def decorator(func):
setattr(klass, func.__name__, func)
return func
return decorator
#this can go in a different module/file
#extends(Car)
def do_start(self):
self.start()
#so can this
car = Car()
car.do_start()
#=> Car has started
Class definitions containing hundreds of lines do occur "in the wild" (I have seen some in popular open-source Python-based frameworks), but I believe that if you ponder what the methods are doing, it will be possible to reduce the length of most classes to a manageable point. Some examples:
Look for places where mostly the same code occurs more than once. Break that code out into its own method and call it from each place with arguments.
"Private" methods that do not use any of the object state can be brought out of the class as stand-alone functions.
Methods that should be called only under certain conditions may indicate a need to place those methods in a subclass.
To directly address your question, it is possible to split up the definition of a class. One way is to "monkey-patch" the class by defining it and then adding outside functions to it as methods. Another is to use the built-in type function to create the class "by hand", supplying its name, any base classes, and its methods and attributes in a dictionary. But I do not recommend doing this just because the definition would be long otherwise. That sort of cure is worse than the disease in my opinion.
I've previously toyed around with something similar. My usecase was a class hierarchy of nodes in an abstract syntax tree, and then I wanted to put all e.g. prettyprinting functions in a separate prettyprint.py file but still have them as methods in the classes.
One thing I tried was to use a decorator that puts the decorated function as an attribute on a specified class. In my case this would mean that prettyprint.py contains lots of def prettyprint(self) all decorated with different #inclass(...)
A problem with this is that one must make sure that the sub files are always imported, and that they depend on the main class, which makes for a circular dependency, which may be messy.
def inclass(kls):
"""
Decorator that adds the decorated function
as a method in specified class
"""
def _(func):
setattr(kls,func.__name__, func)
return func
return _
## exampe usage
class C:
def __init__(self, d):
self.d = d
# this would be in a separate file.
#inclass(C)
def meth(self, a):
"""Some method"""
print "attribute: %s - argument: %s" % (self.d, a)
i = C(10)
print i.meth.__doc__
i.meth(20)
I've not used it, but this package called partial claims to add support for partial classes.
It seems like there's a few other ways you could implement this yourself as well.
You could implement separate parts of the class as mixins in seperate files, then import them all somewhere and subclass them.
Alternatively, you could implement each of the methods of your class somewhere then in a central file import them and assign them as attributes on a class, to create the whole object. Like so:
a.py:
def AFunc( self, something ):
# Do something
pass
b.py:
def BFunc( self, something ):
# Do something else
pass
c.py:
import a, b
class C:
AFunc = a.AFunc
BFunc = b.BFunc
You could even go so far as to automate this process if you really wanted - loop through all the functions provided by modules a and b and then add them as attributes on C. Though that might be total overkill.
There might be other (possibly better) ways to go about it, but those are the 2 that popped into mind.
I would like to add that the pythonic way of doing this is through multiple inheritance, not necessarily using mixins. Instance attributes can be added using super().__init__(*args, **kwargs) in __init__ calls to pass arguments to baseclasses (see ‘super considered super’ presentation by Raymond Hettinger 1). This also enables dependency injection and kind of forces you to think about organization of base classes (it works best if only one baseclass sets an attribute in __init__ and all classes using the attribute inherit from it).
This does usually require you having control over the base classes (or they being written for this pattern).
Another option is using descriptors returning functions through __get__ to add functionality to classes in a decoupled way.
You could also look at __init_subclass__ to add e.g. methods to classes during class generation (i think added in python 3.6, but check)
First I'd like to say that something this complicated it probably not a good idea just to make finding your place in the class easier - it would be best to add comments, highlight sections etc. However, I see two ways you could do this:
Write the class in several files, then read them in as text, concatenate them and exec the resulting string.
Create a separate class in each file, then inherit them all into a master class as mixins. However, if you're subclassing another class already this could lead to MRO problems. You could get around this by creating a metaclass for your master class which manually resolves the MRO, but this could get messy.
The easiest would be the first option.
First off, I don't see how splitting the class into multiple files makes editing any easier. A decent IDE should be able to find any method easily whether in one file or multiple; if you're not using a decent IDE, splitting the class means the maintainer has to guess which file a given method is in, which sounds harder rather than easier.
More fundamentally, this class - so large that you want a special language feature just to support its weight - sounds fundamentally broken. How many lines of code are we talking about? Almost certainly, it would be a better idea to do one of:
Refactor duplicated code into fewer, more general primitives
Define a base class and extend it with subclasses as Karoly Horvath suggests in comments (this is the closest thing to the 'partial classes' that you're asking for that I would endorse)
Define a few separate classes to encapsulate different parts of this
class's functionality, and compose this class of instances of those
smaller ones.
I met the same situation - I want to slipt my class to 2 files.
the reason is that - I want part 1 for GUI layout, only layout
and another file keeps all function.
like c#'s Partial class. one for XAML and another one for functions.
I have several classes that share some invariants and have a common interface, and I would like to run automatically the same test for each of them.
As an example, suppose I have several classes that implement different approaches for partitioning a data-set. The common invariant here would be, that for all of these classes the union over all partitions should equal the original data-set.
What I currently have looks something like this:
class PartitionerInvariantsTests(unittest.TestCase):
def setUp(self):
self.testDataSet = range(100) # create test-data-set
def impl(self, partitioner):
self.assertEqual(self.testDataSet,
chain.from_iterable(partitioner(self.testDataSet))
Then I add a different function that calls impl for each of the classes I want to test with an instance of that class. The problem with this becomes apparent when doing this for more than one test-function. Suppose I have 5 test-functions and 5 classes I want to test. This would make 25 functions that look almost identical for invoking all the tests.
Another approach I was thinking about was to implement the template as a super-class, and then create a sub-class for each of the classes I want to test. The sub-classes could provide a function for instantiating the class. The problem with that is that the default test-loader would consider the (unusable) base-class a valid test-case and try to run it, which would fail.
So, what are your suggestions?
P.S.: I am using Python 2.6
You could use multiple inheritance.
class PartitionerInvariantsFixture(object):
def setUp(self):
self.testDataSet = range(100) # create test-data-set
super(PartitionInvariantsFixture, self).setUp()
def test_partitioner(self):
TestCase.assertEqual(self.testDataSet,
chain.from_iterable(self.partitioner(self.testDataSet))
class MyClassTests(TestCase, PartitionerInvariantsFixture):
partitioner = Partitioner
Subclass PartitionerInvariantsTests:
class PartitionerInvariantsTests(unittest.TestCase):
def test_impl(self):
self.assertEqual(self.testDataSet,
chain.from_iterable(self.partitioner(self.testDataSet))
class PartitionerATests(PartitionerInvariantsTests):
for each Partitioner class you wish to test. Then test_impl would be run for each Partitioner class, by virtue of inheritance.
Following up on Nathon's comment, you can prevent the base class from being tested by having it inherit only from object:
import unittest
class Test(object):
def test_impl(self):
print('Hi')
class TestA(Test,unittest.TestCase):
pass
class TestB(Test,unittest.TestCase):
pass
if __name__ == '__main__':
unittest.sys.argv.insert(1,'--verbose')
unittest.main(argv = unittest.sys.argv)
Running test.py yields
test_impl (__main__.TestA) ... Hi
ok
test_impl (__main__.TestB) ... Hi
ok
----------------------------------------------------------------------
Ran 2 tests in 0.000s
OK