We Keep Coding

How can I separate the functions of a class into multiple files?

I have a main class that has a ton of different functions in it. It's getting hard to manage. I'd like to be able to separate those functions into a separate file, but I'm finding it hard to come up with a good way to do so.
Here's what I've done so far:
File main.py
import separate
class MainClass(object):
self.global_var_1 = ...
self.global_var_2 = ...
def func_1(self, x, y):
...
def func_2(self, z):
...
# tons of similar functions, and then the ones I moved out:
def long_func_1(self, a, b):
return separate.long_func_1(self, a, b)
File separate.py
def long_func_1(obj, a, b):
if obj.global_var_1:
...
obj.func_2(z)
...
return ...
# Lots of other similar functions that use info from MainClass
I do this because if I do:
obj_1 = MainClass()
I want to be able to do:
obj_1.long_func_1(a, b)
instead of:
separate.long_func_1(obj_1, a, b)
I know this seems kind of nit-picky, but I want just about all of the code to start with obj_1., so there isn't confusion.
Is there a better solution that what I'm currently doing? The only issues that I have with my current setup are:
I have to change arguments for both instances of the function
It seems needlessly repetitive
I know this has been asked a couple of times, but I couldn't quite understand the previous answers and/or I don't think the solution quite represents what I'm shooting for. I'm still pretty new to Python, so I'm having a tough time figuring this out.

Here is how I do it:
Class (or group of) is actually a full module. You don't have to do it this way, but if you're splitting a class on multiple files I think this is 'cleanest' (opinion).
The definition is in __init__.py, methods are split into files by a meaningful grouping.
A method file is just a regular Python file with functions, except you can't forget 'self' as a first argument. You can have auxiliary methods here, both taking self and not.
Methods are imported directly into the class definition.
Suppose my class is some fitting GUI (this is actually what I did this for first time). So my file hierarchy may look something like
mymodule/
__init__.py
_plotstuff.py
_fitstuff.py
_datastuff.py
So plot stuff will have plotting methods, fit stuff contains fitting methods, and data stuff contains methods for loading and handling of data - you get the point. By convention I mark the files with a _ to indicate these really aren't meant to be imported directly anywhere outside the module. So _plotsuff.py for example may look like:
def plot(self,x,y):
#body
def clear(self):
#body
etc. Now the important thing is file __init__.py:
class Fitter(object):
def __init__(self,whatever):
self.field1 = 0
self.field2 = whatever
# Imported methods
from ._plotstuff import plot, clear
from ._fitstuff import fit
from ._datastuff import load
# static methods need to be set
from ._static_example import something
something = staticmethod(something)
# Some more small functions
def printHi(self):
print("Hello world")
Tom Sawyer mentions PEP-8 recommends putting all imports at the top, so you may wish to put them before __init__, but I prefer it this way. I have to say, my Flake8 checker does not complain, so likely this is PEP-8 compliant.
Note the from ... import ... is particularly useful to hide some 'helper' functions to your methods you don't want accessible through objects of the class. I usually also place the custom exceptions for the class in the different files, but import them directly so they can be accessed as Fitter.myexception.
If this module is in your path then you can access your class with
from mymodule import Fitter
f = Fitter()
f.load('somefile') # Imported method
f.plot() # Imported method
It is not completely intuitive, but not too difficult either. The short version for your specific problem was you were close - just move the import into the class, and use
from separate import long_func_1
and don't forget your self!
How to use super addendum
super() is a useful nifty function allowing parent method access in a simple and readable manner from the child object. These kind of classes are big to begin with, so inheritance not always make sense, but if it does come up:
For methods defined in the class itself, within __init__.py, you can use super() normally, as is.
If you define you method in another module (which is kind of the point here), you can't use super as is since the function is not defined in the context of your cell, and will fail. The way to handle this is to use the self argument, and add the context yourself:
def print_super(self):
print('Super is:', super(type(self), self))
Note you cannot omit the second argument, since out of context super does not bind the object method (which you usually want for calls like super(...).__init__()).
If this is something you want to do in many methods in different modules, you may want to provide a super method in the __init__.py file for use:
def MySuper(self):
return super()
usable by self in all methods.

I use the approach I found here. It shows many different approaches, but if you scroll down to the end, the preferred method is to basically go the opposite direction of #Martin Pieter's suggestion which is have a base class that inherits other classes with your methods in those classes.
So the folder structure is something like:
_DataStore/
__init__.py
DataStore.py
_DataStore.py
So your base class would be:
File DataStore.py
import _DataStore
class DataStore(_DataStore.Mixin): # Could inherit many more mixins
def __init__(self):
self._a = 1
self._b = 2
self._c = 3
def small_method(self):
return self._a
Then your Mixin class:
File _DataStore.py
class Mixin:
def big_method(self):
return self._b
def huge_method(self):
return self._c
Your separate methods would be located in other appropriately named files, and in this example it is just _DataStore.
I am interested to hear what others think about this approach. I showed it to someone at work and they were scared by it, but it seemed to be a clean and easy way to separate a class into multiple files.

Here is an implementation of Martijn Pieters's comment to use subclasses:
File main.py
from separate import BaseClass
class MainClass(BaseClass):
def long_func_1(self, a, b):
if self.global_var_1:
...
self.func_2(z)
...
return ...
# Lots of other similar functions that use info from BaseClass
File separate.py
class BaseClass(object):
# You almost always want to initialize instance variables in the `__init__` method.
def __init__(self):
self.global_var_1 = ...
self.global_var_2 = ...
def func_1(self, x, y):
...
def func_2(self, z):
...
# tons of similar functions, and then the ones I moved out:
#
# Why are there "tons" of _similar_ functions?
# Remember that functions can be defined to take a
# variable number of/optional arguments, lists/tuples
# as arguments, dicts as arguments, etc.
from main import MainClass
m = MainClass()
m.func_1(1, 2)
....

How do you decide which level a function should be at in python?

I have a file called file_parsers.py and it contains the following class:
class FileParser():
def __init__(self, file_text):
self.file_text = file_text
def do_something(self):
my_value = func_with_no_state()
I'm not sure what questions to ask when deciding whether func_with_no_state() should be inside the class or outside of the class as a file-level function?
Also, is it easier to stub this function when it is at a file-level or inside the class?

So... Does any other class use func_with_no_state? If not, it should be hidden within FileParser. If something else does use it, you have a bigger question. If OtherClass uses func_with_no_state pretty frequently (on par with FileParser) then it would be a good idea to keep func_with_no_state outside so that both classes can use it. But if FileParser is by far the main user, then OtherClass could just pull the function from FileParser's definition.

Using a metaclass to substitute a class definition?

Python 3.6
I'm trying to modify the behavior of a third party library.
I don't want to directly change the source code.
Considering this code below:
class UselessObject(object):
pass
class PretendClassDef(object):
"""
A class to highlight my problem
"""
def do_something(self):
# Allot of code here
result = UselessObject()
return result
I'd like to substitute my own class for UselessObject
I'd like to know if using a metaclass in my module to intercept the creation of UselessObject is a valid idea?
EDIT
This answer posted by Ashwini Chaudhary on the same question, may be of use to others. As well as the below answer.
P.S. I also discovered that 'module' level __metaclass__ does't work in python 3. So my initial question of it 'being a valid idea' is False

FWIW, here's some code that illustrates Rawing's idea.
class UselessObject(object):
def __repr__(self):
return "I'm useless"
class PretendClassDef(object):
def do_something(self):
return UselessObject()
# -------
class CoolObject(object):
def __repr__(self):
return "I'm cool"
UselessObject = CoolObject
p = PretendClassDef()
print(p.do_something())
output
I'm cool
We can even use this technique if CoolObject needs to inherit UselessObject. If we change the definition of CoolObject to:
class CoolObject(UselessObject):
def __repr__(self):
s = super().__repr__()
return "I'm cool, but my parent says " + s
we get this output:
I'm cool, but my parent says I'm useless
This works because the name UselessObject has its old definition when the CoolObject class definition is executed.

This is not a job for metaclasses.
Rather, Python allows you to do this through a technique called "Monkeypatching", in which you, at run time, substitute one object for another in run time.
In this case, you'd be changing the thirdyparty.UselessObject for your.CoolObject before calling thirdyparty.PretendClassDef.do_something
The way to do that is a simple assignment.
So, supposing the example snippet you gave on the question is the trirdyparty module, on the library, your code would look like:
import thirdyparty
class CoolObject:
# Your class definition here
thirdyparty.UselesObject = Coolobject
Things you have to take care of: that you change the object pointed by UselessObject in the way it is used in your target module.
If for example, your PretendedClassDef and UselessObject are defined in different modules, you have to procees in one way if UselessObject is imported with from .useless import UselessObject (in this case the example above is fine), and import .useless and later uses it as useless.UselessObject - in this second case, you have to patch it on the useless module.
Also, Python's unittest.mock has a nice patch callable that can properly perform a monkeypatching and undo it if by some reason you want the modification to be valid in a limited scope, like inside a function of yours, or inside a with block. That might be the case if you don't want to change the behavior of the thirdyparty module in other sections of your program.
As for metaclasses, they only would be of any use if you would need to change the metaclass of a class you'd be replacing in this way - and them they only could have any use if you'd like to insert behavior in classes that inherit from UselessObject. In that case it would be used to create the local CoolObject and you'd still perform as above, but taking care that you'd perform the monkeypatching before Python would run the class body of any of the derived classes of UselessObject, taking extreme care when doing any imports from the thirdparty library (that would be tricky if these subclasses were defined on the same file)

This is just building on PM 2Ring's and jsbueno's answers with more contexts:
If you happen to be creating a library for others to use as a third-party library (rather than you using the third-party library), and if you need CoolObject to inherit UselessObject to avoid repetition, the following may be useful to avoid an infinite recursion error that you might get in some circumstances:
module1.py
class Parent:
def __init__(self):
print("I'm the parent.")
class Actor:
def __init__(self, parent_class=None):
if parent_class!=None: #This is in case you don't want it to actually literally be useless 100% of the time.
global Parent
Parent=parent_class
Parent()
module2.py
from module1 import *
class Child(Parent):
def __init__(self):
print("I'm the child.")
class LeadActor(Actor): #There's not necessarily a need to subclass Actor, but in the situation I'm thinking, it seems it would be a common thing.
def __init__(self):
Actor.__init__(self, parent_class=Child)
a=Actor(parent_class=Child) #prints "I'm the child." instead of "I'm the parent."
l=LeadActor() #prints "I'm the child." instead of "I'm the parent."
Just be careful that the user knows not to set a different value for parent_class with different subclasses of Actor. I mean, if you make multiple kinds of Actors, you'll only want to set parent_class once, unless you want it to change for all of them.

Passing an instance to __init__. Is this a good idea?

Suppose I have a simple class like this:
class Class1(object):
def __init__(self, property):
self.property = property
def method1(self):
pass
An instances of Class1 returns a value that can be used in other class:
class Class2(object):
def __init__(self, instance_of_class1, other_property):
self.other_property = other_property
self.instance_of_class1 = instance_of_class1
def method1(self):
# A method that uses self.instance_of_class1.property and self.other_property
This is working. However, I have the feeling that this is not a very common approach and maybe there are alternatives. Having said this, I tried to refactor my classes to pass simpler objects to Class2, but I found that passing the whole instance as an argument actually simplifies the code significantly. In order to use this, I have to do this:
instance_of_class1 = Class1(property=value)
instance_of_class2 = Class2(instance_of_class1, other_property=other_value)
instance_of_class2.method1()
This is very similar to the way some R packages look like. Is there a more "Pythonic" alternative?

There's nothing wrong with doing that, though in this particular example it looks like you could just as easily do
instance_of_class2 = Class2(instance_of_class1.property, other_property=other_value).
But if you find you need to use other properties/methods of Class1 inside of Class2, just go ahead and pass the whole Class1 instance into Class2. This kind of approach is used all the time in Python and OOP in general. Many common design patterns call for a class to take an instance (or several instances) of other classes: Proxy, Facade, Adapter, etc.

Inheritance vs class instance in python module import

Apologies if this doesn't make sense, i'm not much of an experienced programmer.
Consider the following code:
import mymodule
class MyClass:
def __init__(self):
self.classInstance = myModule.classInstance()
and then ......
from mymodule import classInstance
class MyClass(classInstance):
def __init__(self):
pass
If I just wanted to use the one classInstance in MyClass, is it ok to import the specific class from the module and have MyClass inherit this class ?
Are there any best practices, or things I should be thinking about when deciding between these two methods ?
Many thanks

Allow me to propose a different example.
Imagine to have the class Vector.
Now you want a class Point. Point can be defined with a vector but maybe it has other extra functionalities that Vector doesn't have.
In this case you derive Point from Vector.
Now you need a Line class.
A Line is not a specialisation of any of the above classes so probably you don't want to derive it from any of them.
However Line uses points. In this case you might want to start you Line class this way:
class Line(object):
def __init__(self):
self.point1 = Point()
self.point2 = Point()
Where point will be something like this:
class Point(Vector):
def __init__(self):
Vector.__init__(self)
So the answer is really: Depends what you need to do, but when you have a clear idea of what you are coding, than choosing between sub-classing or not becomes obvious.
I hope it helped.

You make it sound like you're trying to "choose between" those two approaches, but they do completely different things. The second one defines a class that inherits from a class (confusingly) called classInstance. The first one defines a class called MyClass (not inheriting from anything except the base obect type) that has an instance variable called self.classInstance, which happens to be set to an instance of the classInstance class.
Why are you naming your class classInstance?