We Keep Coding

Where should I put abstract classes in a python package?

I am adding abstract classes to my python package like this:
class AbstractClass(ABC):
#abstractmethod
def do_something(self):
pass
There will be multiple subclasses that inherit from AbstractClass like this:
class SubClass(AbstractClass):
def do_something(self):
pass
I am wondering if there are any conventions for python packages regarding abstract classes that I am unaware of.
should the abstract classes be separated from the subclasses, or should they all be in the same directory?
what about naming of abstract functions, any conventions there?
I realize this is fairly subjective. I am asking for opinions, not what is 'right' or 'wrong'
Thanks!

No convention comes to mind. If this is all for just one project, I'd be looking to put the abstract class in with the concrete subclasses at whatever level they're at. If all the subclasses were in one file, then I'd put the abstract class in that file too. If all the subclasses were in individual files in a dir, I'd put the abstract class right next to them in its own file. The reason I'd put the abstract class somewhere else is if I'm separating off utility code that can be reused with other projects. In short, an abstract class is just a piece of code. Just another class. Treat it like any other.
As far as naming, the natural naming of the class given what it is/does is usually enough. If your abstract class were Animal, and your subclasses were Goat, Horse, etc., I'd see no reason to call your abstract class AbstractAnimal, as it's pretty clear already what's going on...that you wouldn't instantiate Animal directly. Also, if you're looking at a class thinking of reusing it, and you see an abstract method in it, then you know it's abstract.

Depending on your style guide there might be. The only thing in my opinion is that any abstract functions should raise errors if not implemented, and should be part of the abstract method docstring.
For example if you're using numpystyle docstrings something like this is ideal:
class AbstractClass(ABC):
#abstractmethod
def do_something(self):
"""Does something
Raises
------
NotImplementedError:
Error is raised if function is not implemented in subclass
"""
raise NotImplementedError

How to create a mock that behaves like sub-classes from abstract class

I'm trying to create mocks from scratch that can pass the test issubclass(class_mock, base_class) where the base class is an abstract class derived from abc.ABC. Before you ask the question, I will answer why I'm trying to do it.
I have an internal package containing a base class and a collection of sub-classes that properly implement the abstract interface. Besides, I have a factory class that can instantiate the sub-classes. The factory is built is such a way that it can inspect its own package and have access to the existing sub-classes. The factory is meant to be always in the same package as the derived and base class (constraint). I think you guessed that I'm actually testing the factory... However, since the sub-classes can change in number, their name or their package name, etc., I cannot implement a correct unit test that directly refers to the actual cub-classes (because it introduces a coupling) and I need mocks.
The problem is that I didn't succeed to create a mock that satisfies the above conditions for a class derived from an abstract class. What I was able to achieve is for a class derived from another non-abstract class.
Here is the code that illustrates the problem more concretely.
import unittest.mock
import inspect
import abc
class A:
pass
class B(A):
pass
class TestSubClass(unittest.TestCase):
def test_sub_class(self):
b_class_mock = self._create_class_mock("B", A)
print(isinstance(b_class_mock, type))
print(inspect.isclass(b_class_mock))
print(issubclass(b_class_mock, A))
#staticmethod
def _create_class_mock(mock_name, base_class):
class_mock = unittest.mock.MagicMock(spec=type(base_class), name=mock_name)
class_mock.__bases__ = (base_class,)
return class_mock
So, for this code, everything is ok. It prints 3 True as wanted.
But as long as the class A is defined as abstract (class A(abc.ABC)), the last test is failing with an error saying that the mock is not a class even if the 2 previous tests are saying the opposite.
I dived a bit into the implementation of abc.ABCMeta and found out that __subclasscheck__ is overridden. I tried to know the process behind it but when I reached the C code and everything became a way more complicated, I tried to rather track when the error message is generated. Unfortunately, I didn't succeed to understand why it is actually not working.

What' the meaning of the brackets in the class?

In python, when I read others' code, I meet this situation where a class is defined and after it there is a pair of brackets.
class AStarFoodSearchAgent(SearchAgent):
def __init__():
#....
I don't know what is the meaning of '(SearchAgent)',because what I usually meet and use doesn't seem that.

It indicates that AStarFoodSearchAgent is a subclass of SearchAgent. It's part of a concept called inheritance.
What is inheritance?
Here's an example. You might have a Car class, and a RaceCar class. When implementing the RaceCar class, you may find that it has a lot of behavior that is very similar, or exactly the same, as a Car. In that case, you'd make RaceCar a subclass ofCar`.
class Car(object):
#Car is a subclass of Python's base objeect. The reasons for this, and the reasons why you
#see some classes without (object) or any other class between brackets is beyond the scope
#of this answer.
def get_number_of_wheels(self):
return 4
def get_engine(self):
return CarEngine(fuel=30)
class RaceCar(Car):
#Racecar is a subclass of Car
def get_engine(self):
return RaceCarEngine(fuel=50)
my_car = Car() #create a new Car instance
desired_car = RaceCar() #create a new RaceCar instance.
my_car.get_engine() #returns a CarEngine instance
desired_car.get_engine() #returns a RaceCarEngine instance
my_car.get_number_of_wheels() #returns 4.
desired_car.get_number_of_wheels() # also returns 4! WHAT?!?!?!
We didn't define get_number_of_wheels on RaceCar, and still, it exists, and returns 4 when called. That's because RaceCar has inherited get_number_of_wheels from Car. Inheritance is a very nice way to reuse functionality from other classes, and override or add only the functionality that needs to be different.
Your Example
In your example, AStarFoodSearchAgent is a subclass of SearchAgent. This means that it inherits some functionality from SearchAgemt. For instance, SearchAgent might implement a method called get_neighbouring_locations(), that returns all the locations reachable from the agent's current location. It's not necessary to reimplement this, just to make an A* agent.
What's also nice about this, is that you can use this when you expect a certain type of object, but you don't care about the implementation. For instance, a find_food function may expect a SearchAgent object, but it wouldn't care about how it searches. You might have an AStarFoodSearchAgent and a DijkstraFoodSearchAgent. As long as both of them inherit from SearchAgent, find_food can use ìsinstanceto check that the searcher it expects behaves like aSearchAgent. Thefind_food`function might look like this:
def find_food(searcher):
if not isinstance(searcher, SearchAgent):
raise ValueError("searcher must be a SearchAgent instance.")
food = searcher.find_food()
if not food:
raise Exception("No, food. We'll starve!")
if food.type == "sprouts":
raise Exception("Sprouts, Yuk!)
return food
Old/Classic Style Classes
Upto Python 2.1, old-style classes were the only type that existed. Unless they were a subclass of some other class, they wouldn't have any parenthesis after the class name.
class OldStyleCar:
...
New style classes always inherit from something. If you don't want to inherit from any other class, you inherit from object.
class NewStyleCar(object):
...
New style classes unify python types and classes. For instance, the type of 1, which you can obtain by calling type(1) is int, but the type of OldStyleClass() is instance, with new style classes, type(NewStyleCar) is Car.

SearchAgent is the superclass of the class AStarFoodSearchAgent. This basically means that an AStarFoodSearchAgent is a special kind of SearchAgent.

It means that class AStarFoodSearchAgent extends SearchAgent.
Check section 9.5 here
https://docs.python.org/2/tutorial/classes.html

This is inheritance in python, just like in any other OO language
https://docs.python.org/2/tutorial/classes.html#inheritance

It means that SearchAgent is a base class of AStarFoodSearchAgent. In other word, AStarFoodSearchAgent inherits from SearchAgent class.
See Inheritance - Python tutorial.

pick a subclass based on a parameter

I have a module (db.py) which loads data from different database types (sqlite,mysql etc..) the module contains a class db_loader and subclasses (sqlite_loader,mysql_loader) which inherit from it.
The type of database being used is in a separate params file,
How does the user get the right object back?
i.e how do I do:
loader = db.loader()
Do I use a method called loader in the db.py module or is there a more elegant way whereby a class can pick its own subclass based on a parameter? Is there a standard way to do this kind of thing?

Sounds like you want the Factory Pattern. You define a factory method (either in your module, or perhaps in a common parent class for all the objects it can produce) that you pass the parameter to, and it will return an instance of the correct class. In python the problem is a bit simpler than perhaps some of the details on the wikipedia article as your types are dynamic.
class Animal(object):
#staticmethod
def get_animal_which_makes_noise(noise):
if noise == 'meow':
return Cat()
elif noise == 'woof':
return Dog()
class Cat(Animal):
...
class Dog(Animal):
...

is there a more elegant way whereby a class can pick its own subclass based on a parameter?
You can do this by overriding your base class's __new__ method. This will allow you to simply go loader = db_loader(db_type) and loader will magically be the correct subclass for the database type. This solution is mildly more complicated than the other answers, but IMHO it is surely the most elegant.
In its simplest form:
class Parent():
def __new__(cls, feature):
subclass_map = {subclass.feature: subclass for subclass in cls.__subclasses__()}
subclass = subclass_map[feature]
instance = super(Parent, subclass).__new__(subclass)
return instance
class Child1(Parent):
feature = 1
class Child2(Parent):
feature = 2
type(Parent(1)) # <class '__main__.Child1'>
type(Parent(2)) # <class '__main__.Child2'>
(Note that as long as __new__ returns an instance of cls, the instance's __init__ method will automatically be called for you.)
This simple version has issues though and would need to be expanded upon and tailored to fit your desired behaviour. Most notably, this is something you'd probably want to address:
Parent(3) # KeyError
Child1(1) # KeyError
So I'd recommend either adding cls to subclass_map or using it as the default, like so subclass_map.get(feature, cls). If your base class isn't meant to be instantiated -- maybe it even has abstract methods? -- then I'd recommend giving Parent the metaclass abc.ABCMeta.
If you have grandchild classes too, then I'd recommend putting the gathering of subclasses into a recursive class method that follows each lineage to the end, adding all descendants.
This solution is more beautiful than the factory method pattern IMHO. And unlike some of the other answers, it's self-maintaining because the list of subclasses is created dynamically, instead of being kept in a hardcoded mapping. And this will only instantiate subclasses, unlike one of the other answers, which would instantiate anything in the global namespace matching the given parameter.

I'd store the name of the subclass in the params file, and have a factory method that would instantiate the class given its name:
class loader(object):
#staticmethod
def get_loader(name):
return globals()[name]()
class sqlite_loader(loader): pass
class mysql_loader(loader): pass
print type(loader.get_loader('sqlite_loader'))
print type(loader.get_loader('mysql_loader'))

Store the classes in a dict, instantiate the correct one based on your param:
db_loaders = dict(sqlite=sqlite_loader, mysql=mysql_loader)
loader = db_loaders.get(db_type, default_loader)()
where db_type is the paramter you are switching on, and sqlite_loader and mysql_loader are the "loader" classes.

Difference between abstract class and interface in Python

What is the difference between abstract class and interface in Python?

What you'll see sometimes is the following:
class Abstract1:
"""Some description that tells you it's abstract,
often listing the methods you're expected to supply."""
def aMethod(self):
raise NotImplementedError("Should have implemented this")
Because Python doesn't have (and doesn't need) a formal Interface contract, the Java-style distinction between abstraction and interface doesn't exist. If someone goes through the effort to define a formal interface, it will also be an abstract class. The only differences would be in the stated intent in the docstring.
And the difference between abstract and interface is a hairsplitting thing when you have duck typing.
Java uses interfaces because it doesn't have multiple inheritance.
Because Python has multiple inheritance, you may also see something like this
class SomeAbstraction:
pass # lots of stuff - but missing something
class Mixin1:
def something(self):
pass # one implementation
class Mixin2:
def something(self):
pass # another
class Concrete1(SomeAbstraction, Mixin1):
pass
class Concrete2(SomeAbstraction, Mixin2):
pass
This uses a kind of abstract superclass with mixins to create concrete subclasses that are disjoint.

What is the difference between abstract class and interface in Python?
An interface, for an object, is a set of methods and attributes on that object.
In Python, we can use an abstract base class to define and enforce an interface.
Using an Abstract Base Class
For example, say we want to use one of the abstract base classes from the collections module:
import collections
class MySet(collections.Set):
pass
If we try to use it, we get an TypeError because the class we created does not support the expected behavior of sets:
>>> MySet()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MySet with abstract methods
__contains__, __iter__, __len__
So we are required to implement at least __contains__, __iter__, and __len__. Let's use this implementation example from the documentation:
class ListBasedSet(collections.Set):
"""Alternate set implementation favoring space over speed
and not requiring the set elements to be hashable.
"""
def __init__(self, iterable):
self.elements = lst = []
for value in iterable:
if value not in lst:
lst.append(value)
def __iter__(self):
return iter(self.elements)
def __contains__(self, value):
return value in self.elements
def __len__(self):
return len(self.elements)
s1 = ListBasedSet('abcdef')
s2 = ListBasedSet('defghi')
overlap = s1 & s2
Implementation: Creating an Abstract Base Class
We can create our own Abstract Base Class by setting the metaclass to abc.ABCMeta and using the abc.abstractmethod decorator on relevant methods. The metaclass will be add the decorated functions to the __abstractmethods__ attribute, preventing instantiation until those are defined.
import abc
For example, "effable" is defined as something that can be expressed in words. Say we wanted to define an abstract base class that is effable, in Python 2:
class Effable(object):
__metaclass__ = abc.ABCMeta
#abc.abstractmethod
def __str__(self):
raise NotImplementedError('users must define __str__ to use this base class')
Or in Python 3, with the slight change in metaclass declaration:
class Effable(object, metaclass=abc.ABCMeta):
#abc.abstractmethod
def __str__(self):
raise NotImplementedError('users must define __str__ to use this base class')
Now if we try to create an effable object without implementing the interface:
class MyEffable(Effable):
pass
and attempt to instantiate it:
>>> MyEffable()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: Can't instantiate abstract class MyEffable with abstract methods __str__
We are told that we haven't finished the job.
Now if we comply by providing the expected interface:
class MyEffable(Effable):
def __str__(self):
return 'expressable!'
we are then able to use the concrete version of the class derived from the abstract one:
>>> me = MyEffable()
>>> print(me)
expressable!
There are other things we could do with this, like register virtual subclasses that already implement these interfaces, but I think that is beyond the scope of this question. The other methods demonstrated here would have to adapt this method using the abc module to do so, however.
Conclusion
We have demonstrated that the creation of an Abstract Base Class defines interfaces for custom objects in Python.

Python >= 2.6 has Abstract Base Classes.
Abstract Base Classes (abbreviated
ABCs) complement duck-typing by
providing a way to define interfaces
when other techniques like hasattr()
would be clumsy. Python comes with
many builtin ABCs for data structures
(in the collections module), numbers
(in the numbers module), and streams
(in the io module). You can create
your own ABC with the abc module.
There is also the Zope Interface module, which is used by projects outside of zope, like twisted. I'm not really familiar with it, but there's a wiki page here that might help.
In general, you don't need the concept of abstract classes, or interfaces in python (edited - see S.Lott's answer for details).

In a more basic way to explain:
An interface is sort of like an empty muffin pan.
It's a class file with a set of method definitions that have no code.
An abstract class is the same thing, but not all functions need to be empty. Some can have code. It's not strictly empty.
Why differentiate:
There's not much practical difference in Python, but on the planning level for a large project, it could be more common to talk about interfaces, since there's no code. Especially if you're working with Java programmers who are accustomed to the term.

Python doesn't really have either concept.
It uses duck typing, which removed the need for interfaces (at least for the computer :-))
Python <= 2.5:
Base classes obviously exist, but there is no explicit way to mark a method as 'pure virtual', so the class isn't really abstract.
Python >= 2.6:
Abstract base classes do exist (http://docs.python.org/library/abc.html). And allow you to specify methods that must be implemented in subclasses. I don't much like the syntax, but the feature is there. Most of the time it's probably better to use duck typing from the 'using' client side.

In general, interfaces are used only in languages that use the single-inheritance class model. In these single-inheritance languages, interfaces are typically used if any class could use a particular method or set of methods. Also in these single-inheritance languages, abstract classes are used to either have defined class variables in addition to none or more methods, or to exploit the single-inheritance model to limit the range of classes that could use a set of methods.
Languages that support the multiple-inheritance model tend to use only classes or abstract base classes and not interfaces. Since Python supports multiple inheritance, it does not use interfaces and you would want to use base classes or abstract base classes.
http://docs.python.org/library/abc.html

Abstract classes are classes that contain one or more abstract methods. Along with abstract methods, Abstract classes can have static, class and instance methods.
But in case of interface, it will only have abstract methods not other. Hence it is not compulsory to inherit abstract class but it is compulsory to inherit interface.

For completeness, we should mention PEP3119
where ABC was introduced and compared with interfaces,
and original Talin's comment.
The abstract class is not perfect interface:
belongs to the inheritance hierarchy
is mutable
But if you consider writing it your own way:
def some_function(self):
raise NotImplementedError()
interface = type(
'your_interface', (object,),
{'extra_func': some_function,
'__slots__': ['extra_func', ...]
...
'__instancecheck__': your_instance_checker,
'__subclasscheck__': your_subclass_checker
...
}
)
ok, rather as a class
or as a metaclass
and fighting with python to achieve the immutable object
and doing refactoring
...
you'll quite fast realize that you're inventing the wheel
to eventually achieve
abc.ABCMeta
abc.ABCMeta was proposed as a useful addition of the missing interface functionality,
and that's fair enough in a language like python.
Certainly, it was able to be enhanced better whilst writing version 3, and adding new syntax and immutable interface concept ...
Conclusion:
The abc.ABCMeta IS "pythonic" interface in python