We Keep Coding

How to organize Python API module to make it neat?

I'm writing a Python library that represents some web API. Right now, my library directory looks close to this:
__init__.py
Account.py
Order.py
Category.py
requests.py
In __init__.py, I have something like this:
from .Account import Account
from .Order import Order
from .Category import Category
from . import requests
This allows to use import cool_site and then cool_site.Account(…) and so on, but it has a following problem: when I play around with my code in IDLE, the object is then called cool_site.Account.Account, which I feel is bad.
1. Is there any way to avoid class name duplication and still have separate file for every class?
The next thing I don't feel great about is my code organization. Right now, my Account class takes credentials on initialization, creates a requests.Session object and then handles all communication with server, i.e. searching for orders and so on. This Account class instance will then pass itself to all other instances, for example to Order - so the order's instance will have .account property holding the Account instance which created it. When another class instance itself has to do something, for example change an order's comment (by calling o.comment = 'new comment', so by #comment.setter decorator in the Order class), it forwards that to an Account object which is passed to it on initialization, and then uses for example self.account.set_order_comment(new_comment). This method will then use all the web requests to achieve that goal.
2. Is it better to hold server communication logic in one class or to spread different aspects of it to classes that are affected by them?
The last thing I'd like to ask about is how and where to keep low-level request templates. Right now I have it in cool_site.requests submodule, and there are different functions for different requests, for example SetOrderComment for the case mentioned above (it's a function, so it should be lowercase, but in this case I think it resembles a class in a way - is that OK?). The Account.set_order_comment will use it like this:
r = cool_site.requests.SetOrderComment(order.id, new_comment)
response = self._session.request(**r)
because this function returns a dict with arguments to Session.request function from requests library. The authentication headers are already set in the _session property of Account class instance. I feel it's a little bit ugly, but I don't have any better idea.
3. How to organize web requests to keep it all clean?
Post scriptum
I'm sorry this question is so long and covers many aspects of API library design, but all the tips will be appreciated. In a way, all of the three questions above could be expressed as "How to do it better and cleaner?" or "How most of the Python developers do it?", or maybe even "What would feel most Pythonic?".
Throw at me any little tips you can think of.

I've been thinking about very similar things lately working on wistiapy. Examples of my current thinking about client code organisation are in there. YMMV.
"One class per file" is more of a Java style guideline than Python. Python modules are a legitimate and important level in the code hierarchy, and you shouldn't worry about having more than one function or class in the same module. You could put all the model classes in a .models module, and then from .models import (Account, Order, Category) in __init__.py.
More-or-less common practice for client libraries seems to be to have a client module, containing something like a MyServiceClient class. (eg the Segment client). This is where the networking logic goes. If you want to have the public interface be module-level functions, you can do some clever stuff with creating a default client and having the functions call methods on that.
Functions should be snake_case, classes should be PascalCase. Doing anything else tends to cause more confusion than benefit.
It seems like the big question you're dealing with is trying to choose between the "Active Record" pattern (some_account.set_order_comment(comment)), and the "Data Mapper" pattern (set_order_comment(account, comment)). Either will work and they each have their benefits and drawbacks. I've found the data mapper pattern -- using smart functions to manipulate fairly simple data classes -- simpler to begin with.
I find it helpful to design the public interface concurrently with something that uses that interface. In the calling code, you can write what you'd like to have to call, and then implement the client code "outside-in".

1) no upper case in names of .py file (also try to avoid _)
so your files should be
__init__.py
account.py
order.py
category.py
requests.py
2) if you want to use like cool_site.Account you need to add to __init__.py
from .account import Account
from .order import Order
from .category import Category
__all__ = [
'Account',
'Order',
'Category',
]
3) SetOrderComment is bad name, use set_order_comment
4) If you write a python wrapper for communication with API, make method that do authorisation and other other stuff that is same in every API request. This method should take as params part of requests kwargs that are different for different API calls
for example
class API:
def __init__(self, endpoint:s str, api_key: str):
self.endpoint = endpoint
self.api_key = api_key
def _get_auth_headers(self) -> Dict[str, str]:
return {
'Authorization': 'Bearer ' + self.api_key,
}
def get(self, path, params)
resp = requester.get(
self.endpoint + path,
headers=self._get_auth_headers(),
params=params,
timeout=30,
)
self._check_api_response(resp)
payload = resp.json()
return payload
5) If you write a python API look at flask and django frameworks and projects with API written on them. You should find some good ides overthere.

I can throw in an easy tip:
You could use __all__ = ["Account"] in your Account.py module and then use from .Account import * in your __init__.pyfile. I believe that addresses your first question.
You can read up on the __all__ magic method here. In short, you can specify what methods etc you would like to import when you use from x import *. You can hide 'private' methods from being imported with _one_leading_underscore.

Test Driven Development with Django

I have a conceptual question about doing test driven development with Django, may also apply to other frameworks as well.
TDD states that the firts step in the development cycle is to write failing tests.
Suppose for a unit test, I want to verify that an item is actually created when a request arrives. To test this functioanlty, I want to issue a request with the test client, and check with the db that this object is actully created. To be able to do that, I need to import the related model in the test file, but as the first step is writing this test, I don't even have a model yet. So I won't be able to run the tests to see them fail.
What is the suggested approach here? Maybe write a simpler test first, then modify the test after enough level of production code is implemented?

Important note: What you describe is not a unit test. It does not test one unit. It tests whole bunch of things starting form django url wiring, views and ending with models. This is integration tests. Secondly, don't create a test that uses external API (or test client which is mostly the same) to create data but checks that entity was created by going directly to DB. This is not good. If you create data via some API you should use the API of the same level to check the data is created. So my explanation will talk about this approach.
What you describe is a common problem when you start with TDD.
Important things about TDD are that you:
do small steps
refactor after test is green (including test refactoring)
This may sound like simple and you most probably have read and know that but the consequences for how you structure you work might not be that obvious.
Main consequence is that you do not write the full test from the scratch before implementing the functionality. You start with simplest test you can do, make it work (by implementing some piece of functionality), refactor. Then you change the test by adding more things that you want to check to it, implement that piece to make test green, refactor and so on.
That has consequence that you need to split the work (or plan how you implement it by simple steps) to be able to work in this mode. This requires some practice and I guess is one the main barriers for TDD adoption.
It is similar (but with important difference) to what you wrote:
Maybe write a simpler test first, then modify the test after enough level of production code is implemented?
You need to have simple test first, then modify it iteratively with small steps but before you implement production code not after.
In this particular case you can implement it in following steps:
1 Create the test that uses test client
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
You have a test that fails but no line of code is written.
Note that no data is passed yet and the check is very basic.
2 Create url wiring and empty view
Do it so that the test pass. You need very few changes in the code and the view will not return much if anything. View can return some hardcoded json/dict at this point.
3.1 Check that id of the entity is generated
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
assert_that(get_result, has_field('id', not_none()))
You can make this test work by adding id to the hardcoded dict.
3.1 Check that unique id of the entity is generated
Add a new test that checks that ids are unique:
def test_create_generates_unique_id(self):
post_result1 = test_client.post(POST_URL, {})
post_result2 = test_client.post(POST_URL, {})
assert_that(get_id(post_result1), not_(equal_to(get_id(post_result2)))
4 Add the model with only id
It is not hard to add a model with only id and add its creation and retrieval from the view. Don't add all the fields that you need, you will do that step by step later.
5 Add one field to you test
def test_entity_creation(self):
post_result = test_client.post(POST_URL, {'field': 'value'})
get_result = test_client.get(get_entity_url_from(post_result))
assert_that(get_result, not_none())
assert_that(get_result, has_field('field', 'value'))
Add a field to the model and make the test pass.
6 Continue doing TDD
Add more tests and production code.
Some more thoughts
Step 4 might be too big as for one TDD cycle. It requires making changes at least to three things:
post view handler
get view handler
model
In many cases it makes sense to split it by first creation a test for the model itself. The test that will not work with test client but will look like this:
def test_entity(self):
entity = Entity.objects.create()
entity = Entity.objects.get(entity.id)
assert_that(entity.id, not_none())
Then you add a model. Make sure that test_entity pass and only after that modify view to use you (already tested) model.
I hope this gives the idea how to approach this problem.

In Django, the approach is always to recreate a working environment for testing and staging. In testing the data is fake, in staging the data is "old" or very similar to the production.

Advice on database access approach in a custom environment using Python Pyramid

I’m new to Pyramid. I’ve used Python for a few months. I've created a Python application on Linux to maintain an Oracle database using weekly data feeds from a vendor. To get that done, one of the things I did was to create a customized database wrapper class using the cx_Oracle package. I had specific requirements for maintaining history in the DB. All Oracle access goes through the methods in this wrapper. I now want to use Pyramid to create a simple reporting browser interface to the Oracle DB. To allow me the greatest flexibility, I’d like to use the wrapper I already have to get to the data on Oracle instead of Alchemy (or possibly with it, I'm not sure).
In my Pyramid app, I’ve considered importing my wrapper in my views.py init method but that seems to get executed with every browser submit.
Can anyone suggest how I might create a persistent connection to Oracle that I can use over and over from my reporting application which uses my wrapper class? I’m finding Pyramid a bit opaque. I’m never sure what’s happening behind the scenes but I’m willing to operate on trust until I get the swing of it. I need the benefit of the automatic authorization/ authentication and login.
What I’m really looking for is a good approach from experienced Pyramid users before going down the wrong track.
Many thanks.

You should definitely use SQLAlchemy as it makes use of connection pooling and such. In your case SQLAlchemy would use cx_Oracle underneath but make you be able to concentrate writing actual code instead of maintaining connections/pooling them/and such.
You should follow the patterns in Wiki2 tutorial to set up basic SQLAlchemy.

So, basically, the question boils down to "how do I use my existing API with Pyramid", right? This is quite easy as Pyramid is database-agnostic and very transparent in this area despite what you say :) Basically, you import it, call its methods, retrieve data and send it to the template:
import pokemon_api as api
#view_config(route_name='list_pokemon', renderer='pokemon_list.mako')
def list_pokemon(request):
# To illustrate how to get data from the request to send to your API
batch_start = request.GET.get('batch_start', 0)
batch_end = batch_start + request.GET.get('batch_size', 0)
sort_order = request.GET.get('sort_by', 'name')
# Here we call our API - we don't actually care where it gets the data from, it's a black box
pokemon = api.retrieve_pokemon(from=batch_start, to=batch_end, sort=sort_order)
# send the data to the renderer/template
return {
'pokemon': pokemon
}
#view_config(route_name='add_pokemon', request_method='POST')
def add_pokemon(request):
"""
Add a new Pokemon
"""
name = request.POST.get('name', 0)
weight = request.GET.get('weight', 0)
hight = request.GET.get('hight', 0)
api.create(name=name, weight=weight, height=height)
# go back to the listing
return HTTPFound('/pokemon_list')
and if your API needs some initialization, you can do it at startup time
import pokemon_api as api
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
config = Configurator(settings=settings)
...
api.init("MAGIC_CONNECTION_STRING")
return config.make_wsgi_app()
Of course, this assumes your API already handles transactions, connections, pooling and other boring stuff :)
One last point to mention - in Python you generally don't import things inside methods, you import them at the top of the file at module-level scope. There are exceptions for that rule, but I don't see why you would need that in this case. Also, importing a module should be free of side-effects (which you might have since "importing my wrapper in my views.py __init__ method" seems to be causing trouble).

How do I properly unit test a Django session?

The behavior of Django sessions changes between "standard" views code and test code, making it unclear how test code is written for sessions. Googling this yields two relevant discussions about this issue:
Easier manipulation of sessions by
test client
test.Client.session.save() raises
error for anonymous users
I'm confused because both tickets have different ways of dealing with this problem and they were both Accepted. I assume this means they were patched and the behavior is now different. I also don't know to which versions these patches would pertain.
If I'm writing a unit test in Django 1.0, how would I set up my session store for sessions to work as they do in the browser?

I don't quite understand what do you mean by saying the behavior changes between "standard" view and "test" code, maybe you should elaborate on that.
but regarding how to test the session, I do think there are approaches.
you have to understand how django session works, read the unit test for the session package you used in your application. this is regarding understand how server side works.
you probably need to capture a few conversations between browser and server( using FIREBUG for example )
so the issue for you looks like that you are not passing session_id you get when you log in back to server when you talk to server. like put it in (POST,GET,COOKIES I don't quite remember that ).
The important thing here is understand how session works in HTTP, once you get that, you definitely have a clear idea about what is happening there, and make explainations accordingly.

Design pattern to organize non-trivial ORM queries?

I am developing a web API with 10 tables or so in the backend, with several one-to-many and many-to-many associations. The API essentially is a database wrapper that performs validated updates and conditional queries. It's written in Python, and I use SQLAlchemy for ORM and CherryPy for HTTP handling.
So far I have separated the 30-some queries the API performs into functions of their own, which look like this:
# in module "services.inventory"
def find_inventories(session, user_id, *inventory_ids, **kwargs):
query = session.query(Inventory, Product)
query = query.filter_by(user_id=user_id, deleted=False)
...
return query.all()
def find_inventories_by(session, app_id, user_id, by_app_id, by_type, limit, page):
....
# in another service module
def remove_old_goodie(session, app_id, user_id):
try:
old = _current_goodie(session, app_id, user_id)
services.inventory._remove(session, app_id, user_id, [old.id])
except ServiceException, e:
# log it and do stuff
....
The CherryPy request handler calls the query methods, which are scattered across several service modules, as needed. The rationale behind this solution is, since they need to access multiple model classes, they don't belong to individual models, and also these database queries should be separated out from direct handling of API accesses.
I realize that the above code might be called Foreign Methods in the realm of refactoring. I could well live with this way of organizing for a while, but as things are starting to look a little messy, I'm looking for a way to refactor this code.
Since the queries are tied directly to the API and its business logic, they are hard to generalize like getters and setters.
It smells to repeat the session argument like that, but as the current implementation of the API creates a new CherryPy handler instance for each API call and therefore the session object, there is no global way of getting at the current session.
Is there a well-established pattern to organize such queries? Should I stick with the Foreign Methods and just try to unify the function signature (argument ordering, naming conventions etc.)? What would you suggest?

The standard way to have global access to the current session in a threaded environment is ScopedSession. There are some important aspects to get right when integrating with your framework, mainly transaction control and clearing out sessions between requests. A common pattern is to have an autocommit=False (the default) ScopedSession in a module and wrap any business logic execution in a try-catch clause that rolls back in case of exception and commits if the method succeeded, then finally calls Session.remove(). The business logic would then import the Session object into global scope and use it like a regular session.
There seems to be an existing CherryPy-SQLAlchemy integration module, but as I'm not too familiar with CherryPy, I can't comment on its quality.
Having queries encapsulated as functions is just fine. Not everything needs to be in a class. If they get too numerous just split into separate modules by topic.
What I have found useful is too factor out common criteria fragments. They usually fit rather well as classmethods on model classes. Aside from increasing readability and reducing duplication, they work as implementation hiding abstractions up to some extent, making refactoring the database less painful. (Example: instead of (Foo.valid_from <= func.current_timestamp()) & (Foo.valid_until > func.current_timestamp()) you'd have Foo.is_valid())

SQLAlchemy strongly suggests that the session maker be part of some global configuration.
It is intended that the sessionmaker()
function be called within the global
scope of an application, and the
returned class be made available to
the rest of the application as the
single class used to instantiate
sessions.
Queries which are in separate modules isn't an interesting problem. The Django ORM works this way. A web site usually consists of multiple Django "applications", which sounds like your site that has many "service modules".
Knitting together multiple services is the point of an application. There aren't a lot of alternatives that are better.