We Keep Coding

How to organize Python API module to make it neat?

I'm writing a Python library that represents some web API. Right now, my library directory looks close to this:
__init__.py
Account.py
Order.py
Category.py
requests.py
In __init__.py, I have something like this:
from .Account import Account
from .Order import Order
from .Category import Category
from . import requests
This allows to use import cool_site and then cool_site.Account(…) and so on, but it has a following problem: when I play around with my code in IDLE, the object is then called cool_site.Account.Account, which I feel is bad.
1. Is there any way to avoid class name duplication and still have separate file for every class?
The next thing I don't feel great about is my code organization. Right now, my Account class takes credentials on initialization, creates a requests.Session object and then handles all communication with server, i.e. searching for orders and so on. This Account class instance will then pass itself to all other instances, for example to Order - so the order's instance will have .account property holding the Account instance which created it. When another class instance itself has to do something, for example change an order's comment (by calling o.comment = 'new comment', so by #comment.setter decorator in the Order class), it forwards that to an Account object which is passed to it on initialization, and then uses for example self.account.set_order_comment(new_comment). This method will then use all the web requests to achieve that goal.
2. Is it better to hold server communication logic in one class or to spread different aspects of it to classes that are affected by them?
The last thing I'd like to ask about is how and where to keep low-level request templates. Right now I have it in cool_site.requests submodule, and there are different functions for different requests, for example SetOrderComment for the case mentioned above (it's a function, so it should be lowercase, but in this case I think it resembles a class in a way - is that OK?). The Account.set_order_comment will use it like this:
r = cool_site.requests.SetOrderComment(order.id, new_comment)
response = self._session.request(**r)
because this function returns a dict with arguments to Session.request function from requests library. The authentication headers are already set in the _session property of Account class instance. I feel it's a little bit ugly, but I don't have any better idea.
3. How to organize web requests to keep it all clean?
Post scriptum
I'm sorry this question is so long and covers many aspects of API library design, but all the tips will be appreciated. In a way, all of the three questions above could be expressed as "How to do it better and cleaner?" or "How most of the Python developers do it?", or maybe even "What would feel most Pythonic?".
Throw at me any little tips you can think of.

I've been thinking about very similar things lately working on wistiapy. Examples of my current thinking about client code organisation are in there. YMMV.
"One class per file" is more of a Java style guideline than Python. Python modules are a legitimate and important level in the code hierarchy, and you shouldn't worry about having more than one function or class in the same module. You could put all the model classes in a .models module, and then from .models import (Account, Order, Category) in __init__.py.
More-or-less common practice for client libraries seems to be to have a client module, containing something like a MyServiceClient class. (eg the Segment client). This is where the networking logic goes. If you want to have the public interface be module-level functions, you can do some clever stuff with creating a default client and having the functions call methods on that.
Functions should be snake_case, classes should be PascalCase. Doing anything else tends to cause more confusion than benefit.
It seems like the big question you're dealing with is trying to choose between the "Active Record" pattern (some_account.set_order_comment(comment)), and the "Data Mapper" pattern (set_order_comment(account, comment)). Either will work and they each have their benefits and drawbacks. I've found the data mapper pattern -- using smart functions to manipulate fairly simple data classes -- simpler to begin with.
I find it helpful to design the public interface concurrently with something that uses that interface. In the calling code, you can write what you'd like to have to call, and then implement the client code "outside-in".

1) no upper case in names of .py file (also try to avoid _)
so your files should be
__init__.py
account.py
order.py
category.py
requests.py
2) if you want to use like cool_site.Account you need to add to __init__.py
from .account import Account
from .order import Order
from .category import Category
__all__ = [
'Account',
'Order',
'Category',
]
3) SetOrderComment is bad name, use set_order_comment
4) If you write a python wrapper for communication with API, make method that do authorisation and other other stuff that is same in every API request. This method should take as params part of requests kwargs that are different for different API calls
for example
class API:
def __init__(self, endpoint:s str, api_key: str):
self.endpoint = endpoint
self.api_key = api_key
def _get_auth_headers(self) -> Dict[str, str]:
return {
'Authorization': 'Bearer ' + self.api_key,
}
def get(self, path, params)
resp = requester.get(
self.endpoint + path,
headers=self._get_auth_headers(),
params=params,
timeout=30,
)
self._check_api_response(resp)
payload = resp.json()
return payload
5) If you write a python API look at flask and django frameworks and projects with API written on them. You should find some good ides overthere.

I can throw in an easy tip:
You could use __all__ = ["Account"] in your Account.py module and then use from .Account import * in your __init__.pyfile. I believe that addresses your first question.
You can read up on the __all__ magic method here. In short, you can specify what methods etc you would like to import when you use from x import *. You can hide 'private' methods from being imported with _one_leading_underscore.

Do we need to join in the controller or model when you use SQL Alchemy?

In MVC frameworks, where do you usually embed SQL Alchemy Code, is it ideal to put the query in Controller Methods or just use the Model Methods?
query = session.query(User, Document, DocumentsPermissions).join(Document).join(DocumentsPermissions).filter_by(Document.name=="Something")
Or I just delegate this to a Model Method which takes a args? What is the preferred way to do this? One of the benefits of the latter is that it can be re-used and it almost presents a view for the API programmers. Another advantage is that I can easily over-ride this if I make it a class method. This is usually helpful in customizations especially in commercial softwares.
#Ctrl.py
self.getdocument("Foo")
#Mdl.py
def getdocument(name):
query = session.query(User, Document, DocumentsPermissions).join(Document).join(DocumentsPermissions).filter_by(Document.name=="Something")
TL;DR: Isn't the concept of "M" in MVC blurred when you use ORM's like SQL Alchemy? I didn't have any problems with Model View Controller design patterns.
[PS: I am not sure if this belongs to Code Review Site, if so please let me know, I can just transfer over.]

I strongly prefer the second approach. It has a few advantages:
Your controller code can be dumb. This is good. Controllers that just fetch data from the backend, possibly reformat it a little bit, and pass it on to views are very easy to reason about.
It's easier to test that method in isolation. You can run getdocument('valid_name'), getdocument(None), getdocument(123), etc. to ensure they all work or fail as expected without dealing with all the surrounding controller code.
It's easier to test the controller. You can write a mock for getdocument() so that it always returns a known value and test that your controller processes it correctly.

I tend to put database query code in the Controller rather than the Model. As my understanding goes, Model methods are used to transform the data of the model into something else.
For example, a UserModel may have a FullName() method to return the concatenation of the user's first and last names.
Whereas, a UserController contains a GetAll() method to get a list of all users, which is where the database query code is found.

How to replace template method pattern with functional style?

You can see the code here
The concrete problem that I'm trying to solve is this.
Say that I need to provide REST interface to some entities modeled (sqlalchemy in my case) with some tool stored in database. Say that this collection is called parents.
I would need handlers like this
GET /parents
GET /parents/some_id
POST /parents
DELETE /parents/some_id
Conceptually all this handlers are very similar.
They all take ids from url, then create appropriate query. Then they fetching data with that query, then turn this data into dict and then call jsonify to create correct http response.
So with OOP I could design this like that.
class AbstractHandler():
def __init__(serializer, **specs):
self.specs = specs
def handle_request(self, spec_data, *_ids):
q = self.create_query(_ids)
d = self.fetch_data(self.specs[spec_data['name']](**(spec_data['args'] + (query, ))
out = serializer(d)
return jsonify(out)
The spec is a function that takes some parameters and query and produce more refined query based of this parameters.
So for example
GET /parents?spec={'name': 'by_name', 'args': ['adam'}
would return parent named Adam from collection.
This code has some flaws but I hope you see the point how template method makes flow of control here and subclasses can change how they would create query, how they would fetch data (item handler would need to call query.one() and collection handler would need to call query.all() for example)
So I can replace create_query, fetch_data with dependency injection instead. But that would create a problem that someone could create wrong configuration by giving the wrong dependency. That's basically what I've done, but with using partial functions instead.
So what I'm thinking right now is that I can solve this problem by creating factory functions for every type of handler that I need, that would give appropriate dependency to handler.
That's very much alike with template method solution I think. The difference basically is that in template method correctness dependencies is guarantied by object type and in my solution it's guarantied by type of factory function.
So enough with what I think, I'd like to know what you think about that?
How people in functional world solve this kinds of problem?

How to setup a 3-tier web application project

EDIT:
I have added [MVC] and [design-patterns] tags to expand the audience for this question as it is more of a generic programming question than something that has direclty to do with Python or SQLalchemy. It applies to all applications with business logic and an ORM.
The basic question is if it is better to keep business logic in separate modules, or to add it to the classes that our ORM provides:
We have a flask/sqlalchemy project for which we have to setup a structure to work in. There are two valid opinions on how to set things up, and before the project really starts taking off we would like to make our minds up on one of them.
If any of you could give us some insights on which of the two would make more sense and why, and what the advantages/disadvantages would be, it would be greatly appreciated.
My example is an HTML letter that needs to be sent in bulk and/or displayed to a single user. The letter can have sections that display an invoice and/or a list of articles for the user it is addressed to.
Method 1:
Split the code into 3 tiers - 1st tier: web interface, 2nd tier: processing of the letter, 3rd tier: the models from the ORM (sqlalchemy).
The website will call a server side method in a class in the 2nd tier, the 2nd tier will loop through the users that need to get this letter and it will have internal methods that generate the HTML and replace some generic fields in the letter, with information for the current user. It also has internal methods to generate an invoice or a list of articles to be placed in the letter.
In this method, the 3rd tier is only used for fetching data from the database and perhaps some database related logic like generating a full name from a users' first name and last name. The 2nd tier performs most of the work.
Method 2:
Split the code into the same three tiers, but only perform the loop through the collection of users in the 2nd tier.
The methods for generating HTML, invoices and lists of articles are all added as methods to the model definitions in tier 3 that the ORM provides. The 2nd tier performs the loop, but the actual functionality is enclosed in the model classes in the 3rd tier.
We concluded that both methods could work, and both have pros and cons:
Method 1:
separates business logic completely from database access
prevents that importing an ORM model also imports a lot of methods/functionality that we might not need, also keeps the code for the model classes more compact.
might be easier to use when mocking out ORM models for testing
Method 2:
seems to be in line with the way Django does things in Python
allows simple access to methods: when a model instance is present, any function it
performs can be immediately called. (in my example: when I have a letter-instance available, I can directly call a method on it that generates the HTML for that letter)
you can pass instances around, having all appropriate methods at hand.

Normally, you use the MVC pattern for this kind of stuff, but most web frameworks in python have dropped the "Controller" part for since they believe that it is an unnecessary component. In my development I have realized, that this is somewhat true: I can live without it. That would leave you with two layers: The view and the model.
The question is where to put business logic now. In a practical sense, there are two ways of doing this, at least two ways in which I am confrontet with where to put logic:
Create special internal view methods that handle logic, that might be needed in more than one view, e.g. _process_list_data
Create functions that are related to a model, but not directly tied to a single instance inside a corresponding model module, e.g. check_login.
To elaborate: I use the first one for strictly display-related methods, i.e. they are somehow concerned with processing data for displaying purposes. My above example, _process_list_data lives inside a view class (which groups methods by purpose), but could also be a normal function in a module. It recieves some parameters, e.g. the data list and somehow formats it (for example it may add additional view parameters so the template can have less logic). It then returns the data set to the original view function which can either pass it along or process it further.
The second one is used for most other logic which I like to keep out of my direct view code for easier testing. My example of check_login does this: It is a function that is not directly tied to display output as its purpose is to check the users login credentials and decide to either return a user or report a login failure (by throwing an exception, return False or returning None). However, this functionality is not directly tied to a model either, so it cannot live inside an ORM class (well it could be a staticmethod for the User object). Instead it is just a function inside a module (remember, this is Python, you should use the simplest approach available, and functions are there for something)
To sum this up: Display logic in the view, all the other stuff in the model, since most logic is somehow tied to specific models. And if it is not, create a new module or package just for logic of this kind. This could be a separate module or even a package. For example, I often create a util module/package for helper functions, that are not directly tied for any view, model or else, for example a function to format dates that is called from the template but contains so much python could it would be ugly being defined inside a template.
Now we bring this logic to your task: Processing/Creation of letters. Since I don't know exactly what processing needs to be done, I can only give general recommendations based on my assumptions.
Let's say you have some data and want to bring it into a letter. So for example you have a list of articles and a costumer who bought these articles. In that case, you already have the data. The only thing that may need to be done before passing it to the template is reformatting it in such a way that the template can easily use it. For example it may be desired to order the purchased articles, for example by the amount, the price or the article number. This is something that is independent of the model, the order is now only display related (you could have specified the order already in your database query, but let's assume you didn't). In this case, this is an operation your view would do, so your template has the data ready formatted to be displayed.
Now let's say you want to get the data to create a specifc letter, for example a list of articles the user bough over time, together with the date when they were bought and other details. This would be the model's job, e.g. create a query, fetch the data and make sure it is has all the properties required for this specifc task.
Let's say in both cases you with to retrieve a price for the product and that price is determined by a base value and some percentages based on other properties: This would make sense as a model method, as it operates on a single product or order instance. You would then pass the model to the template and call the price method inside it. But you might as well reformat it in such a way, that the call is made already in the view and the template only gets tuples or dictionaries. This would make it easier to pass the same data out as an API (see below) but it might not necessarily be the easiest/best way.
A good rule for this decision is to ask yourself If I were to provide a JSON API additionally to my standard view, how would I need to modify my code to be as DRY as possible?. If theoretical is not enough at the start, build some APIs for the templates and see where you need to change things to the API makes sense next to the views themselves. You may never use this API and so it does not need to be perfect, but it can help you figure out how to structure your code. However, as you saw above, this doesn't necessarily mean that you should do preprocessing of the data in such a way that you only return things that can be turned into JSON, instead you might want to make some JSON specifc formatting for the API view.
So I went on a little longer than I intended, but I wanted to provide some examples to you because that is what I missed when I started and found out those things via trial and error.

Dynamically add URL rules to Flask app

I am writing an app in which users will be able to store information that they can specify a REST interface for. IE, store a list of products at /<username>/rest/products. Since the URLs are obviously not known before hand, I was trying to think of the best way to implement dynamic URL creation in Flask. The first way I thought of would be to write a catch-all rule, and route the URL from there. But then I am basically duplicating URL routing capabilities when Flask already has them built-in. So, I was wondering if it would be a bad idea to use .add_url_rule() (docs here, scroll down a bit) to attach them directly to the app. Is there a specific reason this shouldn't be done?

Every time you execute add_url_rule() the internal routing remaps the URL map. This is neither threadsafe nor fast. I right now don't understand why you need user specific URL rules to be honest. It kinda sounds like you actually want user specific applications mounted?
Maybe this is helpful: http://flask.pocoo.org/docs/patterns/appdispatch/

I have had similar requirement for my application where each endpoint /<SOMEID>/rest/other for given SOMEID should be bounded to a different function. One way to achieve this is keeping a lookup dictionary where values are the function that handle the specific SOMEID. For example take a look at this snippet:
func_look_up_dict = {...}
#app.route('<SOMEID>/rest/other', methods=['GET'])
def multiple_func_router_endpoint(SOMEID):
if SOMEID in func_look_up_dict.keys():
return jsonify({'result' = func_look_up_dict[SOMEID]()}), 200
else:
return jsonify({'result'='unknown', 'reason'='invalid id in url'}), 404
so for this care you don't really need to "dynamically" add url rules, but rather use a url rule with parameter and handle the various cases withing a single function. Another thing to consider is to really think about the use case of such URL endpoint. If <username> is a parameter that needs to be passed in, why not to use a url rule such as /rest/product/<username> or pass it in as an argument in the GET request?
Hope that helps.