I'm writing a Python library that represents some web API. Right now, my library directory looks close to this:
__init__.py
Account.py
Order.py
Category.py
requests.py
In __init__.py, I have something like this:
from .Account import Account
from .Order import Order
from .Category import Category
from . import requests
This allows to use import cool_site and then cool_site.Account(…) and so on, but it has a following problem: when I play around with my code in IDLE, the object is then called cool_site.Account.Account, which I feel is bad.
1. Is there any way to avoid class name duplication and still have separate file for every class?
The next thing I don't feel great about is my code organization. Right now, my Account class takes credentials on initialization, creates a requests.Session object and then handles all communication with server, i.e. searching for orders and so on. This Account class instance will then pass itself to all other instances, for example to Order - so the order's instance will have .account property holding the Account instance which created it. When another class instance itself has to do something, for example change an order's comment (by calling o.comment = 'new comment', so by #comment.setter decorator in the Order class), it forwards that to an Account object which is passed to it on initialization, and then uses for example self.account.set_order_comment(new_comment). This method will then use all the web requests to achieve that goal.
2. Is it better to hold server communication logic in one class or to spread different aspects of it to classes that are affected by them?
The last thing I'd like to ask about is how and where to keep low-level request templates. Right now I have it in cool_site.requests submodule, and there are different functions for different requests, for example SetOrderComment for the case mentioned above (it's a function, so it should be lowercase, but in this case I think it resembles a class in a way - is that OK?). The Account.set_order_comment will use it like this:
r = cool_site.requests.SetOrderComment(order.id, new_comment)
response = self._session.request(**r)
because this function returns a dict with arguments to Session.request function from requests library. The authentication headers are already set in the _session property of Account class instance. I feel it's a little bit ugly, but I don't have any better idea.
3. How to organize web requests to keep it all clean?
Post scriptum
I'm sorry this question is so long and covers many aspects of API library design, but all the tips will be appreciated. In a way, all of the three questions above could be expressed as "How to do it better and cleaner?" or "How most of the Python developers do it?", or maybe even "What would feel most Pythonic?".
Throw at me any little tips you can think of.
I've been thinking about very similar things lately working on wistiapy. Examples of my current thinking about client code organisation are in there. YMMV.
"One class per file" is more of a Java style guideline than Python. Python modules are a legitimate and important level in the code hierarchy, and you shouldn't worry about having more than one function or class in the same module. You could put all the model classes in a .models module, and then from .models import (Account, Order, Category) in __init__.py.
More-or-less common practice for client libraries seems to be to have a client module, containing something like a MyServiceClient class. (eg the Segment client). This is where the networking logic goes. If you want to have the public interface be module-level functions, you can do some clever stuff with creating a default client and having the functions call methods on that.
Functions should be snake_case, classes should be PascalCase. Doing anything else tends to cause more confusion than benefit.
It seems like the big question you're dealing with is trying to choose between the "Active Record" pattern (some_account.set_order_comment(comment)), and the "Data Mapper" pattern (set_order_comment(account, comment)). Either will work and they each have their benefits and drawbacks. I've found the data mapper pattern -- using smart functions to manipulate fairly simple data classes -- simpler to begin with.
I find it helpful to design the public interface concurrently with something that uses that interface. In the calling code, you can write what you'd like to have to call, and then implement the client code "outside-in".
1) no upper case in names of .py file (also try to avoid _)
so your files should be
__init__.py
account.py
order.py
category.py
requests.py
2) if you want to use like cool_site.Account you need to add to __init__.py
from .account import Account
from .order import Order
from .category import Category
__all__ = [
'Account',
'Order',
'Category',
]
3) SetOrderComment is bad name, use set_order_comment
4) If you write a python wrapper for communication with API, make method that do authorisation and other other stuff that is same in every API request. This method should take as params part of requests kwargs that are different for different API calls
for example
class API:
def __init__(self, endpoint:s str, api_key: str):
self.endpoint = endpoint
self.api_key = api_key
def _get_auth_headers(self) -> Dict[str, str]:
return {
'Authorization': 'Bearer ' + self.api_key,
}
def get(self, path, params)
resp = requester.get(
self.endpoint + path,
headers=self._get_auth_headers(),
params=params,
timeout=30,
)
self._check_api_response(resp)
payload = resp.json()
return payload
5) If you write a python API look at flask and django frameworks and projects with API written on them. You should find some good ides overthere.
I can throw in an easy tip:
You could use __all__ = ["Account"] in your Account.py module and then use from .Account import * in your __init__.pyfile. I believe that addresses your first question.
You can read up on the __all__ magic method here. In short, you can specify what methods etc you would like to import when you use from x import *. You can hide 'private' methods from being imported with _one_leading_underscore.
Related
So I am creating an API class in python, the goal of which is to allow me to create custom API routes that I can call methods on to make calls to APIs (as well as other helpful operations). However, I want this to be very general purpose and safe, as some of the API calls look a little different than others. What I want to know is, what is the best way to fill in an API call to a URL as a string using an OO approach? Take the following example:
#
# ...defined apiObject class up here somewhere (name tbd)...
#
myAPI = apiObject('https://jsonplaceholder.typicode.com/{}/{}')
# Note the {} as placeholders for the API calls
# Below is an example of calling the API
myAPI.callAPI('comments', '123')
# This would form the url 'https://jsonplaceholder.typicode.com/comments/123' make the call and do some stuff
# Similarly:
myAPI2 = apiObject('https://anotherrestapi.test.com/{}/{}/{}')
myAPI.callAPI('2020', 'Feb', '13')
# Again, would form the url 'https://anotherrestapi.test.com/2020/Feb/13' make the call and do some stuff
This is not a problem I can't fix, moreover, its a question about best programming practices in Python. My initial reaction is to use **kwargs in the callAPI function as I could then name the placeholders and fill in the url as needed.
Alternatively, is there a way I should form my API class such that this is more set in stone?
Or last of all, do you have any library recommendations for me to check out as an alternative to doing this myself?
I am not as familiar with Python as I am with other languages, so I thought I'd ask and see what other people think. Thanks in advance for taking the time to think about it.
I’m new to Pyramid. I’ve used Python for a few months. I've created a Python application on Linux to maintain an Oracle database using weekly data feeds from a vendor. To get that done, one of the things I did was to create a customized database wrapper class using the cx_Oracle package. I had specific requirements for maintaining history in the DB. All Oracle access goes through the methods in this wrapper. I now want to use Pyramid to create a simple reporting browser interface to the Oracle DB. To allow me the greatest flexibility, I’d like to use the wrapper I already have to get to the data on Oracle instead of Alchemy (or possibly with it, I'm not sure).
In my Pyramid app, I’ve considered importing my wrapper in my views.py init method but that seems to get executed with every browser submit.
Can anyone suggest how I might create a persistent connection to Oracle that I can use over and over from my reporting application which uses my wrapper class? I’m finding Pyramid a bit opaque. I’m never sure what’s happening behind the scenes but I’m willing to operate on trust until I get the swing of it. I need the benefit of the automatic authorization/ authentication and login.
What I’m really looking for is a good approach from experienced Pyramid users before going down the wrong track.
Many thanks.
You should definitely use SQLAlchemy as it makes use of connection pooling and such. In your case SQLAlchemy would use cx_Oracle underneath but make you be able to concentrate writing actual code instead of maintaining connections/pooling them/and such.
You should follow the patterns in Wiki2 tutorial to set up basic SQLAlchemy.
So, basically, the question boils down to "how do I use my existing API with Pyramid", right? This is quite easy as Pyramid is database-agnostic and very transparent in this area despite what you say :) Basically, you import it, call its methods, retrieve data and send it to the template:
import pokemon_api as api
#view_config(route_name='list_pokemon', renderer='pokemon_list.mako')
def list_pokemon(request):
# To illustrate how to get data from the request to send to your API
batch_start = request.GET.get('batch_start', 0)
batch_end = batch_start + request.GET.get('batch_size', 0)
sort_order = request.GET.get('sort_by', 'name')
# Here we call our API - we don't actually care where it gets the data from, it's a black box
pokemon = api.retrieve_pokemon(from=batch_start, to=batch_end, sort=sort_order)
# send the data to the renderer/template
return {
'pokemon': pokemon
}
#view_config(route_name='add_pokemon', request_method='POST')
def add_pokemon(request):
"""
Add a new Pokemon
"""
name = request.POST.get('name', 0)
weight = request.GET.get('weight', 0)
hight = request.GET.get('hight', 0)
api.create(name=name, weight=weight, height=height)
# go back to the listing
return HTTPFound('/pokemon_list')
and if your API needs some initialization, you can do it at startup time
import pokemon_api as api
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
config = Configurator(settings=settings)
...
api.init("MAGIC_CONNECTION_STRING")
return config.make_wsgi_app()
Of course, this assumes your API already handles transactions, connections, pooling and other boring stuff :)
One last point to mention - in Python you generally don't import things inside methods, you import them at the top of the file at module-level scope. There are exceptions for that rule, but I don't see why you would need that in this case. Also, importing a module should be free of side-effects (which you might have since "importing my wrapper in my views.py __init__ method" seems to be causing trouble).
You can see the code here
The concrete problem that I'm trying to solve is this.
Say that I need to provide REST interface to some entities modeled (sqlalchemy in my case) with some tool stored in database. Say that this collection is called parents.
I would need handlers like this
GET /parents
GET /parents/some_id
POST /parents
DELETE /parents/some_id
Conceptually all this handlers are very similar.
They all take ids from url, then create appropriate query. Then they fetching data with that query, then turn this data into dict and then call jsonify to create correct http response.
So with OOP I could design this like that.
class AbstractHandler():
def __init__(serializer, **specs):
self.specs = specs
def handle_request(self, spec_data, *_ids):
q = self.create_query(_ids)
d = self.fetch_data(self.specs[spec_data['name']](**(spec_data['args'] + (query, ))
out = serializer(d)
return jsonify(out)
The spec is a function that takes some parameters and query and produce more refined query based of this parameters.
So for example
GET /parents?spec={'name': 'by_name', 'args': ['adam'}
would return parent named Adam from collection.
This code has some flaws but I hope you see the point how template method makes flow of control here and subclasses can change how they would create query, how they would fetch data (item handler would need to call query.one() and collection handler would need to call query.all() for example)
So I can replace create_query, fetch_data with dependency injection instead. But that would create a problem that someone could create wrong configuration by giving the wrong dependency. That's basically what I've done, but with using partial functions instead.
So what I'm thinking right now is that I can solve this problem by creating factory functions for every type of handler that I need, that would give appropriate dependency to handler.
That's very much alike with template method solution I think. The difference basically is that in template method correctness dependencies is guarantied by object type and in my solution it's guarantied by type of factory function.
So enough with what I think, I'd like to know what you think about that?
How people in functional world solve this kinds of problem?
While Pyramid documentation is usually quite good, I can't find how to set up integration tests for a Pyramid 1.5 application (note essential code fragment missing!). That is, where to put particular configurations.
In the __init__ I have main() function, which includes some other modules, among which there is one, which requires SQLAlchemy. Another complication is that auth policy is dependent on settings, so all that lives in main (otherwise I'd made includeme, not sure if it is relevant to this question).
At first I tried the following (after gathering what I could into includeme function):
class ViewTests(unittest.TestCase):
def setUp(self):
self.config = testing.setUp()
self.config.include('myproject.includeme')
Which of course fails somewhere in the sqlalchemy engine_from_config.
But then I've found, that I can probably use main() and provide test.ini for configuration. Still I do like the all-included approach of the gist:
class IntegrationTestBase(BaseTestCase):
#classmethod
def setUpClass(cls):
cls.app = main({}, **settings)
super(IntegrationTestBase, cls).setUpClass()
def setUp(self):
self.app = webtest.TestApp(self.app)
self.config = testing.setUp()
super(IntegrationTestBase, self).setUp()
Part of the problem with the code above is that settings are not ending up in self.config. If I enhance it with self.config = testing.setUp(settings=settings), the test still fails:
AttributeError: 'DummyRequest' object has no attribute 'include'
pyramid_bowerstatic does not have its chance to modify request, etc. So, instead of having integration tests out-of-the-box (the app itself runs without complains!) with given config and concentrate on writing tests, I need to care for all third-party modules whims.
Thus, I hope there is some better way to do integration tests, which are deal with several included packages, custom auth policies, events, and other "magic".
Py.test is to be used, if that matters. pytest_pyramid seems to be relevant, but docs do not have any examples.
Not exactly, but similar problem: How to Make view_config decorator work with a Pyramid Unit Test?, especially in the comments to the answer.
I have found ad hoc solution (which works for me for now) in Japanese: http://qiita.com/podhmo/items/2c6d8cb78c1f84569f0a
but the question how to reliably setup integration tests, equivalent to application in all aspects, without looking into third-party modules, except for specific ini-file?
Update: this is one of the calls in question:
View:
components = pyramid_bowerstatic.create_components('sbadmin',
os.path.join(os.path.dirname(__file__), 'bower_components'))
class MainViews(Layouts):
#view_config(route_name='home', renderer='templates/mytemplate.pt')
def my_view(self):
self.request.include(components, 'jquery')
return {'project': 'Myproject'}
In tests:
class ViewTests(IntegrationTestBase):
def test_my_view(self):
from myproject.views.main import MainViews
request = self._make_request(self.config) # before was: DummyRequest
info = MainViews(request).my_view()
self.assertEqual(info['project'], 'Myproject')
Due to nicely decoupled architecture of Pyramid, it is probably enough to have all-inclusive request. That is, the question can be reformulated as: What is generic _make_request function, which gives a result, identical to the running application request (with extensions, tweens, added request attributes, including those from 3rd party modules, etc)? Is there some ready factory? IMHO, it is NOT integration testing if a developer needs to mock it's own "integration" to test against vs. take what application really is. I am not sure if these are the only things, but I guess the factory should at least provide a request with all hooks like mentioned here: http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html regardless of which included package added them.
Update 2: It came to my mind that there is quite clear first approximation indicator for discrepancies between test and run-time environments. I have compared requests, for a normal call and test call to the view. First, here is my _make_request, which is sufficient for adding request methods, but much more is missing (below):
def _make_request(self, config, path="/"):
from pyramid.request import Request
from pyramid.interfaces import IRequestExtensions
request = Request.blank(path)
extensions = config.registry.getUtility(IRequestExtensions)
request.registry = config.registry
request._set_extensions(extensions)
return request
Normal call has the following attributes (request.__dict__.keys()):
['traversed',
'virtual_root',
'virtual_root_path',
'subpath',
'request_iface',
'__view__',
'view_name',
'tm',
'environ',
'registry',
'context',
'matched_route',
'get_bowerstatic_path',
'include',
'root',
'matchdict',
'invoke_subrequest']
And test call only these:
['environ',
'registry',
'get_bowerstatic_path',
'include']
This clearly shows, that my approach is not sufficient yet, and I will encounter problems with views, which use other request capabilities. With the _make_request above, only minimal view goes through.
In other words, how to get the same request, which is in use in functional tests e.g. in webtest's TestApp, but instead of doing testapp.get(...), call the view with that request and make assertions on callable's returned result (not rendered as HTML)?
That essential code fragment is missing because of a temporary bug on RTD. An Issue has been logged.
Here's a link to the code that is not getting included.
https://github.com/Pylons/pyramid/blob/master/docs/narr/MyProject/myproject/tests.py#L19-L43
[Edited]
Following are two examples of tests for Pyramid's alchemy scaffold. Each has its own style. Hopefully you find one of them useful to your scenario.
If you have Pyramid installed, you can create a sample project from the alchemy scaffold.
$ $VENV/bin/pcreate -s alchemy tutorial
The SQLAlchemy + URL Dispatch Wiki Tutorial goes into more detail and pre-requisites if that command does not work for you.
The scaffold includes these tests.
We are working on a new branch that updates this scaffold, and is targeted for merge in Pyramid 1.7. Here is its tests.
If you git checkout that branch, and run through the tutorial steps, you can see the full context.
[Edited]
Here's one more place to look: Pyramid's own integration tests.
As an aside, I'm having a hard time giving you an appropriate answer. For starters, I don't see where DummyRequest is getting called in your test's code. Can you give a complete reproducible example in your question?
I am developing a web API with 10 tables or so in the backend, with several one-to-many and many-to-many associations. The API essentially is a database wrapper that performs validated updates and conditional queries. It's written in Python, and I use SQLAlchemy for ORM and CherryPy for HTTP handling.
So far I have separated the 30-some queries the API performs into functions of their own, which look like this:
# in module "services.inventory"
def find_inventories(session, user_id, *inventory_ids, **kwargs):
query = session.query(Inventory, Product)
query = query.filter_by(user_id=user_id, deleted=False)
...
return query.all()
def find_inventories_by(session, app_id, user_id, by_app_id, by_type, limit, page):
....
# in another service module
def remove_old_goodie(session, app_id, user_id):
try:
old = _current_goodie(session, app_id, user_id)
services.inventory._remove(session, app_id, user_id, [old.id])
except ServiceException, e:
# log it and do stuff
....
The CherryPy request handler calls the query methods, which are scattered across several service modules, as needed. The rationale behind this solution is, since they need to access multiple model classes, they don't belong to individual models, and also these database queries should be separated out from direct handling of API accesses.
I realize that the above code might be called Foreign Methods in the realm of refactoring. I could well live with this way of organizing for a while, but as things are starting to look a little messy, I'm looking for a way to refactor this code.
Since the queries are tied directly to the API and its business logic, they are hard to generalize like getters and setters.
It smells to repeat the session argument like that, but as the current implementation of the API creates a new CherryPy handler instance for each API call and therefore the session object, there is no global way of getting at the current session.
Is there a well-established pattern to organize such queries? Should I stick with the Foreign Methods and just try to unify the function signature (argument ordering, naming conventions etc.)? What would you suggest?
The standard way to have global access to the current session in a threaded environment is ScopedSession. There are some important aspects to get right when integrating with your framework, mainly transaction control and clearing out sessions between requests. A common pattern is to have an autocommit=False (the default) ScopedSession in a module and wrap any business logic execution in a try-catch clause that rolls back in case of exception and commits if the method succeeded, then finally calls Session.remove(). The business logic would then import the Session object into global scope and use it like a regular session.
There seems to be an existing CherryPy-SQLAlchemy integration module, but as I'm not too familiar with CherryPy, I can't comment on its quality.
Having queries encapsulated as functions is just fine. Not everything needs to be in a class. If they get too numerous just split into separate modules by topic.
What I have found useful is too factor out common criteria fragments. They usually fit rather well as classmethods on model classes. Aside from increasing readability and reducing duplication, they work as implementation hiding abstractions up to some extent, making refactoring the database less painful. (Example: instead of (Foo.valid_from <= func.current_timestamp()) & (Foo.valid_until > func.current_timestamp()) you'd have Foo.is_valid())
SQLAlchemy strongly suggests that the session maker be part of some global configuration.
It is intended that the sessionmaker()
function be called within the global
scope of an application, and the
returned class be made available to
the rest of the application as the
single class used to instantiate
sessions.
Queries which are in separate modules isn't an interesting problem. The Django ORM works this way. A web site usually consists of multiple Django "applications", which sounds like your site that has many "service modules".
Knitting together multiple services is the point of an application. There aren't a lot of alternatives that are better.