We Keep Coding

How to speed up development of rest api in python?

i'm trying to create back-end app using FastApi and sqlAchemy. I have a lot of entities which has relations with database. So, my question is: How to speed up development? Now i write for each entity code:
#app.get("/holidays")
def getHolidays():
session = Session(bind=engine)
holidays: List[Holiday] = session.query(Holiday).all()
return [x.to_json() for x in holidays]
#app.get("/exclusive_operations")
def getExclusiveOperations():
session = Session(bind=engine)
exclusive_operations: List[ExclusiveOperation] = session.query(ExclusiveOperation).all()
return [x.to_json() for x in exclusive_operations]
#app.get('/category_descriptions')
def getCategoryDescr():
session = Session(bind=engine)
category_descrs: List[CategoryDescr] = session.query(CategoryDescr).all()
return [x.to_json() for x in category_descrs]
So if i want to create all crud operations, i need to create 12 typical methods for 3 entities. Maybe another solution exists?

It is Python - as a dynamic language, the functions and methods are created at runtime. The "#app.get" decorator is what registers your views in the application, not their existence in the top level of a module.
Therefore, you can create a for loop that simply recreates and registers the view for each of your entities - it can be done either at the module level or inside a function.
(it is nice to have in mind that the "#xxxx" decorator syntax is just syntax sugar for calling the decorator passing the decorated function as its sole parameter)
for Entity, name in [(Holiday, "holidays"), (ExclusiveOperation, "exclusive_operations"), (CategoryDescr, "category_descriptions")]:
def freeze_var_wrapper(Entity, name):
# this intermediary function is needed, otherwise the Entity and name
# variables would be always up-to-date inside the view function
# and always point to the last value in the external for-loop after
# it finished execution:
def view():
session = Session(bind=engine)
entities = session.query(Entity).all()
return [x.to_json() for x in entities]
# optional, may facilitate debugging:
view.__name__ = f"get{Entity.__name__}s"
# actually registers the view function with the framework:
# (could be done in the same line, without the "view_registrer" var)
view_registrer = app.get(f"/{name}")
view_registrer(view)
freeze_var_wrapper(Entity, name)
There are other ways of doing this that might remove the boiler-plate and look more elegant - for example with class inheritance and an apropriate __init__subclass__in a base class (even if the framework does not use "class views", we will register the bound method for each class, which is just a callable):
class BaseView:
Entity: cls
view_name: str
def __init_subclass__(cls, *args, **kw):
super().__init_subclass__(*args, **kw)
app.get(f"/{cls.view_name}")(cls.view)
# above, cls.view is bound to the subclass being processed, therefore
# the class attributes as defined in each class body are used inside the method
# this could easily register post, delete and detail views as well
#classmethod
def view(cls);
session = Session(bind=engine)
entities = session.query(cls.Entity).all()
return [x.to_json() for x in entities]
class HolydayView(BaseView):
Entity = Holyday
view_name = "holydays"
# thats is just it.
class ExclusiveOperationView(BaseView):
Entity = ExclusiveOperation
view_name = "exclusive_operations"
class CatewgoryDescriptionView(BaseView):
Entity = CategoryDescription
view_name = "category_descriptions"

Correct way to register flask admin views with application factory

I am using an application factory to add views to my flask application like so :
(this is not my actual application factory, and has been shortened for the sake of brevity)
def create_app(config_name='default'):
app = Flask(__name__, template_folder="templates", static_folder='static')
admin_instance = Admin(app, name='Admin')
admin_instance.add_view(EntityAdmin(Entity, db.session))
My EntityAdmin class looks like this :
class EntityAdmin(ModelView):
column_filters = [
MyCustomFilter(column=None, name='Custom')
]
My custom filter looks like this :
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
entities = Entity.query.filter(Entity.active == True).all()
return [(entity.id, entity.name) for entity in entities]
The problem is that it seems that the get_options function is called when the app is instantiated, running a select query every time the create_app function gets called.
So if I update my database schema and run the flask db migrate command, I get an error because the new column I added does not exist when the select query is run. The query raises an error because my database schema is not in sync with the actual database.
Can I register my views only when an actual HTTP request is made ? How can I differentiate between a request and a command ?

You have one more problem with this filter: its options are created on the application instantiation so if your list of entities was changed during the application running it would still return the same list of options.
To fix both problems you don't need to postpone views registrations. You need the filter to get the list of options every time it is used.
This SO answer to the question "Resetting generator object in Python" describes a way to reuse a generator (in your case — a database query):
from flask import has_app_context
def get_entities():
# has_app_context is used to prevent database access
# when application is not ready yet
if has_app_context():
for entity in Entity.query.filter(Entity.active.is_(True)):
yield entity.id, entity.name
class ReloadingIterator:
def __init__(self, iterator_factory):
self.iterator_factory = iterator_factory
def __iter__(self):
return self.iterator_factory()
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
# This will return a generator which is
# reloaded every time it is used
return ReloadingIterator(get_entities)
The problem is that the query to the Entity table can be called multiple times during request. So I usually cache the result for a single request using Flask globals:
def get_entities():
if has_app_context():
if not hasattr(g, 'entities'):
query = Entity.query.filter(Entity.active.is_(True))
g.entities = [(entity.id, entity.name) for entity in query]
for entity_id, entity_name in g.entities:
yield entity_id, entity_name

Django - create a class instance in AppConfig.ready() only once

I need to create a class instance (lets say backend requests session) on the app startup(runserver), and I don't want to rewrite this session after running other management command. How can I achieve this? I tried several approaches and I'm not sure why something like this doesn't work.
# app/apps.py
class MyConfig(AppConfig):
....
requests_session = None
....
def ready(self):
if MyConfig.requests_session is None:
MyConfig.requests_session = requests.Session()
Unfortunately, the condition is always met and the session is recreated. This approach is recommended in the documentation though.
Other solution for me would be to run MyConfig.ready() only after using selected subset of management commands, is that possible?
Is there completely different better way for me to store requests session?
TIA

I think it should work if you use an instance variable instead of a class variable:
# app/apps.py
class MyConfig(AppConfig):
def __init__(self, app_name, app_module):
super(MyConfig, self).__init__(app_name, app_module)
self.requests_session = None
def ready(self):
if self.requests_session is None:
self.requests_session = requests.Session()
The question now is how to access this instance variable elsewhere. You can do that like so:
from django.apps import apps
# Here myapp is the label of your app - change it as required
# This returns the instance of your app config that was initialised
# at startup.
my_app_config = apps.get_app_config('myapp')
# Use the stored request session
req = my_app_config.requests_session
Note that this instance variable only exists in the context of the current process. If you run a management command in a separate process (e.g., manage.py ...) then that will create a new instance of each app.

set namespace for all db operations

I'm trying to set the namespace for all DB operations for the Google App Engine in python, but i can't get it done.
Currently my code looks something like this:
""" Set Google namespace """
if user:
namespace = thisUser.namespace
namespace_manager.set_namespace(namespace)
""" End Google namespace """
#Then i have all sorts of classes:
class MainPage(BaseHandler):
def get(self):
#code with DB operations like get and put...
class MainPage2(BaseHandler):
def get(self):
#code with DB operations like get and put...
class MainPage3(BaseHandler):
def get(self):
#code with DB operations like get and put...
app = webapp2.WSGIApplication([ ... ], debug=True, config=webapp2_config)
The problem with this is, is that in the classes all DB operations are still done on the default namespace (so as if no namespace is set). Eventhough i set the namespace in the very top of my code.
When i print the variable "namespace", which i also set in the top of the code, then i do get to see the namespace that i wish to use.
But it looks like Google App Engine somewhere resets the namespace to empty before running the code in the classes.
So now i'm wondering if there's a good way to set the namespace once somewhere.
Currently i set it like this in all "def's":
class MainPage(BaseHandler):
def get(self):
namespace_manager.set_namespace(namespace)
#code with DB operations like get and put...
class MainPage(BaseHandler):
def get(self):
namespace_manager.set_namespace(namespace)
#code with DB operations like get and put...
etc...
It's just not a very elegant solution.

You need to write a middleware that will intercept the request and will set the namespace according to your app logic.

A good solution is to add a hook. Something like that should be works.
from google.appengine.api import apiproxy_stub_map
NAMESPACE_NAME = 'noname'
def namespace_call(service, call, request, response):
if hasattr(request, 'set_name_space'):
request.set_name_space(NAMESPACE_NAME)
apiproxy_stub_map.apiproxy.GetPreCallHooks().Append(
'datastore-hooks', namespace_call, 'datastore_v3')
You can add it in your main.py or appengine_config.py. By this way the hook is configured during the loading of the instances and keeps his state.

You can use appconfig.py and define namespace_manager_default_namespace_for_request()
Have a read of https://developers.google.com/appengine/docs/python/multitenancy/multitenancy see the first section of "Setting the Current Namespace"

Proxy pattern idiom

I'm a web application developer and in using SQLAlchemy I find it clumsy to do this in many of my controllers when I'm wanting a specific row from (say) the users table:
from model import dbsession # SQLAlchemy SessionMaker instance
from model import User
user = dbsession().query(User).filter_by(some_kw_args).first()
Or say I want to add a user to the table (assuming another controller):
from model import dbsession # SQLAlchemy SessionMaker instance
from model import User
user = User("someval", "anotherval", "yanv")
dbsession().add(user)
So, because of that clumsiness (I won't go into some of my other personal idioms) I didn't like having to do all of that just to add a record to the table or to get a record from the table. So I decided (after a lot of nasty hacking on SQLAlchemy and deciding I was doing too many "magical" things) this was appropriate for the proxy pattern.
I (at first) did something like this inside of the model module:
def proxy_user(delete=False, *args, **kwargs):
session = DBSession()
# Keyword args? Let's instantiate it...
if (len(kwargs) > 0) and delete:
obj = session.query(User).filter_by(**kwargs).first()
session.delete(obj)
return True
elif len(kwargs) > 0:
kwargs.update({'removed' : False})
return session.query(User).filter_by(**kwargs).first()
else:
# Otherwise, let's create an empty one and add it to the session...
obj = User()
session.add(obj)
return obj
I did this for all of my models (nasty duplication of code, I know) and it works quite well. I can pass in keyword arguments to the proxy function and it handles all of the session querying for me (even providing a default filter keyword for the removed flag). I can initialize an empty model object and then add data to it by updating the object attributes and all of those changes are tracked (and committed/flushed) because the object has been added to the SQLAlchemy session.
So, to reduce duplication, I put the majority of the logic an decorator function and am now doing this:
def proxy_model(proxy):
"""Decorator for the proxy_model pattern."""
def wrapper(delete=False, *args, **kwargs):
model = proxy()
session = DBSession()
# Keyword args? Let's instantiate it...
if (len(kwargs) > 0) and delete:
obj = session.query(model).filter_by(**kwargs).first()
session.delete(obj)
return True
elif len(kwargs) > 0:
kwargs.update({'removed' : False})
return session.query(model).filter_by(**kwargs).first()
else:
# Otherwise, let's create an empty one and add it to the session...
obj = model()
session.add(obj)
return obj
return wrapper
# The proxy_model decorator is then used like so:
#proxy_model
def proxy_user(): return User
So now, in my controllers I can do this:
from model import proxy_user
# Fetch a user
user = proxy_user(email="someemail#ex.net") # Returns a user model filtered by that email
# Creating a new user, ZopeTransaction will handle the commit so I don't do it manually
new_user = proxy_user()
new_user.email = 'anotheremail#ex.net'
new_user.password = 'just an example'
If I need to do other more complex queries I will usually write function that handles it if I use it often. If it is a one-time thing I will just import the dbsession instance and then do the "standard" SQLAlchemy orm query....
This is much cleaner and works wonderfully but I still feel like it isn't "locked in" quite. Can anyone else (or more experienced python programmers) provide a better idiom that would achieve a similar amount of lucidity that I'm seeking while being a clearer abstraction?

You mention "didn't like having to do 'all of that'" where 'all of that' looks an awful lot like only 1 - 2 lines of code so I'm feeling that this isn't really necessary. Basically I don't really think that either statement you started with is all that verbose or confusing.
However, If I had to come up with a way to express this I wouldn't use a decorator here as you aren't really decorating anything. The function "proxy_user" really doesn't do anything without the decorator applied imo. Since you need to provide the name of the model somehow I think you're better of just using a function and passing the model class to it. I also think that rolling the delete functionality into your proxy is out of place and depending on how you've configured your Session the repeated calls to DBSession() may be creating new unrelated sessions which is going to cause problems if you need to work with multiple objects in the same transaction.
Anyway, here's a quick stab at how I would refactor your decorator into a pair of functions:
def find_or_add(model, session, **kwargs):
if len(kwargs) > 0:
obj = session.query(model).filter_by(**kwargs).first()
if not obj:
obj = model(**kwargs)
session.add(obj)
else:
# Otherwise, let's create an empty one and add it to the session...
obj = model()
session.add(obj)
return obj
def find_and_delete(model, session, **kwargs):
deleted = False
obj = session.query(model).filter_by(**kwargs).first()
if obj:
session.delete(obj)
deleted = True
return deleted
Again, I'm not convinced this is necessary but I think I can agree that:
user = find_or_add(User, mysession, email="bob#localhost.com")
Is perhaps nicer looking than the straight SQLAlchemy code necessary to find / create a user and add them to session.
I like the above functions better than your current decorator approach because:
The names clearly denote what your intent is here, where I feel proxy_user doesn't really make it clear that you want a user object if it exists otherwise you want to create it.
The session is managed explicitly
They don't require me to wrap every model in a decorator
The find_or_add function always returns an instance of model instead of sometimes returning True, a query result set, or a model instance.
the find_and_delete function always returns a boolean indicated whether or not it was successfully able to find and delete the record specified in kwargs.
Of course you might consider using a class decorator to add these functions as methods on your model classes, or perhaps deriving your models from a base class that includes this functionality so that you can do something like:
# let's add a classmethod to User or its base class:
class User(...):
...
#classmethod
def find_or_add(cls, session, **kwargs):
if len(kwargs) > 0:
obj = session.query(cls).filter_by(**kwargs).first()
if not obj:
obj = cls(**kwargs)
session.add(obj)
else:
# Otherwise, let's create an empty one and add it to the session...
obj = cls()
session.add(obj)
return obj
...
user = User.find_or_add(session, email="someone#tld.com")