We Keep Coding

How to use use flask-caching decorator #cache on an external module class

Context:
I have flask application, which is an NGINX authentication module, written in flask, and in some part of the code I am making a call to an LDAP server.
The objective is to use flask-caching library to cache the response from the LDAP server and avoid that costly call.
The code bellow its a very stripped down version showing only the relevant sections.
Problem
In the below scenario I can't use the decorator #cache.memoized on a method of the Ldap class, as the variable cache is not available in that module.
name 'cache' is not defined
main.py
from flask import Flask
from flask_caching import Cache
from ldap import Ldap
app = Flask(__name__)
cache = Cache(app)
#app.route('/', defaults={'path': ''})
#app.route('/<path:path>')
#auth.login_required
def index(path):
code = 200
msg = "Another LDAP Auth"
headers = [('x-username', getRegister('username')),
('x-groups', getRegister('matchedGroups'))]
return msg, code, headers
#auth.verify_password
def login(username, password):
ldap = Ldap()
ldap.verifyUser(username, password)
ldap.py
class Ldap:
#cache.memoized(timeout=300) <<<< error
def validateUser(self, username, password)
if <ldap query goes here>
return True
return False
Research
The weird thing for me here is that this decorator depends on an object instance and not on a class as so many other scenarios I've seen
Alternative:
Definitively if I put the definition of the class in the same main.py and bellow the definition of the cache variable it works, however this will make my main.py file too long.
Attempt 1:
Trying to import my module after the definition of the cache variable has the same error
Attempt 2:
doing from main import cache inside ldap.py creates a circular import error.
Idea:
Pass the cache variable to the Ldap Class constructor and "somehow" use it to decorate the method, but I could not find how to do it exactly.

Avoiding importing application factory into module needing application context

This question is an extension on my previous one here. I was suggested to put more to explain the problem. As the heading says, I am trying to find a way to avoid importing the application factory (create_app function) into a module that needs application context and were "import current_app as app" is not sufficient.
My problem is I have a circular import problem due to this create_app function which I need to pass in order to get the app_context.
In my __ini__.py, I have this:
# application/__init__.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_restful import Api
from application.resources.product import Product, Products
from application.resources.offer import Offer, Offers # HERE IS THE PROBLEM
api = Api()
db = SQLAlchemy()
api.add_resource(Product, "/product/<string:name>") # GET, POST, DELETE, PUT to my local database
api.add_resource(Products, "/products") # GET all products from my local database
api.add_resource(Offer, "/offer/<int:id>") # POST call to the external Offers API microservise
api.add_resource(Offers, "/offers") # GET all offers from my local database
def create_app(config_filename=None):
""" Initialize core application. """
app = Flask(__name__, instance_relative_config=False)
app.config.from_object("config.Config")
db.init_app(app)
api.init_app(app)
with app.app_context():
db.create_all()
return app
The problem is in this line:
from application.resources.offer import Offer, Offers # HERE IS THE PROBLEM
because in that module, I have:
#application/resources/offer.py
from flask_restful import Resource
from application.models.offer import OfferModel # IMPORTING OFFER MODEL
which in turn imports application/models/offer.py where I have the critical part:
#application/models/offer.py
import requests
# from flask import current_app as app
from application import create_app # THIS CAUSES THE CIRCULAR IMPORT ERROR
from sqlalchemy.exc import OperationalError
app = create_app() # I NEED TO CREATE THE APP IN ORDER TO GET THE APP CONTEXT BECASE IN THE CLASS I HAVE SOME FUNCTIONS THAT NEED IT
class OfferModel(db.Model):
""" Data model for offers. """
# some code to instantiate the class... + other methods..
# THIS IS ONE OF THE METHODS THAT NEED APP_CONTEXT OR ELSE IT WILL ERROR OUT
#classmethod
def update_offer_price(cls):
""" Call offers api to get new prices. This function will run in a separated thread in a scheduler. """
with app.app_context():
headers = {"Bearer": app.config["MS_API_ACCESS_TOKEN"]}
for offer_id in OfferModel.offer_ids:
offers_url = app.config["MS_API_OFFERS_BASE_URL"] + "/products/" + str(offer_id) + "/offers"
res = requests.get(offers_url, headers=headers).json()
for offer in res:
try:
OfferModel.query.filter_by(offer_id=offer["id"]).update(dict(price=offer["price"]))
db.session.commit()
except OperationalError:
print("Database does not exists.")
db.session.rollback()
I have tried to use from flask import current_app as app to get the context, it did not work. I don't know why it was not sufficient to pass current_app as app and get the context because it now forces me to pass the create_app application factory which causes the circular import problem.

Your update_offer_price method needs database interaction and an access to the configuration. It gets them from the application context but it works only if your Flask application is initialized. This method is run in a separate thread so you create the second instance of Flask application in this thread.
Alternative way is getting standalone database interaction and configuration access outside the application context.
Configuration
Configuration does not seem a problem as your application gets it from another module:
app.config.from_object("config.Config")
So you can directly import this object to your offer.py:
from config import Config
headers = {"Bearer": Config.MS_API_ACCESS_TOKEN}
Database access
To get standalone database access you need to define your models via SQLAlchemy instead of flask_sqlalchemy. It was already described in this answer but I post here the essentials. For your case it may look like this. Your base.py module:
from sqlalchemy import MetaData
from sqlalchemy.ext.declarative import declarative_base
metadata = MetaData()
Base = declarative_base(metadata=metadata)
And offer.py module:
import sqlalchemy as sa
from .base import Base
class OfferModel(Base):
id = sa.Column(sa.Integer, primary_key=True)
# Another declarations
The produced metadata object is used to initialize your flask_sqlalchemy object:
from flask_sqlalchemy import SQLAlchemy
from application.models.base import metadata
db = SQLAlchemy(metadata=metadata)
Your models can be queried outside the application context but you need to manually create database engine and sessions. For example:
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from config import Config
from application.models.offer import Offer
engine = create_engine(Config.YOUR_DATABASE_URL)
# It is recommended to create a single engine
# and use it afterwards to bind database sessions to.
# Perhaps `application.models.base` module
# is better to be used for this declaration.
def your_database_interaction():
session = Session(engine)
offers = session.query(Offer).all()
for offer in offers:
# Some update here
session.commit()
session.close()
Note that with this approach you can't use your models classes for queriing, I mean:
OfferModel.query.all() # Does not work
db.session.query(OfferModel).all() # Works

ok so this is how I solved it. I made a new file endpoints.py where I put all my Api resources
# application/endpoints.py
from application import api
from application.resources.product import Product, Products
from application.resources.offer import Offer, Offers
api.add_resource(Product, "/product/<string:name>") # GET, POST, DELETE, PUT - calls to local database
api.add_resource(Products, "/products") # GET all products from local database.
api.add_resource(Offer, "/offer/<int:id>") # POST call to the Offers API microservice.
api.add_resource(Offers, "/offers") # GET all offers from local database
Then in init.py I import it at the very bottom.
# aplication/__init__.py
from flask import Flask
from flask_restful import Api
from db import db
api = Api()
def create_app():
app = Flask(__name__, instance_relative_config=False)
app.config.from_object("config.Config")
db.init_app(app)
api.init_app(app)
with app.app_context():
from application import routes
db.create_all()
return app
from application import endpoints # importing here to avoid circular imports
It is not very pretty but it works.

Correct way to register flask admin views with application factory

I am using an application factory to add views to my flask application like so :
(this is not my actual application factory, and has been shortened for the sake of brevity)
def create_app(config_name='default'):
app = Flask(__name__, template_folder="templates", static_folder='static')
admin_instance = Admin(app, name='Admin')
admin_instance.add_view(EntityAdmin(Entity, db.session))
My EntityAdmin class looks like this :
class EntityAdmin(ModelView):
column_filters = [
MyCustomFilter(column=None, name='Custom')
]
My custom filter looks like this :
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
entities = Entity.query.filter(Entity.active == True).all()
return [(entity.id, entity.name) for entity in entities]
The problem is that it seems that the get_options function is called when the app is instantiated, running a select query every time the create_app function gets called.
So if I update my database schema and run the flask db migrate command, I get an error because the new column I added does not exist when the select query is run. The query raises an error because my database schema is not in sync with the actual database.
Can I register my views only when an actual HTTP request is made ? How can I differentiate between a request and a command ?

You have one more problem with this filter: its options are created on the application instantiation so if your list of entities was changed during the application running it would still return the same list of options.
To fix both problems you don't need to postpone views registrations. You need the filter to get the list of options every time it is used.
This SO answer to the question "Resetting generator object in Python" describes a way to reuse a generator (in your case — a database query):
from flask import has_app_context
def get_entities():
# has_app_context is used to prevent database access
# when application is not ready yet
if has_app_context():
for entity in Entity.query.filter(Entity.active.is_(True)):
yield entity.id, entity.name
class ReloadingIterator:
def __init__(self, iterator_factory):
self.iterator_factory = iterator_factory
def __iter__(self):
return self.iterator_factory()
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
# This will return a generator which is
# reloaded every time it is used
return ReloadingIterator(get_entities)
The problem is that the query to the Entity table can be called multiple times during request. So I usually cache the result for a single request using Flask globals:
def get_entities():
if has_app_context():
if not hasattr(g, 'entities'):
query = Entity.query.filter(Entity.active.is_(True))
g.entities = [(entity.id, entity.name) for entity in query]
for entity_id, entity_name in g.entities:
yield entity_id, entity_name

Why doesn't flask session object need to be instantiated?

Reading the documentation I understand that flask defines a class flask.session.
The thing that confuses me is that when people use it they don't instantiate an object of the session class, but use session directly, as in the following code:
from flask import Flask, session
app = Flask(__name__)
#app.route('/')
def index():
session['key'] = 'value'
I don't understand why the code shouldn't look something like this instead:
from flask import Flask, session
app = Flask(__name__)
s = session() # so s is an instance of the flask.session class
#app.route('/')
def index():
s['key'] = 'value'
I am also wondering if this has anything to do with session being a proxy, as it says in the documentation. I read the 'Notes on Proxies' but couldn't understand much.

Awesome question.
It gets initialized in flasks globals.py
https://github.com/pallets/flask/blob/master/flask/globals.py
session = LocalProxy(partial(_lookup_req_object, 'session'))
So when you import from flask you import from its package __init__.py which pulls session from globals.py and initializes it. You grab a reference to that when you directly import it.
I should clarify that session itself is not a class. It’s an instance of the LocalProxy class, which is a proxy to the request context.

Pyramid beaker accessing Sessions that have been created

I am trying to use pyramid beaker in Pyramid framework and its just not working it creates the session objects but i cannot access them with the line
#view_config(route_name='load_qli', renderer='json')
def load_qli(request):
request.environ['beaker.session']
It gives the following error
KeyError
KeyError: 'beaker.session'
My development.ini file looks like this
# pyramid_beaker settings
session.type = file
session.data_dir = %(here)s/data/sessions/data
session.lock_dir = %(here)s/data/sessions/lock
session.key = customerskey
session.secret = customerssecret
session.cookie_on_exception = true
and init.py like this
from pyramid.config import Configurator
from sqlalchemy import engine_from_config
from qlipe.models import DBSession
from pyramid_mailer import mailer_factory_from_settings
from pyramid_beaker import session_factory_from_settings
def main(global_config, **settings):
""" This function returns a Pyramid WSGI application.
"""
engine = engine_from_config(settings, 'sqlalchemy.')
DBSession.configure(bind=engine)
# pyramid_beaker add-on
session_factory = session_factory_from_settings(settings)
config = Configurator(
settings=settings,
session_factory=session_factory
)
I create the session like this
def my_view(request):
session = request.session
session['name'] = 'Fred Smith'
session.save()
Where am i going wrong?

You should be able to just use the include way and the pyramid_beaker package can initialize itself from the ini values.
in your ini file:
pyramid_includes = pyramid_beaker
or inside your main function's __init__.py file:
config.include('pyramid_beaker')
You can read more here http://docs.pylonsproject.org/projects/pyramid_beaker/en/latest/#setup
The usual way to access the session is through the request like you do in my_view:
session = request.session
The pyramid_beaker package use the pyramid session factory and the way it manages the session is not through the request.environement['beaker.session'] object like beaker's example. For more info read http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/narr/sessions.html