This is how I basically create my app and db:
from flask import Flask
from flask_restful import Api
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
api = Api(app)
app.config['SQLALCHEMY_DATABASE_URI'] = mysql+pymysql://user:pass#heroku-hostname
db = SQLAlchemy(app)
#app.before_first_request
def create_tables():
db.create_all()
I do not explicitly handle any connection object myself, only using this db, like:
def save_to_db(self):
db.session.add(self)
db.session.commit()
def find_by_email(email):
return User.query.filter_by(email=email).first()
Pure SQLAlchemy creates some kind of Engine object with pooling parameters while Flask-SQLAlchemy's configuration documentation says there are keys for pooling options, but they are getting deprecated.
This clearly seems as an abstraction built upon this Engine object. Also, it shows these settings are actually getting default values if not specified.
The last config field is the SQLALCHEMY_ENGINE_OPTIONS, and it states indeed: A dictionary of keyword args to send to create_engine().
So how is this happening from here? Are we just basically using this single config field from now to run the original SQLAlchemy Engine functions?
What kind of dictionary values should I provide; do I still get any default values?
Coming from Flask and learning things in a top-down approach I'm a bit confused how things are working at lower layers.
Related
This question is an extension on my previous one here. I was suggested to put more to explain the problem. As the heading says, I am trying to find a way to avoid importing the application factory (create_app function) into a module that needs application context and were "import current_app as app" is not sufficient.
My problem is I have a circular import problem due to this create_app function which I need to pass in order to get the app_context.
In my __ini__.py, I have this:
# application/__init__.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
from flask_restful import Api
from application.resources.product import Product, Products
from application.resources.offer import Offer, Offers # HERE IS THE PROBLEM
api = Api()
db = SQLAlchemy()
api.add_resource(Product, "/product/<string:name>") # GET, POST, DELETE, PUT to my local database
api.add_resource(Products, "/products") # GET all products from my local database
api.add_resource(Offer, "/offer/<int:id>") # POST call to the external Offers API microservise
api.add_resource(Offers, "/offers") # GET all offers from my local database
def create_app(config_filename=None):
""" Initialize core application. """
app = Flask(__name__, instance_relative_config=False)
app.config.from_object("config.Config")
db.init_app(app)
api.init_app(app)
with app.app_context():
db.create_all()
return app
The problem is in this line:
from application.resources.offer import Offer, Offers # HERE IS THE PROBLEM
because in that module, I have:
#application/resources/offer.py
from flask_restful import Resource
from application.models.offer import OfferModel # IMPORTING OFFER MODEL
which in turn imports application/models/offer.py where I have the critical part:
#application/models/offer.py
import requests
# from flask import current_app as app
from application import create_app # THIS CAUSES THE CIRCULAR IMPORT ERROR
from sqlalchemy.exc import OperationalError
app = create_app() # I NEED TO CREATE THE APP IN ORDER TO GET THE APP CONTEXT BECASE IN THE CLASS I HAVE SOME FUNCTIONS THAT NEED IT
class OfferModel(db.Model):
""" Data model for offers. """
# some code to instantiate the class... + other methods..
# THIS IS ONE OF THE METHODS THAT NEED APP_CONTEXT OR ELSE IT WILL ERROR OUT
#classmethod
def update_offer_price(cls):
""" Call offers api to get new prices. This function will run in a separated thread in a scheduler. """
with app.app_context():
headers = {"Bearer": app.config["MS_API_ACCESS_TOKEN"]}
for offer_id in OfferModel.offer_ids:
offers_url = app.config["MS_API_OFFERS_BASE_URL"] + "/products/" + str(offer_id) + "/offers"
res = requests.get(offers_url, headers=headers).json()
for offer in res:
try:
OfferModel.query.filter_by(offer_id=offer["id"]).update(dict(price=offer["price"]))
db.session.commit()
except OperationalError:
print("Database does not exists.")
db.session.rollback()
I have tried to use from flask import current_app as app to get the context, it did not work. I don't know why it was not sufficient to pass current_app as app and get the context because it now forces me to pass the create_app application factory which causes the circular import problem.
Your update_offer_price method needs database interaction and an access to the configuration. It gets them from the application context but it works only if your Flask application is initialized. This method is run in a separate thread so you create the second instance of Flask application in this thread.
Alternative way is getting standalone database interaction and configuration access outside the application context.
Configuration
Configuration does not seem a problem as your application gets it from another module:
app.config.from_object("config.Config")
So you can directly import this object to your offer.py:
from config import Config
headers = {"Bearer": Config.MS_API_ACCESS_TOKEN}
Database access
To get standalone database access you need to define your models via SQLAlchemy instead of flask_sqlalchemy. It was already described in this answer but I post here the essentials. For your case it may look like this. Your base.py module:
from sqlalchemy import MetaData
from sqlalchemy.ext.declarative import declarative_base
metadata = MetaData()
Base = declarative_base(metadata=metadata)
And offer.py module:
import sqlalchemy as sa
from .base import Base
class OfferModel(Base):
id = sa.Column(sa.Integer, primary_key=True)
# Another declarations
The produced metadata object is used to initialize your flask_sqlalchemy object:
from flask_sqlalchemy import SQLAlchemy
from application.models.base import metadata
db = SQLAlchemy(metadata=metadata)
Your models can be queried outside the application context but you need to manually create database engine and sessions. For example:
from sqlalchemy import create_engine
from sqlalchemy.orm import Session
from config import Config
from application.models.offer import Offer
engine = create_engine(Config.YOUR_DATABASE_URL)
# It is recommended to create a single engine
# and use it afterwards to bind database sessions to.
# Perhaps `application.models.base` module
# is better to be used for this declaration.
def your_database_interaction():
session = Session(engine)
offers = session.query(Offer).all()
for offer in offers:
# Some update here
session.commit()
session.close()
Note that with this approach you can't use your models classes for queriing, I mean:
OfferModel.query.all() # Does not work
db.session.query(OfferModel).all() # Works
ok so this is how I solved it. I made a new file endpoints.py where I put all my Api resources
# application/endpoints.py
from application import api
from application.resources.product import Product, Products
from application.resources.offer import Offer, Offers
api.add_resource(Product, "/product/<string:name>") # GET, POST, DELETE, PUT - calls to local database
api.add_resource(Products, "/products") # GET all products from local database.
api.add_resource(Offer, "/offer/<int:id>") # POST call to the Offers API microservice.
api.add_resource(Offers, "/offers") # GET all offers from local database
Then in init.py I import it at the very bottom.
# aplication/__init__.py
from flask import Flask
from flask_restful import Api
from db import db
api = Api()
def create_app():
app = Flask(__name__, instance_relative_config=False)
app.config.from_object("config.Config")
db.init_app(app)
api.init_app(app)
with app.app_context():
from application import routes
db.create_all()
return app
from application import endpoints # importing here to avoid circular imports
It is not very pretty but it works.
My Flask app is hooked up to my postgres database in Heroku like so:
from flask import Flask, render_template, session, request, url_for, redirect, flash
from flask_session import Session
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
import os
app = Flask(__name__)
# Check for environment variable
if not os.getenv("DATABASE_URL"):
raise RuntimeError("DATABASE_URL is not set")
# Configure session to use filesystem
app.config["SESSION_PERMANENT"] = False
app.config["SESSION_TYPE"] = "filesystem"
Session(app)
# Set up database
engine = create_engine(os.getenv("DATABASE_URL"))
db = scoped_session(sessionmaker(bind=engine))
My Flask methods run various SQL statements. The syntax I'm using is typically like bk = db.execute("SELECT * FROM books WHERE isbn=:isbn", {"isbn": isbn}).
I thought the return value of such a statement would be a list of dictionaries, however, when I coded a method to check for len(bk) it said the object had no length even when it should have.
So, what's the return value that I'm receiving, and why doesn't it seem to have a discernible length to Python? Couldn't find a straight answer anywhere.
It's a ResultProxy object, as explained in the documentation of Session.execute(). It is iterable, but does not have a length, because in general it is not known how many rows a query produces before fetching them all. You could pass it to list(), like in the other answer, or use its fetch methods, namely fetchall().
The individual rows are not represented by dict instances, but RowProxy instances. They do act as a mapping, though, so you can use them as they were a dictionary for most purposes (except serialization to JSON, for example).
sqlalchemy will return a lazy generator of records. You can't use len but you can iterate over it and it will yield the records one by one, to save memory. If you have the memory or the table is small and you want to load all records at once, you can call list() in it:
bk = list(bk)
This is the code I have written so far in my views.py file:
from flask import Flask, jsonify
from app import db
from models import Sites
app = Flask(__name__)
app.config.from_pyfile('config.py')
db.init_app(app)
#app.route('/site-list')
def site_list():
site_table = Sites.query.all()
return jsonify({a.name: a.id for a in site_table})
#app.route('/site-geo')
def site_geo():
site_geo = Sites.query.all()
return jsonify({a.name: a.lat + ", " + a.long for a in site_geo})
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8080, passthrough_errors=True)
So I believe that by using the init_app() method of db (a flask-sqlalchemy object) I have created a new Flask instance assigned to app. The object db is imported from app.py (whose purpose in to generate data and insert it into the database). app.py and db.py share the same config.py file which specifies the database URI.
As you can see I have repeated myself within the two Flask routes with Sites.query.all(). The code currently works fine in returning the desired JSON response. However, when I try to move the assignment of Sites.query.all() from within a function/Flask route to outside of the Flask route, it gives a Runtime error: application not registered on db instance and no application bound to current context.
How can I the access the queryset across all my flask routes, so that I don't have to keep repeating myself?
If you are worried about repeating yourself one option is to add this query as a classmethod for the Sites model like below:
#classmethod
def get_all(cls):
sites_all = Sites.query.all()
return sites_all
And then in your views.py you can run Sites.get_all(), you could expand on this to pass a parameter to the method to perform your formatting also.
If you are worried about querying the database twice then you could save the query set in session memory and then retrieve it in each route that is required, although I don't know whether this would give an advantage at all.
To save:
session['sites_all'] = sites_all
To retrieve:
session.pop('sites_all', None)
I've been using Flask-SQLAlchemy for a project for about a year. I like that it abstract away the sessioning for me. But now I need more granular control over my sessioning, namely to make a DB connection in a thread after the user has left my application. Is it possible / are there any dangers to use both Flask-SQLAlchemy and SQLAlchemy at the same time?
Bonus: if I must revert to just SQLAlchemy, what must I know? Is it just session scope?
EDIT trying detached session:
(pdb) db
<SQLAlchemy engine=None>
(Pdb) db.session
<sqlalchemy.orm.scoping.scoped_session object at 0x104b81210>
(Pdb) db.session()
*** RuntimeError: application not registered on db instance and no application bound to current context
You have an app like:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
db = SQLAlchemy(app)
Currently you use:
#app.route('/add-some-item', method=['POST'])
def add_some_item():
some_item = SomeItem(foo=request.form.get('foo'))
db.session.add(some_item)
db.session.commit()
return 'The new id is: {}'.format(some_item.id)
But you can also use:
def i_run_in_some_other_thread():
# no need for a Flask request context, just use:
session = db.session() # a bare SQLAlchemy session
...
some_item = SomeItem(foo=bar)
session.add(some_item)
session.commit()
#app.route('/do-a-cron-job')
def do_a_cron_job()
Thread(target=i_run_in_some_other_thread).start()
return 'Thread started.'
By default a session is bound to a thread. For this simple case you don't need to do any changes to your code at all, but if sessions are shared between threads, then you would need to do a few changes: “Session and sessionmaker()”.
Just don't share sessions or objects between threads I'd say, or things will get messy. Share IDs and you're fine.
Context: I'm working on a Flask app, running on CherryPy, DB handled using SQLAlchemy ORM.
Problem:
The app runs fine and does everything I want, however, if I have a page which fetches some data from DB and displays, and I press and hold "Ctrl + R" or "F5". That is, continuously refresh a page, making that many DB requests. First few goes fine, and then it breaks.
The following errors are logged:
(OperationalError) (2013, 'Lost connection to MySQL server during query')
Can't reconnect until invalid transaction is rolled back (original cause:
InvalidRequestError: Can't reconnect until invalid transaction is rolled back)
This result object does not return rows. It has been closed automatically.
(ProgrammingError) (2014, "Commands out of sync; you can't run this command now")
There's also another error which bothers me (but not logged this time), it's
dictionary changed size during iteration
This happens when I'm iterating through a query, using values obtained to populate a dictionary. The dictionary is local (scope of the dict) to the function.
More info:
How I am handling sessions:
A new session is created when you enter any page, use that session to perform all the DB transactions, and the session is closed right before rendering the HTML. Technically, that means, the scope of session is the same as the HTTP request.
I do a session.rollback() only when there's an exception raised during updating table or inserting into a table. No rollback() during any query() operations.
I'm pretty sure I've made some silly mistakes or am not doing things the right way.
Unlimited refreshes like that is not really a probably scenario, but can't be overlooked.
Also, I think the behavior would be similar when there a lot of users using it at the same time.
How the SQLAlchemy engine, sessionmaker was handled:
sql_alchemy_engine = create_engine(self.db_string, echo=False, encoding="utf8", convert_unicode=True, pool_recycle=9)
sqla_session = sessionmaker(bind=sql_alchemy_engine)
It's done only ONCE like it's recommended in the SQLA documentation, and a new session is created and returned sqla_session() whenever required.
If you're using Flask, you should be using flask-sqlalchemy, and let the Flask request context manage your session, and not handling your engine and sessions by hand. This is how SQLAlchemy recommends it:
Most web frameworks include infrastructure to establish a single Session, associated with the request, which is correctly constructed and torn down corresponding torn down at the end of a request. Such infrastructure pieces include products such as Flask-SQLAlchemy, for usage in conjunction with the Flask web framework, and Zope-SQLAlchemy, for usage in conjunction with the Pyramid and Zope frameworks. SQLAlchemy strongly recommends that these products be used as available.
http://docs.sqlalchemy.org/en/rel_0_9/orm/session.html?highlight=flask
Then you create your engine simply by:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'your db uri'
db = SQLAlchemy(app)
Or, if you're using app factory:
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
db = SQLAlchemy()
def create_app():
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'your db uri'
db.init_app(app)
With that, the base declarative model you should be using will be at db.Model and the session you should be using will be at db.session.