In my Flask application, I hope to use pymongo directly. But I am not sure what's the best way to create pymongo connection for each request and how to reclaim the connection resource.
I know Connection in pymongo is thread-safe and has built-in pooling. I guess I need to create a global Connection instance, and use before_request to put it in flask g.
In the app.py:
from pymongo import Connection
from admin.views import admin
connection = Connection()
db = connection['test']
#app.before_request
def before_request():
g.db = db
#app.teardown_request
def teardown_request(exception):
if hasattr(g, 'db'):
# FIX
pass
In admin/views.py:
from flask import g
#admin.route('/')
def index():
# do something with g.db
It actually works. So questions are:
Is this the best way to use Connection in flask?
Do I need to explicitly reclaim resources in teardown_request and how to do it?
I still think this is an interesting question, but why no response... So here is my update.
For the first question, I think using current_app is more clearer in Flask.
In app.py
app = Flask(__name__)
connection = Connection()
db = connection['test']
app.db = db
In the view.py
from Flask import current_app
db = current_app.db
# do anything with db
And by using current_app, you can use application factory to create more than one app as http://flask.pocoo.org/docs/patterns/appfactories/
And for the second question, I'm still figuring it out.
Here's example of using flask-pymnongo extension:
Example:
your mongodb uri (till db name) in app.config like below
app.config['MONGO_URI'] = 'mongodb://192.168.1.1:27017/your_db_name'
mongo = PyMongo(app, config_prefix='MONGO')
and then under your api method where u need db do the following:
db = mongo.db
Now you can work on this db connection and get your data:
users_count = db.users.count()
I think what you present is ok. Flask is almost too flexible in how you can organize things, not always presenting one obvious and right way. You might make use of the flask-pymongo extension which adds a couple of small conveniences. To my knowledge, you don't have to do anything with the connection on request teardown.
Related
I am practicing python with flask, flask_mongoengine and mongoDB.
I have followed various tutorials and for the moment am using application factory to set up my App (webservice for querying a database).
Code looks something like this:
db = MongoEngine()
def initialise_db(app):
db.init_app(app)
def create_app():
app = Flask(__name__)
api = Api(app)
initialise_db(app)
initialise_routes(api)
return app
I'd like to know if there is a way of checking the database connection is OK, before creating the app. Here the initialise_db(app) runs with no error even if there is no running instance of mongoDB (on port 21017 or other if specified in app.config).
I only get an error at the point were I try to do a db.Document.save()...
EDIT:
I have implemented a simple check_connection() function where I try to save a generic document in my DB, and raise an exception if this fails:
from db_models import checkElement
def check_connection():
try:
checkElement(checkID=1).save()
saved = True
except Exception as e:
saved = False
raise e
finally:
if saved:
checkElement.objects.get(checkID=1).delete()
I'm pretty sure this is not a very clean way to check for a DB connection, is there a better way to do the same with mongoengine/pymongo ??
I have a setup where a node.js app is making ajax requests to a flask based python server. Since ajax requests lack cookie data, I can't use the simple flask session object to persist data across requests. To remedy this, I'd like to implement a redis based server side implementation of a session storage system, but the solutions I've found so far do not work.
One solution I've tried is the following this snippet.
But this doesn't work. Is there more setup I need to do to configure redis beyond what is mentioned in the quickstart guide? Here is my attempt:
...
from flask import session
# Snippet code is copy pasted here verbatum
import session_interface
...
app = Flask(__name__)
app.session_interface = session_interface.RedisSessionInterface()
...
# Can't access this as session['key'] across requests
session['key'] = value
...
if __name__ == '__main__':
app.secret_key = '123456789012345678901234'
app.run(debug=True)
Another solution I've tried is importing the Flask-Session extention.
However, I can't get this to work either. The section I'm confused about is the following:
"We are not supplying something like SESSION_REDIS_HOST and SESSION_REDIS_PORT, if you want to use the RedisSessionInterface, you should configure SESSION_REDIS to your own redis.Redis instance. This gives you more flexibility, like maybe you want to use the same redis.Redis instance for cache purpose too, then you do not need to keep two redis.Redis instance in the same process."
What is meant by this section and how would I have figured this out? Here is my attempt to make this extension work:
...
from flask import session
from flask_session import Session
import redis
...
app = Flask(__name__)
SESSION_TYPE = 'redis'
app.config.from_object(__name__)
Session(app)
...
# Can't access this as session['key'] across requests
session['key'] = value
...
if __name__ == '__main__':
app.secret_key = '123456789012345678901234'
app.run(debug=True)
Has anyone successfully implemented manual session storage on a server running flask? Are there other options for getting this functionality?
Thanks for your input.
I think that's because you missed the URL configuration for your storage Redis, to check that, you can use Redis-CLI to see if there is anything being inserted into Redis.
I use this code and it worked:
from flask import Flask
from flask_session import Session
import redis
……
app = Flask(__name__)
app.config['SESSION_TYPE'] = 'redis'
app.config['SESSION_REDIS'] = redis.from_url('127.0.0.1:6379')
sess = Session()
sess.init_app(app)
def getSession():
return session.get('key', 'not set')
def setSession():
session.set('key')=123
return 'ok'
……
The following works for me:
...
from flask_session import Session
import redis
...
app = Flask(__name__)
SECRET_KEY = '123456789012345678901234'
SESSION_TYPE = 'redis'
SESSION_REDIS = redis.from_url('localhost:6379')
app.config.from_object(__name__)
sess = Session()
sess.init_app(app)
...
# Should be available accross requests now
session['key'] = value
...
if __name__ == '__main__':
app.run(debug=True)
Using sess.init_app(app) instead of Session(app) did the trick.
How should I use psycopg2 with Flask? I suspect it wouldn't be good to open a new connection every request so how can I open just one and make it globally available to the application?
from flask import Flask
app = Flask(__name__)
app.config.from_object('config') # Now we can access the configuration variables via app.config["VAR_NAME"].
import psycopg2
import myapp.views
Using connection Pooling is needed with Flask or any web server, as you rightfully mentioned, it is not wise to open and close connections for every request.
psycopg2 offers connection pooling out of the box. The AbstractConnectionPool class which you can extend and implement or a SimpleConnectionPool class that can be used out of the box. Depending on how you run the Flask App, you may what to use ThreadedConnectionPool which is described in docs as
A connection pool that works with the threading module.
Creating a simple Flask app and adding a ConnectionPool to it
import psycopg2
from psycopg2 import pool
from flask import Flask
app = Flask(__name__)
postgreSQL_pool = psycopg2.pool.SimpleConnectionPool(1, 20, user="postgres",
password="pass##29",
host="127.0.0.1",
port="5432",
database="postgres_db")
#app.route('/')
def hello_world():
# Use getconn() to Get Connection from connection pool
ps_connection = postgreSQL_pool.getconn()
# use cursor() to get a cursor as normal
ps_cursor = ps_connection.cursor()
#
# use ps_cursor to interact with DB
#
# close cursor
ps_cursor.close()
# release the connection back to connection pool
postgreSQL_pool.putconn(ps_connection)
return 'Hello, World!'
The Flask App itself is not complete or production-ready, please follow the instructions on Flask Docs to manage DB credentials and use the Pool object across the Flask App within the Flask context
I would strongly recommend using Libraries such as SQLAlchemy along with Flask (available as a wrapper) which will maintain connections and manage the pooling for you. Allowing you to focus on your logic
I'm having trouble with the MongoHQ Heroku addon. Locally my app works and the os variable is present and well-formed on Heroku. However, when I attempt to access the db it throws an error: OperationFailure: database error: unauthorized db:my_database ns:my_database.cars lock type:0 client:128.62.187.133. If I try to hard-code the connection string from MongoHQ and run locally, I get the same error.
My app is below:
import os
import datetime
from flask import Flask
from flask import g
from flask import jsonify
from flask import json
from flask import request
from flask import url_for
from flask import redirect
from flask import render_template
from flask import make_response
import pymongo
from pymongo import Connection
from bson import BSON
from bson import json_util
app = Flask(__name__)
def mongo_conn():
# Format: MONGOHQ_URL: mongodb://<user>:<pass>#<base_url>:<port>/<url_path>
if os.environ.get('MONGOHQ_URL'):
return Connection(os.environ['MONGOHQ_URL'])
else:
return Connection()
#app.route('/', methods=['GET', 'POST'])
def hello():
# Get your DB
connection = mongo_conn()
db = connection.my_database
# Create an object
car = {"brand": "Ford",
"model": "Mustang",
"date": datetime.datetime.utcnow()}
# Get your collection
cars = db.cars # crashes
# Insert it
cars.insert(car)
...
Edit: MongoHQ support helped me. Problem was that I was calling my database my_database instead of the actual DB name given to me by the MongoHQ addon. E.g., db = connection.app52314314. That change fixed it.
You likely need to run the authenticate command against the DB directly after you connect.
Try something like this:
db.authenticate([USER], [PASSWORD])
If that doesn't work, feel free to email support#mongohq.com and we can help you out with your specific DB.
You don't need to do all that. You can simply:
from pymongo import MongoClient
client = MongoClient(os.environ['MONGOHQ_URL'])
mongo_db = client.get_default_database()
It will automatically authenticate you, and connect to the provisioned database, the <url_path> part of your connection url.
I'm writing a web application that connects to a database. I'm currently using a variable in a module that I import from other modules, but this feels nasty.
# server.py
from hexapoda.application import application
if __name__ == '__main__':
from paste import httpserver
httpserver.serve(application, host='127.0.0.1', port='1337')
# hexapoda/application.py
from mongoalchemy.session import Session
db = Session.connect('hexapoda')
import hexapoda.tickets.controllers
# hexapoda/tickets/controllers.py
from hexapoda.application import db
def index(request, params):
tickets = db.query(Ticket)
The problem is that I get multiple connections to the database (I guess that because I import application.py in two different modules, the Session.connect() function gets executed twice).
How can I access db from multiple modules without creating multiple connections (i.e. only call Session.connect() once in the entire application)?
Try the Twisted framework with something like:
from twisted.enterprise import adbapi
class db(object):
def __init__(self):
self.dbpool = adbapi.ConnectionPool('MySQLdb',
db='database',
user='username',
passwd='password')
def query(self, sql)
self.dbpool.runInteraction(self._query, sql)
def _query(self, tx, sql):
tx.execute(sql)
print tx.fetchone()
That's probably not what you want to do - a single connection per app means that your app can't scale.
The usual solution is to connect to the database when a request comes in and store that connection in a variable with "request" scope (i.e. it lives as long as the request).
A simple way to achieve that is to put it in the request:
request.db = ...connect...
Your web framework probably offers a way to annotate methods or something like a filter which sees all requests. Put the code to open/close the connection there.
If opening connections is expensive, use connection pooling.