I have deployed my flask application on apache+mod_wsgi
I'm using WSGI Daemon mode and have this config in apache httpd.conf:
WSGIDaemonProcess flask_test user=apache group=apache threads=20
For simplicity lets say for each request, I need to execute a query to insert data into Oracle DataBase.
So in my flask application, I have done something like this:
# DB.py
import cx_Oracle
class DB:
def __init__(self, connection_string):
self.conn = cx_Oracle.connect(connection_string, threaded=True)
def insert(query):
cur = self.conn.cursor()
cur.execute(query)
cur.close()
self.conn.commit()
# flask_app.py
from flask import Flask, request, jsonify
from DB import DB
app = Flask(__name__)
db = DB(connection_string)
#app.route("/foo", methods=["POST"])
def foo():
post_data = request.get_json()
# parse above data
# create insert query with parsed data values
db.insert(insert_processed_data_QUERY)
# generate response
return jsonify(response)
When I start the apache+mod_wsgi server, the DB object is created and the DB connection is established.
For all incoming requests, the same DB object is used to execute insert query.
So far this works fine for me. However my concern is that if there are no requests for a long period of time, the DB connection might time out, and then my app will not work for a new request when it comes.
I've been monitoring my application and have observed that the DB connection persists for hours and hours. But I'm pretty sure it might timeout if there is no request for 2-3 days(?)
What would be the correct way to ensure that the DB connection will stay open forever? (i.e. as long as the apache server is running)
Use a pool instead of a standalone connection. When you acquire a connection from the pool it will check to see if the connection is no longer valid and automatically dispense a new one. So you need something like this:
pool = cx_Oracle.SessionPool(user=user, password=password, dsn=dsn, min=1,
max=2, increment=1)
Then in your code you need to do the following:
with pool.acquire() as connection:
# do what you need to do with the connection
Related
Here is my python flask code:
from flask import *
import mysql.connector
app = Flask(__name__)
conn = mysql.connector.connect(
host="<my db host>",
user="<my db user>",
password = "<my db password>",
database = '<my db>'
)
#app.route('/posts/<int:post_id>')
def get_post(post_id):
with conn.cursor(dictionary=True) as cur:
cur.execute('select * from posts where ID=%s',(post_id,))
result = cur.fetchone()
ans=result['post_content']
return ans
app.run(debug=False,threaded=True,host='0.0.0.0',port=80)
Note how I don't create a new connection for each request. Instead, I use the same connection for all requests.
My question is: is there any potential problems in this approach?
You should not use the same connection, because it will stay open forever, only open an SQL connection when you need to use it, and close it afterward. Not doing so can lead to a lot of errors.
I am using Flask-MySQL to connect to my database in a view. The view works the first time I go to it, but when I go to it the second time it always crashes with the error:
ProgrammingError: closing a closed connection
Why am I getting this error? How do I connect successfully the second time?
from flask import Flask, render_template, request
from flaskext.mysql import MySQL
app = Flask(__name__)
#app.route('/hello/', methods=['POST'])
def hello():
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'xxx'
app.config['MYSQL_DATABASE_DB'] = 'pies'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
query = request.form['yourname']
mysql = MySQL(app)
conn = mysql.connect()
with conn as cursor:
try:
cursor.execute(query)
name = str(cursor.fetchone())
except:
name = "SQL is wrong"
conn.close()
return render_template('form_action.html', name=name)
if __name__ == '__main__':
app.run(debug=True)
You should not be initializing the extension during every request. Create it during app setup, then use it during the requests. Setting configuration should also be done during app setup.
The extension adds a teardown function that executes after every request and closes the connection, which is stored as a threadlocal. Since on the second request you've registered the extension multiple times, it is trying to close the connection multiple times.
There's no need to call connect, the extension adds a before request function that does that. Use get_db to get the connection for the request. Since the extension closes this connection for you, don't call close either.
from flask import Flask
from flaskext.mysql import MySQL
MYSQL_DATABASE_USER = 'root'
MYSQL_DATABASE_PASSWORD = 'xxx'
MYSQL_DATABASE_DB = 'pies'
MYSQL_DATABASE_HOST = 'localhost'
app = Flask(__name__)
app.config.from_object(__name__)
mysql = MySQL(app)
#app.route('/hello/')
def hello():
conn = mysql.get_db()
# ...
Note that Flask-MySQL is using a very old extension pattern that is no longer supported by Flask. It also depends on an unsupported mysql package that does not support Python 3. Consider using Flask-MySQLdb instead.
I've the code like this below. Is it necessary to close mysql connection because whenever my home page is requested, a new sql connection will be created?
I randomly get Connection Limit error. But I'm not sure if the DB connection is the problem.
#app.route("Home", methods=["GET"])
def get_home_page():
db = mysql.connect(host, user, password, db_name, charset='utf8', use_unicode=True)
...
It is good practice to close the connection. You can put your codes inside a try..finally block.
#app.route("Home", methods=["GET"])
def get_home_page():
db = mysql.connect(host, user, password, db_name, charset='utf8', use_unicode=True)
try:
... do something ...
finally:
db.close()
from my experience, close session after use it take significant difference amount of time to response in api whom i've experienced in flask
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
try:
db.session.query( Any Model ...
except:
finally:
db.close_all_sessions
code:
def before_request():
try:
g.db = mysql.connector.connect(user=user1,password=pass1, host=hos1,
database=testdb1)
g.db1 = mysql.connector.connect(user=user2,password=pass2, host=host2,
database=testdb2)
#app.route('/user/<db>', methods=['POST'])
def insert(db):
body=request.json
try:
#Http request for inserting data into testdb1:"curl 'http://localhost:5000/user/**testdb1**' testdb1 is the database".
same way ##Http request for inserting data into testdb2:"curl 'http://localhost:5000/user/**testdb2**' testdb2 is 2nd database".
#code for inserting data into database(testdb1/testdb2) according to http request.
Everything is working fine but the problem is when i am sending (http request)
for lets say for inserting data into testdb1.then the connection to both of the databases get opened.But i want that only the requested database connection should get opened not both.
Question:1.What changes i need to make that only one connection get opened?
2.Is it possible that to have g.db as a connection variable ,which store both connection and when http request comes for the database.Let say for testdb1 then only that connection get opened?
You should set do it like this. when you start your server, you make two global connection for doing sth or like that:
For example,
conn1 = None
conn2 = None
def connection1():
global conn1
if conn1 is None:
conn1 = ...
return conn1
if u want to sql,
from initConnection import conn1, conn2
conn1.execute(sql) or conn2.execute()
Only one connection1 or connection2 is using.
I'm developing on heroku using their Postgres add-on with the Dev plan, which has a connection limit of 20. I'm new to python and this may be trivial, but I find it difficult to abstract the database connection without causing OperationalError: (OperationalError) FATAL: too many connections for role.
Currently I have databeam.py:
import os
from flask import Flask
from flask.ext.sqlalchemy import SQLAlchemy
from settings import databaseSettings
class Db(object):
def __init__(self):
self.app = Flask(__name__)
self.app.config.from_object(__name__)
self.app.config['SQLALCHEMY_DATABASE_URI'] = os.environ.get('DATABASE_URL', databaseSettings())
self.db = SQLAlchemy(self.app)
db = Db()
And when I'm creating a controller for a page, I do this:
import databeam
db = databeam.db
locations = databeam.locations
templateVars = db.db.session.query(locations).filter(locations.parent == 0).order_by(locations.order.asc()).all()
This does produce what I want, but slowly and at times causes the error metioned above. Since I come from a php background I have a certain mindset of how to deal with DB connections (I.e. like the example above), but I fear it doesn't fit well with python.
What is the proper way of abstracting the db connection in one place and then just using the same connection in all imports?
Within SQL Alchemy you should be able to create a connection pool. This pool is what the pool size would be for each Dyno. On the Dev and Basic plan since you could have up to 20, you could set this at 20 if you run 1 dyno, 10 if you run 2, etc. To configure your pool you can setup the engine:
engine = create_engine('postgresql://me#localhost/mydb',
pool_size=20, max_overflow=0)
This sets up your db engine with a pool which you pull from automatically then. You can also configure the pool manually, more details on that can be found on the pooling guide of SQL Alchemy - http://docs.sqlalchemy.org/en/latest/core/pooling.html