I have a simple Flask web app that uses peewee to manage the MySQL connection. Unfortunately it seems I missed something important from my code, therefore I'm getting an pymysql.err.OperationalError: (2006, "MySQL server has gone away.. error after 10-20 minutes of using the site. If I restart the app it works fine again, so I assume I handle the connection/connections wrong.
I have some basic queries that I'm using to display lists on my UI. Since I'm new to Python it's possible that my whole logic is wrong, so I would be really happy if somebody could me explain what is the right way to manage (connect-close connection) queries with peewee in my case (simple website without any user functions).
You can see on the below code how I'm doing it now. It's fine until I connect, but after I connected to the db I'm just calling the queries without dealing the connection. I assume I should close the connection based on the query that has been called, but couldn't find a proper way. Calling the myDB.close() isn't enough or I'm using it in wrong.
# this is how I connect to the MySQL
import peewee as pw
from flask import Flask, request, session, g, redirect, url_for, abort, render_template, flash
myDB = pw.MySQLDatabase("", host="", user="", passwd="")
class MySQLModel(pw.Model):
"""A base model that will use our MySQL database"""
class Meta:
database = myDB
class city(MySQLModel):
...
class building(MySQLModel):
...
myDB.connect()
# this is how I manage the homepage
cityList = []
cityList = city.select().order_by(city.objectId).limit(15)
buildings = []
buildings = building.select().order_by(building.buildingName).limit(180)
def getSearchResult (content_from_url):
searchResultList = []
searchResultList = city.select().where(city.objTitle.contains(content_from_url))
return searchResultList
#app.route('/', methods=['GET', 'POST'])
def main_page():
search = request.args.get('search')
if search:
return render_template("results.html", search = getSearchResult(search), str = search)
else:
return render_template("home.html", name=cityList, buildList=buildings)
# and this is how the subpage
relatedCityList = []
slugObj = []
buildings2 = []
buildings2 = building.select().order_by(building.buildingName).limit(180)
def getLink (title_for_link):
slugObj = city.get(city.urlSlug == title_for_link)
return slugObj
def displayRelatedCity (city_to_filter):
resultCity = city.get(city.urlSlug == city_to_filter)
relatedCityList = city.select().where(city.objTitle == resultCity.objTitle).limit(20)
return relatedCityList
#app.route('/city-list/<content>', methods=['GET', 'POST'])
def city_page(content):
linkText = getLink(content)
return render_template("details.html", relatedCity = displayRelatedCity(content), buildingName = linkText.buildingName, buildz = buildings2)
myDB.close()
You need to be reconnecting on request/response. http://docs.peewee-orm.com/en/latest/peewee/database.html#adding-request-hooks
There's even a section on Flask.
Related
SQLite objects created in a thread can only be used in that same thread.
app = Flask(__name__)
#app.route("/test/")
def test():
conn = sqlite3.connect("god_attributes.db")
c = conn.cursor()
c.execute("SELECT * FROM god_icon_table")
all = c.fetchall()
return render_template("test.html", all = all)
I'm making a flask app and I have a lot of methods that need to pull data from a db using SQL db calls. I am wondering if I can store methods somewhere else and call them by importing to organize things. Basically I want the entire app route for test to be like this:
app = Flask(__name__)
#app.route("/test/")
def test():
all = get_all()
return render_template("test.html", all = all)
where get_all() does everything from conn to fetchall in the first code sample
So I figured out a structural solution. Wasnt able to place a get_all() method that does the SQL commands directly. I made a new json file from the database and used that json to get the information
I am trying to connect my flask application to a DB2 database,but somehow I am unable to do. here is what I am trying to do,
I am using flask extension flask_db2 for this purpose and below is how I am trying to connect
from flask import Flask
from flask_db2 import DB2
app = Flask(__name__)
#app.route('/')
def index()
cur = db.connection.cursor("DATABASE=name;HOSTNAME=host;PORT=60000;PROTOCOL=TCPIP;UID=username;PWD=password)
from flask import Flask
from flask_db2 import DB2
app = Flask(__name__)
app.config['DB2_DATABASE'] = 'sample'
app.config['DB2_HOSTNAME'] = 'localhost'
app.config['DB2_PORT'] = 50000
app.config['DB2_PROTOCOL'] = 'TCPIP'
app.config['DB2_USER'] = 'db2inst1'
app.config['DB2_PASSWORD'] = 'db2inst1'
db = DB2(app) #You forgot that
#app.route('/')
def index():
cur = db.connection.cursor()
cur.execute(...)
First: You forgot to have an object to use the flask db2 module.
The DB2(app) is a constructor for your Flask extension and will bind your application to this module.
where as
db = DB2(app)
Initialises your db object with help of your extension's constructor which you can further use for your Database operations.
Secondly: use app configs to play with DB2 credentials. (This step should be done before you bind the app)
Third: You can execute your query like this
cur.execute("SELECT id FROM users") #something like that
I hope I have clarified :D just drop message. Ill try helping more.
I am using Flask-MySQL to connect to my database in a view. The view works the first time I go to it, but when I go to it the second time it always crashes with the error:
ProgrammingError: closing a closed connection
Why am I getting this error? How do I connect successfully the second time?
from flask import Flask, render_template, request
from flaskext.mysql import MySQL
app = Flask(__name__)
#app.route('/hello/', methods=['POST'])
def hello():
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'xxx'
app.config['MYSQL_DATABASE_DB'] = 'pies'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
query = request.form['yourname']
mysql = MySQL(app)
conn = mysql.connect()
with conn as cursor:
try:
cursor.execute(query)
name = str(cursor.fetchone())
except:
name = "SQL is wrong"
conn.close()
return render_template('form_action.html', name=name)
if __name__ == '__main__':
app.run(debug=True)
You should not be initializing the extension during every request. Create it during app setup, then use it during the requests. Setting configuration should also be done during app setup.
The extension adds a teardown function that executes after every request and closes the connection, which is stored as a threadlocal. Since on the second request you've registered the extension multiple times, it is trying to close the connection multiple times.
There's no need to call connect, the extension adds a before request function that does that. Use get_db to get the connection for the request. Since the extension closes this connection for you, don't call close either.
from flask import Flask
from flaskext.mysql import MySQL
MYSQL_DATABASE_USER = 'root'
MYSQL_DATABASE_PASSWORD = 'xxx'
MYSQL_DATABASE_DB = 'pies'
MYSQL_DATABASE_HOST = 'localhost'
app = Flask(__name__)
app.config.from_object(__name__)
mysql = MySQL(app)
#app.route('/hello/')
def hello():
conn = mysql.get_db()
# ...
Note that Flask-MySQL is using a very old extension pattern that is no longer supported by Flask. It also depends on an unsupported mysql package that does not support Python 3. Consider using Flask-MySQLdb instead.
I have been learning how to use the Flask framework using a tutorial, but the code in my app.py keeps returning an error 500, and I can't figure out why (my code is identical to the tutorial).
Here's the app.py:
from flask import Flask, render_template, json, request
from flask.ext.mysql import MySQL
from werkzeug import generate_password_hash, check_password_hash
mysql = MySQL()
app = Flask(__name__)
# MySQL configurations
app.config['MYSQL_DATABASE_USER'] = 'root'
app.config['MYSQL_DATABASE_PASSWORD'] = 'root'
app.config['MYSQL_DATABASE_DB'] = 'BucketList'
app.config['MYSQL_DATABASE_HOST'] = 'localhost'
mysql.init_app(app)
#app.route('/')
def main():
return render_template('index.html')
#app.route('/showSignUp')
def showSignUp():
return render_template('signup.html')
#app.route('/signUp',methods=['POST','GET'])
def signUp():
try:
_name = request.form['inputName']
_email = request.form['inputEmail']
_password = request.form['inputPassword']
# validate the received values
if _name and _email and _password:
# All Good, let's call MySQL
conn = mysql.connect()
cursor = conn.cursor()
_hashed_password = generate_password_hash(_password)
cursor.callproc('sp_createUser',(_name,_email,_hashed_password))
data = cursor.fetchall()
if len(data) is 0:
conn.commit()
return json.dumps({'message':'User created successfully !'})
else:
return json.dumps({'error':str(data[0])})
else:
return json.dumps({'html':'<span>Enter the required fields</span>'})
except Exception as e:
return json.dumps({'error':str(e)})
return traceback.format_exc()
finally:
cursor.close()
conn.close()
if __name__ == "__main__":
app.run(port=5002)
It's for a signup system.
a 500 error usually means there is an error in your python instead when you run try it withh app.run(port=5002,debug=True) this wont solve your problem ... but it should tell you whats going on
I know you are following this tutorial, because I am having the same problem - http://code.tutsplus.com/tutorials/creating-a-web-app-from-scratch-using-python-flask-and-mysql--cms-22972
The issue is that inside the stored procedure they are having you set a column of size 20:
CREATE DEFINER=`root`#`localhost` PROCEDURE `sp_createUser`(
IN p_name VARCHAR(20),
IN p_username VARCHAR(20),
IN p_password VARCHAR(20)
)
But when they tell you to salt the password in your python code, like you do:
_hashed_password = generate_password_hash(_password)
You are creating a string much longer than 20 characters, so if you ran this in debug mode you'd see the error says invalid column length for column password. I fixed this by just changing the size of the column to 100. :)
I know this tutorial and I was getting the same error a couple of minutes back.
I changed -
_hashed_password = generate_password_hash(_password)
to
_hashed_password = _password
and it worked! :).
I am assuming that the reason is size that we have declared for password field is less than what is actually needed if we hash. But for now, you can do the same and get the app running.
Happy Coding!
So,a after reading every page around for the last 1h i'm still not able to find a solution for this problem.
This is my connection.py file:
from sqlalchemy import create_engine, Table, Column, String, Integer, MetaData
from sqlalchemy.sql import select
class DatabaseConnectionManager:
def __init__(self):
self.host = 'localhost:3306'
self.db = 'xxx'
self.user = 'xxx'
self.pwd = 'xxx'
self.connect_string = 'mysql://{u}:{p}#{s}/{d}'.format(u=self.user,
p=self.pwd,
s=self.host,
d=self.db)
self.metadata = MetaData()
self.engine = create_engine(self.connect_string, echo=False,
pool_size=100, pool_recycle=3600)
self.conn = self.engine.connect()
def insert_table(self, inputs):
self.conn.execute(self.tbl_auctions().insert(), inputs)
# Update 1 : conn.close() removed.
#self.conn.close()
def select_servers(self):
try:
s = select([self.tbl_servers()])
result = self.conn.execute(s)
except:
raise
else:
return result
And this is my bulk_inserter.py file:
import sys
import time
import json
from connector import DatabaseConnectionManager
def build_auctions_list():
server_list = []
db = DatabaseConnectionManager()
# Loop over regions
for server, env in db.select_servers_table():
request_auction_data = json.loads(dump_auction_url(region, realm))
for auction in request_auction_data:
auction_list.append(auction)
db.insert_table(server_list)
if __name__ == '__main__':
start = time.time()
build_auctions_list()
print time.time() - start
So, the problem happens when i try to insert all the bulk data using db.insert_table(server_list) for 2 or more servers returned by the loop for server, env in db.select_servers_table():
But, if the result on that loop is for only one server, the flows happen normally without problems.
So, resuming :
this program retrieve a list of servers from a db table and dumps the json data into the db.
Bulk insert performs well if only one server is retrieved.
if two or more servers , the following error happens:
sqlalchemy.exc.OperationalError: (OperationalError) (2006, 'MySQL server has gone away')
Anyoen have any idea what could be happening? I allready incresed the timeout and buffer size on mysql config file. So i'm not sure what the problem could be...
Update #1 It seems i cant bulk insert array with more than 50k values. I'm still trying to figure out how to do it.