I have an issue with in my Flask app concerning SQLAlchemy and MySQL.
In one of my file: connection.py I have a function that creates a DB connection and set it as a global variable:
db = None
def db_connect(force=False):
global db
db = pymysql.connect(.....)
def makecursor():
cursor = db.cursor(pymysql.cursors.DictCursor)
return db, cursor
And then I have a User Model created with SQL ALchemy models.ppy
class User(Model):
id = column........
It inherits from Model which is a class that I create in another file orm.py
import connection
Engine = create_engine(url, creator=lambda x: connection.makecursor()[0], pool_pre_ping=True)
session_factory = sessionmaker(bind=Engine, autoflush=autoflush)
Session = scoped_session(session_factory)
class _Model:
query = Session.query_property()
Model = declarative_base(cls=_Model, constructor=model_constructor)
In my application I can have long script running so the DB timeout. So I have a function that "reconnect" my DB (it actually only create a new connexion and replace the global DB variable)
My goal is to be able to catch the close of my DB and reconnect it instantly. I tried with SQLAlchemy events but it never worked. (here)
Here is some line that reproduces the error:
res = User.query.filter_by(username="myuser#gmail.com").first()
connection.db.close()
# connection.reconnect() # --> SOLUTION
res = User.query.filter_by(username="myuser#gmail.com").first()
If you guys have any ideas of how to achieve that, let me know 🙏🏻
Oh and I forgot, this application is still running with python2.7.
Related
I'm trying to test my FASTAPI app. Seems to me, all settings are correct.
test_users.py
engine = create_engine(
f"postgresql"
f"://{settings.database_username}"
f":{settings.database_password}"
f"#{settings.database_hostname}"
f":{settings.database_port}"
f"/test_{settings.database_name}"
)
TestingSessionLocal = sessionmaker(autocommit=False, autoflush=False, bind=engine)
Base.metadata.create_all(bind=engine)
def override_get_db():
try:
db = TestingSessionLocal()
yield db
finally:
db.close()
app.dependency_overrides[get_db] = override_get_db
client = TestClient(app)
def test_create_user():
response = client.post(
"/users/",
json={"email": "nikita#gmail.com", "password": "password"}
)
new_user = schemas.UserOutput(**response.json())
assert response.status_code == 201
assert new_user.email == "nikita#gmail.com"
When I run pytest, I get this error:
sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) connection to server at "localhost" (::1), port 5432 failed: FATAL: database "test_social_media_api" does not exist
Why is the code not creating the database?
With engine = create_engine("postgresql://...") you define a connection to an existing PostgreSql database.
And with Base.metadata.create_all(bind=engine) you create the tables - according to your models - in the existing database.
So the code that you written does not create a database, it expects that you give it an already existing database.
And that has to do with PostgreSQL itself.
PostgreSQL runs as a server, and a PostgreSQL server can run multiple databases. And each database has to be created explicitly.
Just telling SQLAlchemy the connection string is not enough.
It's possible to create a new database from Python itself by connecting to the PostgreSQL server (see https://www.tutorialspoint.com/python_data_access/python_postgresql_create_database.htm), or alternatively you can create it manually before you run your script. E.g. by running CREATE DATABASE databasename; inside psql (or any other database tool).
However if you want to test using a running database, I would suggest using testcontainers. They will spawn a new PostgreSQL server with an empty database everytime you run the tests.
Notice, that the example from the FastAPI documentation works differently.
They just use
SQLALCHEMY_DATABASE_URL = "sqlite:///./test.db"
engine = create_engine(
SQLALCHEMY_DATABASE_URL, connect_args={"check_same_thread": False}
)
Base.metadata.create_all(bind=engine)
which creates the database.
This works, because SQLite doesn't run as a server. It's just one file that represents the full database, and if the file doesn't exist, the sqlite database adapter will assume that the database is just empty, and create a new file for you. PostgreSQL doesn't work like this though.
I'm utilizing Flask and SqlAlchemy. The database I've created for SqlAlchemy seems to mess up when I try to run my website and will pop up with the error stating that there's a thread error. I'm wondering if it's because I haven't dropped my table from my previous schema. I'm using a linux server to try and run the "python3" and the file to set up my database.
I've tried to physically delete the table from my local drive and the re run it but I still up this error.
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.orm import scoped_session
from database_setup import Base, Category, Item
engine = create_engine('sqlite:///database_tables.db')
Base.metadata.bind = engine
Session = sessionmaker()
Session.bind = engine
session = Session()
brushes = Category(id = 1, category_name = 'Brushes')
session.add(brushes)
session.commit()
pencils = Category(id = 2, category_name = 'Pencils')
session.add(pencils)
session.commit()
When I am in debug mode using Flask, I click the links I've made using these rows, but after three clicks I get the error
"(sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 140244909291264 and this is thread id 140244900898560 [SQL: SELECT category.id AS category_id, category.category_name AS category_category_name FROM category] [parameters: [{}]] (Background on this error at: http://sqlalche.me/e/f405)"
you can use for each thread a session, by indexing them using the thread id _thread.get_ident():
import _thread
engine = create_engine('sqlite:///history.db', connect_args={'check_same_thread': False})
...
Base.metadata.create_all(engine)
sessions = {}
def get_session():
thread_id = _thread.get_ident() # get thread id
if thread_id in sessions:
return sessions[thread_id]
session_factory = sessionmaker(bind=engine)
Session = scoped_session(session_factory)
sessions[thread_id] = Session()
return sessions[thread_id]
then use get_session() where it is needed, in your case:
get_session().add(brushes)
get_session().commit()
SQLite objects created in a thread can only be used in that same thread.
app = Flask(__name__)
#app.route("/test/")
def test():
conn = sqlite3.connect("god_attributes.db")
c = conn.cursor()
c.execute("SELECT * FROM god_icon_table")
all = c.fetchall()
return render_template("test.html", all = all)
I'm making a flask app and I have a lot of methods that need to pull data from a db using SQL db calls. I am wondering if I can store methods somewhere else and call them by importing to organize things. Basically I want the entire app route for test to be like this:
app = Flask(__name__)
#app.route("/test/")
def test():
all = get_all()
return render_template("test.html", all = all)
where get_all() does everything from conn to fetchall in the first code sample
So I figured out a structural solution. Wasnt able to place a get_all() method that does the SQL commands directly. I made a new json file from the database and used that json to get the information
I'm totally new using sqlalchemy and postgresql. I read this tutorial to build the following piece of code :
import sqlalchemy
from sqlalchemy import create_engine
from sqlalchemy import engine
def connect(user, password, db, host='localhost', port=5432):
'''Returns a connection and a metadata object'''
# We connect with the help of the PostgreSQL URL
# postgresql://federer:grandestslam#localhost:5432/tennis
url = 'postgresql://{}:{}#{}:{}/{}'
url = url.format(user, password, host, port, db)
# The return value of create_engine() is our connection object
con = sqlalchemy.create_engine(url, client_encoding='utf8')
# We then bind the connection to MetaData()
meta = sqlalchemy.MetaData(bind=con, reflect=True)
return con, meta
con, meta = connect('federer', 'grandestslam', 'tennis')
con
engine('postgresql://federer:***#localhost:5432/tennis')
meta
MetaData(bind=Engine('postgresql://federer:***#localhost:5432/tennis'))
When running it I have this error :
File "test.py", line 22, in <module>
engine('postgresql://federer:***#localhost:5432/tennis')
TypeError: 'module' object is not callable
what should I do ? thanks !
So, your problem is happening because you've made this call:
from sqlalchemy import engine
And then you've used this later in the file:
engine('postgresql://federer:***#localhost:5432/tennis')
Strangely, in that section, you have some statements that are just con and meta with no assignments or calls or anything. I'm not sure what you're doing there. I would suggest that you check out SQLalchemy's page on engine and connection use to help get you sorted.
It will of course depend on exactly how you've set up your database. I used the declarative_base module in one of my projects, so my process of setting up a session to connect to my DB looks like this:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Connect to Database and create database session
engine = create_engine('postgresql://catalog:catalog#localhost/menus')
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
And in my database setup file, I've assigned:
Base = declarative_base()
But you'll have to customize it a bit to your particular setup. I hope that helps.
Edit: I see now where those calls to con and meta were coming from, as well as your other confusing lines, it's part of the tutorial you linked to. What he was doing in that tutorial was using the Python interpreter in command line. I'll explain a few of the things he did there in the hope that it helps you some more. Lines beginning with >>> are what he enters in as commands. The other lines are the output he receives back.
>>> con, meta = connect('federer', 'grandestslam', 'tennis') # he creates the connection and meta objects
>>> con # now he calls the connection by itself to have it show that it's connected to his DB
Engine(postgresql://federer:***#localhost:5432/tennis)
>>> meta # here he calls his meta object to show how it, too, is connected
MetaData(bind=Engine(postgresql://federer:***#localhost:5432/tennis))
I tried to totally seperate Flask and SQLAlchemy using this method but Flask still seems to be able to detect my database and start a new transaction at the beginning of each request.
The db.py file creates a new session and defines a simple model of a table:
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, String
engine = create_engine("mysql://web:kingtezdu#localhost/web_unique")
print("creating new session")
db_session = scoped_session(sessionmaker(bind=engine))
Base = declarative_base()
Base.query = db_session.query_property()
# define model of 'persons' table
class Person(Base):
__tablename__ = "persons"
name = Column(String(30), primary_key=True)
def __repr__(self):
return "Person(\"{0.name}\")".format(self)
# create table
Base.metadata.create_all(bind=engine)
And app.py, a simple Flask application using the SQLAlchemy session and model:
from flask import Flask, escape
app = Flask(__name__)
# importing new session
from db import db_session, Person
# registering for app teardown to remove session
#app.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
#app.route("/query")
def query():
# query all persons in the database
all_persons = Person.query.all()
print all_persons
return "" # we use the console output
if __name__ == "__main__":
app.run(debug=True)
Let's run this:
$ python app.py
creating new session
* Running on http://127.0.0.1:5000/
* Restarting with reloader
creating new session
Weired enough it runs db.py two times but we just ignore this, let's access the webpage /query:
[]
127.0.0.1 - - [23/Dec/2015 18:20:14] "GET /query HTTP/1.1" 200 -
We can see that our request was answered, though we only use the console output. There is no Person in the database yet, let's add one:
mysql> INSERT INTO persons (name) VALUES ("Marie");
Query OK, 1 row affected (0.11 sec)
Marie is part of the database now so we reload the webpage:
[Person("Marie")]
127.0.0.1 - - [23/Dec/2015 18:24:48] "GET /query HTTP/1.1" 200 -
As you can see the session already knows about Marie. Flask didn't create a new session. That means that there was a new transaction started. Contrast this to the plan python example below to see the difference.
My question is how Flask is able to start a new transaction on the begin of each request. Flask shouldn't know about the database but seems to be able to change something about it's behaviour.
In case you don't know what a SQLAlchemy transaction is read this paragraph extracted from Managing Transactions:
When the transactional state is completed after a rollback or commit,
the Session releases all Transaction and Connection resources, and
goes back to the “begin” state, which will again invoke new Connection
and Transaction objects as new requests to emit SQL statements are
received.
So a transaction is ended by a commit and will cause a new connection to be set up which will then make the session read the database again. In reality this means that you have to commit when you want to see changes made to the database:
First in interactive python mode:
>>> from db import db_session, Person
creating new session
>>> Person.query.all()
[]
Switch over to MySQL and insert a new Person:
mysql> INSERT INTO persons (name) VALUES ("Paul");
Query OK, 1 row affected (0.03 sec)
Finally try to load Paul into our session:
>>> Person.query.all()
[]
>>> db_session.commit()
>>> Person.query.all()
[Person("Paul")]
I think the issue here is that scoped_session somewhat hides what happens to the actual sessions in use. When your teardown handler
# registering for app teardown to remove session
#app.teardown_appcontext
def shutdown_session(exception=None):
db_session.remove()
runs at the end of each request, you call db_session.remove() which disposes of the session used in that particular request along with any transaction context. See http://docs.sqlalchemy.org/en/latest/orm/contextual.html for the details, particularly
The scoped_session.remove() method first calls Session.close() on the
current Session, which has the effect of releasing any
connection/transactional resources owned by the Session first, then
discarding the Session itself. “Releasing” here means that connections
are returned to their connection pool and any transactional state is
rolled back, ultimately using the rollback() method of the underlying
DBAPI connection.