When running tests in pytest, the database is modified. What is the best way to undo changes to the database?
DBSession rollback
For those tests where I can access the backend directly, I currently use pytest fixture to start a new DBSession for every test function, and rollback the session at the end of it
#pytest.fixture(scope='session')
def db(app, request):
"""Session-wide test database."""
def teardown():
pass
_db = SQLAlchemy(app)
return _db
#pytest.fixture(scope='function')
def db_session(db, request):
"""Creates a new database session for a test."""
engine = create_engine(
TestConfig.SQLALCHEMY_DATABASE_URI,
connect_args={"options": "-c timezone=utc"})
DbSession = sessionmaker(bind=engine)
session = DbSession()
connection = engine.connect()
transaction = connection.begin()
options = dict(bind=connection, binds={})
session = db.create_scoped_session(options=options)
db.session = session
yield session
transaction.rollback()
connection.close()
session.remove()
In the test code, I simply use the fixture
def test_create_project(db_session):
project = _create_test_project(db_session)
assert project.project_id > 0
Flask / HTTP Testing
But for testing the API via Flask/HTTP, I cannot use db_session. Even when I create a fixture to explicitly DROP the test database and restore from production, it will not work because there is no direct database code
#pytest.fixture(scope='function')
def db_session_refresh(db, request):
"""Refresh the test database from production after running the test"""
engine = create_engine(
TestConfig.SQLALCHEMY_DATABASE_URI,
connect_args={"options": "-c timezone=utc"})
DbSession = sessionmaker(bind=engine)
session = DbSession()
connection = engine.connect()
transaction = connection.begin()
options = dict(bind=connection, binds={})
session = db.create_scoped_session(options=options)
db.session = session
yield session
transaction.rollback()
connection.close()
session.remove()
refresh_test_db_sql = """
SELECT pg_terminate_backend(pid)
FROM pg_stat_activity
WHERE datname = 'appdb_test';
DROP DATABASE appdb_test;
CREATE DATABASE appdb_test TEMPLATE appdb;
"""
engine.execute(refresh_test_db_sql)
Even if this works, it is inefficient to refresh the database for every function.
What is the proper/better way to run test that modifies the database?
as commented earlier - creation and destruction of the DB should be taken care out of you unit test and moved into a wrapper.
but to answer you question - I have the impression that you're using postgresql. try removing the database name from your connection string and connect to it directly and add it to the search_path
Related
I am using FastAPI to build an API backend for my URL shortener. I have the database connected to the API as a dependency. For the CRUD operations, I am using the SQL Alchemy ORM.
The code for my main app works perfectly fine and performs all the major CRUD operations I have mapped through the API endpoints.
The problem arises when I try to override the DB dependency to use a test db instead of my production db for testing purposes.
There are no errors associated with this override, however, the test database does not contain any of the tables that would be created when Base.metadata.creat_all(bind=engine) is called.
When running the tests using pytest, it gives me this error:
sqlalchemy.exc.ProgrammingError: (pymysql.err.ProgrammingError) (1146, "Table 'testurldb.urls' doesn't exist")
The code for my tests:
engine = create_engine(
"mysql+pymysql://{user}:{password}#{ip}:{port}/testurldb".format(
user=user, password=password, ip=ip, port=port
)
)
Session = sessionmaker(bind=engine)
Base.metadata.create_all(bind=engine)
def overrideDB():
db = Session()
try:
yield db
finally:
db.close()
app.dependency_overrides[get_db] = overrideDB
client = TestClient(app)
The module where Base is instantiated:
engine = create_engine(
"mysql+pymysql://{root}:{password}#{ip}:{port}/urldb".format(
root=root, password=password, ip=ip, port=port
)
)
SessionLocal = sessionmaker(bind=engine)
Base = declarative_base()
The table that extends Base:
class URL(Base):
__tablename__ = "urls"
short_url = Column(String(256), primary_key=True, unique=True, nullable=False)
long_url = Column(String(256), nullable=False, unique=True)
time = Column(String(256), nullable=False)
def __init__(self, short_url, long_url, time):
self.short_url = short_url
self.long_url = long_url
self.time = time
There seems to be nothing wrong with the imports, so I don't understand why it's not creating the tables.
And second, it might be useful information that my main production db already has the Tables created.
Create a file and import all models into it, at the end of all imports, put the Model import. When doing this, try to create the models again through Base.metadata
i hope it'll helps you
db.py
Base = declarative_base()
async def init_db():
try:
Base.metadata.create_all(bind=engine)
except Exception as e:
raise e
main.py
#app.on_event("startup")
async def on_startup():
await init_db()
I am working on testing my flask app model. I'm using mysql 5.7, sqlalchemy and pytest.
Within my model, I have a CRUD mixin that I used to manage creating, updating and deleting. Whenever I try to access the object in the Mixin before returning the object to the test function, SQLAlchemy hangs at db.drop_all in my tear down. When I look in mysql at PROCESSLIST, it shows 1 sleep query and 1 query waiting for table metadata lock.
I can fix this by calling db.session.commit in the create method in the mixin before returning the object. However, if I call it at the test teardown (or in the main test function), it doesn't work. I'd prefer not to add an extra commit just to make my tests work as it doesn't feel correct. Does anyone know why this is happening or have any suggested fixes?
models.py
class CRUDMixin(object):
#classmethod
def create(cls, **kwargs):
instance = cls(**kwargs)
saved_instance = instance.save()
# do stuff with saved_instance (i.e. add to full text search engine)
# db.drop_all in teardown works if add db.session.commit() here
return saved_instance
def save(self, commit=True):
db.session.add(self)
if commit:
try:
db.session.commit()
except Exception:
db.session.rollback()
raise
return self
class User(CRUDMixin, db.model):
id = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(50))
conftest.py
#pytest.fixture(scope='session')
def app():
app = create_app(TestConfig)
ctx = app.app_context()
ctx.push()
yield app
ctx.pop()
#pytest.fixture(scope='session')
def test_db(app):
db.drop_all()
db.create_all()
# add test db information
yield db
db.session.remove()
db.drop_all() # test hangs within drop all
#pytest.fixture(scope='function')
def db_session(test_db):
connection = db.engine.connect()
transaction = connection.begin()
options = dict(bind=connection, binds={})
session = db.create_scoped_session(options)
db.session = session
yield db
db.session.remove() # tables won't drop if I put db.session.commit before the remove call
transaction.rollback()
connection.close() # even though connection closes, mysql still shows process
test_models.py
class TestUser(object):
def test_add_new(self, db_session):
u = User.create(name='test_name')
assert u.name == 'test_name'
# if I put db.session.commit() here, tables won't drop
This is my first time writing a web application and using SQLAlchemy and I'm not sure I completely understand the concept of sessions. Currently I am loading a new session whenever the db needs to be queried. Is it sufficient to close it with sql_session.close() as I have done below?
Does not closing it cause many problems?
engine = create_engine('sqlite:///database.db', echo=True)
Base = declarative_base(engine)
class Kinases(Base):
__tablename__ = 'Kinase'
full_name = Column(String)
uniprot_code = Column(String)
def loadSession():
metadata = Base.metadata
Session = sessionmaker(bind=engine)
session = Session()
return session
#app.route("/search/kinases/<query>")
def kinase_results(query):
sql_session = loadSession()
kinase = sql_session.query(Kinases).get(query)
if kinase is None:
return redirect(url_for('user_message', query=query))
name = kinase.full_name
sql_session.close()
In most cases, creating session in scope of view is a bad idea. Please read session basics for sqla.
I am trying sqlalchemy with pytest, and having the following issues
#pytest.fixture(scope='function')
def my_session(my_db, request):
from my.models import Session, Base
Base.metadata.bind = my_db
Base.metadata.create_all()
def teardown():
Base.metadata.drop_all()
Base.metadata.create_all()
request.addfinalizer(teardown)
Session.configure(bind=my_db)
return Session()
But for some reason, the data which was stored to database during previous tests is still there. And I was kind of expecting it to vanish after .drop_all() call :(
I am trying convert unit test into py test. I am using the unit test example
class TestCase(unittest.TestCase):
def setUp(self):
app.config['TESTING'] = True
app.config['CSRF_ENABLED'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir,
'test.db')
db.create_all()
def tearDown(self):
db.session.remove()
db.drop_all()
I am not sure, What should be its py test version.
I searched high and low for a well explained solution to use SqlAlchemy without Flask-SQLAlchemy and run tests with Pytest, so here's how i have achieved this:
Set up your engine & Session objects as per the docs. (I have opted for sessionmaker as i want to check in my app if the session is still available in the Flask's request thread pool, see: https://dev.to/nestedsoftware/flask-and-sqlalchemy-without-the-flask-sqlalchemy-extension-3cf8
Import your Base object from wherever you've created it in your app. This will create all the tables in your database defined by the engine.
Now we want to Yield a Session back to your unit tests. The idea is to setup before calling Yield & teardown after. Now, in your test you can create a table and populate it with some rows of data etc.
Now we must close the Session, this is important!
Now by calling Base.metadata.drop_all(bind=engine) we drop all the tables in the database ( we can define a table(s) to drop if required, default is: tables=None)
engine = create_engine(create_db_connection_str(config), echo=True)
Session = scoped_session(sessionmaker(bind=engine))
#pytest.fixture(scope="function") # or "module" (to teardown at a module level)
def db_session():
Base.metadata.create_all(engine)
session = Session()
yield session
session.close()
Base.metadata.drop_all(bind=engine)
Now we can pass the function scoped fixture to each unit test:
class TestNotebookManager:
"""
Using book1.mon for this test suite
"""
book_name = "book1"
def test_load(self, client: FlaskClient, db_session) -> None:
notebook = Notebook(name=self.book_name)
db_session.add(book)
db_session.commit()
rv = client.get(f"/api/v1/manager/load?name={self.name}")
assert "200" in rv.status
First off, py.test should just run the existing unittest test case. However the native thing to do in py.test is use a fixture for the setup and teardown:
import pytest
#pytest.fixture
def some_db(request):
app.config['TESTING'] = True
app.config['CSRF_ENABLED'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///' + os.path.join(basedir, 'test.db')
db.create_all()
def fin():
db.session.remove()
db.drop_all()
request.addfinalizer(fin)
def test_foo(some_db):
pass
Note that I have no idea about SQLAlchemy and whether there are better ways of handling it's setup and teardown. All this example demonstrates is how to turn the setup/teardown methods into a fixture.