I have a database class in my module, db.py that I can only call once, my design goal is to have one single database object and have that be used across all other modules. So db has the layout
#db.py
Class DatabaseManager
def __init__():
# initialize engine and database
db = DatabaseManager()
The problem is across multiple imports db is reinitialized each time, what I want to be able to do is something along the lines of:
# polygon.py
from db import db
class Polygon:
def something(self):
db.commitChange(...)
# main.py
class GUIWindow:
def something(self):
db.getJSON(...)
How can I create one object for the entire program, and have all other modules importing db use that one object? I was under the impression db would not be reinitialized, but I am receiving the engine initialization output twice, here's an example and my output
# db.py
class DatabaseManager(object):
'''
classdocs
'''
def __init__(self):
'''
Constructor
'''
print "hi"
db = DatabaseManager()
# polygon.py
from db import db
# main.py
from db import db
output:
hi
hi
Related
I have 3 classes in my project. One is a DB class where I have the DB connection for my project, which looks like this.
class DB:
def __init__(self):
self.createConnection()
def createConnection(self):
db = DataBase(
[config.endpoint],
http_auth=(config.username, config.password),
scheme="https"
)
self.__db = db
Now in other classes I want to access and use self.__db How can I use this? My 2nd class is a helper class:
import db
class Helper:
def connection(self):
db = db.createConnection()
self.__db = db
print(self.__db)
The intent: Refactor my code into MVC (this is just the model/database part), and have the server create the database with tables on first run if the database or tables does not exist.
This works when using a "flat" file with all the classes and functions defined in that file, but after moving out the functions into a service class and the models into their own folder with model classes, the db.create_all() function does not seem to be able to detect the table class correctly any more.
Example structure, (minimum viable problem):
server.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///database.sqlite'
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
db = SQLAlchemy(app)
def main():
# Intentionally moved into the main() function to prevent import loops
from services.users import UserService
users = UserService(db)
db.create_all()
app.run(debug=True)
if __name__ == '__main__':
main()
services\users.py
# Class used to access users data using a session
from models.users import Users
class UserService:
def __init__(self, db):
self.db = db
def get_all(self):
return self.db.session.query(Users).all()
def get(self, uid):
return self.db.session.query(Users).get(uid)
def add(self, json):
user = Users(email=json['email'], password=json['password'])
self.db.session.add(user)
self.db.session.commit()
return user
models\users.py
# The actual model
from server import db
class Users(db.Model):
_id = db.Column("id", db.Integer, primary_key=True)
email = db.Column(db.Text)
password = db.Column(db.Text)
Result: The database is created, but it is just an empty file with no tables inside of it.
I have also tried placing the db.create_all() inside the service class def __init__(self, db) (grasping at straws here), both as a self reference and as an argument reference. Neither have worked.
I am sure it is something obvious I am missing, but I have boiled down my project to just the bare minimum and still fail to see why it is not working - so I have to ask. How can I get the db.create_all() to detect my table classes correctly and actually create the required tables, while using this code structure (or something similar, in case I have misunderstood MVC)?
The problem is that server.py is executed twice
when it's imported in models/users.py
when server.py is called to run the app
Each execution generates a new db instance. The db imported by the model file adds the models to its metadata, the db created when the app is run has empty metadata.
You can confirm this by printing id(db) and db.metadata.tables at the end of models/users.py and just before the call to db.create_all() in the main function.
You need to structure your code so that only one db gets created. For example, you could move the app configuration and creation code into its own module, mkapp.py (feel free to come up with a better name):
mkapp.py
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///database.sqlite'
app.config["SQLALCHEMY_TRACK_MODIFICATIONS"] = False
app.config['SQLALCHEMY_ECHO'] = True
db = SQLAlchemy(app)
And in server.py do
from mkapp import app, db
and in models/users.py do
from mkapp import db
As a bonus, this should also remove the import cycle.
I don't use flask much, so this solution can probably be improved on. For example, having a function create app and db and memoise the results might be better than creating them in top-level module code.
While creating a web app using Flask or FastAPI, there would be a main.py file that basically instantiates and runs everything. I think that is also the right place for the database connections and initialization. So ideally I'd like to have a separate model.py file that basically just has the object document mapping definitions and nothing else.
Is it possible to do something like that in umongo?
I mean we need to call #instance.register above every object document map class. But if that is in a separate file and the DB is not initialized there, then in that file there are no instances. The instance would be declared in the main.py file.
For example, when you use Tortoise, it allows you to pass the whole model.py file as a module and register it with FastAPI like the following -
register_tortoise(
app,
db_url=os.environ.get("DATABASE_URL"),
modules={"models": ["models.model"]}, #model -> model.py which has all the class definitions
generate_schemas=True
)
For demo, you can have the following as model.py file contents -
from umongo import Document
from umongo.fields import StringField, URLField, DateTimeField
from datetime import datetime
class WebData(Document):
url = URLField()
summary = StringField()
created_at = DateTimeField(missing=datetime.now)
def __str__(self):
return self.url
And the following main.py file that uses FastAPI -
from fastapi import FastAPI
from pymongo import MongoClient
from umongo import Instance
#db = MongoClient().test
#instance = Instance(db)
app = FastAPI()
#app.get('/ping')
async def pong():
return {'ping': 'pong'}
if __name__ == "__main__":
import uvicorn
uvicorn.run("app.main:app", host="0.0.0.0", port=8000, reload=True)
You can instantiate the instance at import but pass it a DB connection at app init.
common.py
from umongo.frameworks import PyMongoInstance
instance = PyMongoInstance()
model.py
from umongo import Document
from .common import instance
#instance.register
class MyDocument(Document)
init.py
from .common import .instance
import .model
def create_app():
database = MongoClient().test
instance.init(database)
...
(Note: in umongo 3 beta, instance.init is renamed to instance.set_db.)
As far as I know from the research I've made, the typical way of defining a table using PonyORM in Python is like the following:
from pony.orm import *
db = Database()
# Database connection ...
class SampleTable(db.entity):
sample_int_field = Required(int)
sample_string_field = Required(str)
# ...
db.generate_mapping(create_tables=True)
My Problem: this uses db.entity
I wish to define a table without using the specific Databse instance in an abstract general manner, and connect it to the instance when I need to.
is there a way to do so?
concept (not real runnable code presumably):
# SampleAbstractTable.py
from pony.orm import *
class SampleAbstractTable(Database):
sample_int_field = Required(int)
sample_string_field = Required(str)
# ...
# main.py
from pony.orm import *
import SampleAbstractTable
db = Database()
# Database connection ...
db.connectTables((SampleAbstractTable.SampleAbstractTable, ...))
db.generate_mapping(create_tables=True)
EDIT:
One idea I have is to create a wrapper class for the database I wish to use with a certain group of tables, and define the tables in the init, because the whole point of me wishing to define tables dynamically is to seperate the Database instance creation from the table classes' definitions, namely:
from pony.orm import *
class sampleDatabase:
def __init__(self):
self._db = Database()
# Database connection ...
class TableA(db.entity):
# ...
class TableB(db.entity):
# ...
self._db.generate_mapping(create_tables=True)
but then I have issues in accessing the database tables...
First of all you're working with Entities, not Tables. They're not the same thing.
Your problem can be solved just defining the function like factory
def define_entities(db):
class Entity1(db.Entity):
attr1 = Required(str)
... and so on
And then later when you create your Database instance you just call
db = Database(...)
define_entities(db)
I'm setting up unit-testing for a Flask project using SQLAlchemy as ORM. For my tests I need to setup a new test database every time I run a single unit-test. Somehow, I cannot seem to run consecutive tests that query the database, even though if I run these tests in isolation they succeed.
I use the flask-testing package, and follow their documentation here.
Here is a working example to illustrate the problem:
app.py:
from flask import Flask
def create_app():
app = Flask(__name__)
return app
if __name__ == '__main__':
app = create_app()
app.run(port=8080)
database.py:
from flask_sqlalchemy import SQLAlchemy
db = SQLAlchemy()
models.py:
from database import db
class TestModel(db.Model):
"""Model for testing."""
__tablename__ = 'test_models'
id = db.Column(db.Integer,
primary_key=True
)
test/__init__.py:
from flask_testing import TestCase
from app import create_app
from database import db
class BaseTestCase(TestCase):
def create_app(self):
app = create_app()
app.config.update({
'SQLALCHEMY_DATABASE_URI': 'sqlite:///:memory:',
'SQLALCHEMY_TRACK_MODIFICATIONS': False,
'TESTING': True
})
db.init_app(app)
return app
def setUp(self):
db.create_all()
def tearDown(self):
db.session.remove()
db.drop_all()
test/test_app.py:
from models import TestModel
from test import BaseTestCase
from database import db
test_model = TestModel()
class TestApp(BaseTestCase):
"""WebpageEnricherController integration test stubs"""
def _add_to_db(self, record):
db.session.add(record)
db.session.commit()
self.assertTrue(record in db.session)
def test_first(self):
"""
This test runs perfectly fine
"""
self._add_to_db(test_model)
result = db.session.query(TestModel).first()
self.assertIsNotNone(result, 'Nothing in the database')
def test_second(self):
"""
This test runs fine in isolation, but fails if run consecutively
after the first test
"""
self._add_to_db(test_model)
result = db.session.query(TestModel).first()
self.assertIsNotNone(result, 'Nothing in the database')
if __name__ == '__main__':
import unittest
unittest.main()
So, I can run TestApp.test_first and TestApp.test_second fine if run in isolation. If I run them consecutively, the first test passes, but the second test fails with:
=================================== FAILURES ===================================
_____________________________ TestApp.test_second ______________________________
self = <test.test_app.TestApp testMethod=test_second>
def test_second(self):
"""
This test runs fine in isolation, but fails if run consecutively
after the first test
"""
self._add_to_db(test_model)
result = db.session.query(TestModel).first()
> self.assertIsNotNone(result, 'Nothing in the database')
E AssertionError: unexpectedly None : Nothing in the database
Something is going wrong in the database setup and teardown, but I cannot figure out what. How do I set this up correctly?
The answer is that you are leaking state between one test and the next by reusing a single TestModel instance defined once in the module scope (test_model = TestModel()).
The state of that instance at the commencement of the first test is transient:
an instance that’s not in a session, and is not saved to the database;
i.e. it has no database identity. The only relationship such an object
has to the ORM is that its class has a mapper() associated with it.
The state of the object at commencement of the second test is detached:
Detached - an instance which corresponds, or previously corresponded,
to a record in the database, but is not currently in any session. The
detached object will contain a database identity marker, however
because it is not associated with a session, it is unknown whether or
not this database identity actually exists in a target database.
Detached objects are safe to use normally, except that they have no
ability to load unloaded attributes or attributes that were previously
marked as “expired”.
This kind of interdependence between tests is almost always a bad idea. You could use make_transient() on the object at the end of every test:
class BaseTestCase(TestCase):
...
def tearDown(self):
db.session.remove()
db.drop_all()
make_transient(test_model)
Or you should construct a new TestModel instance for each test:
class BaseTestCase(TestCase):
...
def setUp(self):
db.create_all()
self.test_model = TestModel()
class TestApp(BaseTestCase):
...
def test_xxxxx(self):
self._add_to_db(self.test_model)
I think the latter is the better choice as there is no danger of any other leaky state getting carried between tests.