We Keep Coding

Is there a way to combine a pydantic and SQLite model into one class?

I'm using fastapi and SQLite to make a simple API that can write to a database.
The pydantic model (Component class in main.py) is almost identical to the SQLite one (Component class in models.py). These two models could end up having many more properties. Each added property adds a line of duplicated code to the create and update methods that exists only to copy data from one model to the other. It seems like there must be a more elegant way to do this (sharing a model, multiple inheritance?) rather than defining two nearly identical classes and copying individual properties line by line multiple times.
models.py
A simple SQLite model:
from sqlalchemy import Column, Integer, String
from database import Base
class Component(Base):
__tablename__ = "components"
id = Column(Integer, primary_key=True, index=True)
name = Column(String)
author = Column(String)
description = Column(String)
main.py
I use fastapi and pydantic to define a basic CRUD API in a main.py. I've included only two of the operations:
from fastapi import FastAPI, HTTPException, Depends
from pydantic import BaseModel, Field
from sqlalchemy.orm import Session
from database import engine, SessionLocal
import models
app = FastAPI()
models.Base.metadata.create_all(bind=engine)
def get_db():
try:
db = SessionLocal()
yield db
finally:
db.close()
class Component(BaseModel):
name: str = Field(min_length=1)
author: str = Field(min_length=1, max_length=100)
description: str = Field(min_length=1, max_length=100)
#app.post("/")
def create_component(component: Component, db: Session = Depends(get_db)):
c = models.Component()
c.name = component.name
c.author = component.author
c.description = component.description
db.add(c)
db.commit()
return component
#app.put("/{component_id}")
def update_component(component_id: int, component: Component, db: Session = Depends(get_db)):
c.name = component.name
c.author = component.author
c.description = component.description
db.add(c)
db.commit()
return component

Select from table defined by SQLAlchemy declarative_base

I am working on FastAPI tutorial and I am trying to create tables using SQLAlchemy+Alembic+databases.
In my main.py I have:
from typing import List
import databases
import sqlalchemy
from fastapi import FastAPI
from pydantic import BaseModel
from sqlalchemy import Table
DATABASE_URL = "sqlite:///./test.db"
database = databases.Database(DATABASE_URL)
metadata = sqlalchemy.MetaData()
notes = sqlalchemy.Table(
"note",
metadata,
sqlalchemy.Column("id", sqlalchemy.Integer, primary_key=True),
sqlalchemy.Column("text", sqlalchemy.String),
sqlalchemy.Column("completed", sqlalchemy.Boolean),
)
class Note2(BaseModel):
id: int
text: str
completed: bool
app = FastAPI()
#app.on_event("startup")
async def startup():
await database.connect()
#app.on_event("shutdown")
async def shutdown():
await database.disconnect()
#app.get("/notes/", response_model=List[Note2])
async def read_notes():
query = notes.select()
return await database.fetch_all(query)
And this works - I can GET /notes/ endpoint. But it looks newbie to create a datatabase table in the same module with endpoints, so I decided to make models.py file and create a normal model there, like this:
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Note(Base):
__tablename__ = "note"
id = sa.Column(sa.Integer, primary_key=True)
text = sa.Column(sa.String)
completed = sa.Column(sa.Boolean)
And here comes a problem - when I change the endpoint like this:
from app_second.models import Note
#app.get("/notes/", response_model=List[Note2])
async def read_notes():
query = Note().select()
return await database.fetch_all(query)
I recieve an error:
AttributeError: 'Note' object has no attribute 'select'
As it is mentioned here - declarative_base() is just a syntactic shugar for Table + mapper. But what is the right way to select/filter/update tables declared that way?

You can use the table property to access the table methods and then use database.fetch_all() or similar. E.g:
from sqlalchemy import select
...
skip = 0
limit = 100
query = (
Note.__table__.select()
.offset(skip)
.limit(limit)
)
return await database.fetch_all(query)

Custom engine creation SQLAlchemy in module

I have a project where I want to isolate DB initialization (SQLAlchemy) from the other module so I've create a module
db_initializer.py:
engine = create_engine('sqlite:///db') # TODO from config file
Session = sessionmaker(bind=engine)
Base = declarative_base(bind=engine)
def create_tables():
Base.metadata.create_all(engine)
First of all I need to put create_all in a function because my model is in another package.
model/foo.py:
from core.db_initializer import Base
class User(Base):
__tablename__ = 'user'
id = Column(Integer, primary_key=True)
name = Column(String)
def __init__(self, name: str = None):
self.name = name
def __repr__(self):
return "<User(id=%d, name=%s)>" % (int(self.id), str(self.name))
And my main call create_tables.
Is there any other to do that? And now I need to create the engine with custom config (IP,User, ...) and only the main script know the config it's possible?
Something like
main.py:
import db_initializer as db
import model.foo
db.init(config) # which create the engine and create table
When I do something like that I got problem with the Base object which have not bind to the engine which is not created yet. Any solutions?

You don't need to create engine or session before declaring your models. You can declare models in model/foo.py:
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class User(Base):
__tablename__ = 'user'
Let's assume you have some application in myapp.py. Declare it so it can be initialized with engine:
from sqlalchemy.orm import Session
import model.foo
class MyApp:
def __init__(self, engine):
self.engine = engine
def get_users(self):
session = Session(self.engine)
users = session.query(model.foo.User).all()
session.close()
return users
Create engine in main.py and use it to initialize models.foo.Base.metadata and other applications where you need it:
from sqlalchemy import create_engine
import model.foo
import myapp
engine = create_engine('sqlite:///db')
model.foo.Base.metadata.bind = engine
model.foo.Base.metadata.create_all()
app = myapp.MyApp(engine)
UPD: For scoped_session approach myapp.py can be look like this:
import model.foo
class MyApp:
def __init__(self, session):
self.session = session
def get_users(self):
return self.session.query(model.foo.User).all()
And main.py:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
import model.foo
import myapp
engine = create_engine('sqlite:///db')
session = scoped_session(sessionmaker(engine))
model.foo.Base.metadata.bind = engine
model.foo.Base.metadata.create_all()
app = myapp.MyApp(session)

Python3 SQLAlchemy sqlalchemy.exc.UnboundExecutionError: Could not locate a bind configured on mapper Mapper or this Session

Working with SQL Alchemy for the first time and I keep getting an error when trying to commit.
Python3
SQLAlchemy 1.0.12
database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from sqlalchemy.ext.declarative import declarative_base
engine = create_engine("sqlite:///db.test")
db_session = scoped_session(sessionmaker(autocommit=False,
autoflush=False,
bind=engine))
Base = declarative_base()
Base.query = db_session.query_property()
def init_db():
# import all modules here that might define models so that
# they will be registered properly on the metadata. Otherwise
# you will have to import them first before calling init_db()
import src.models
Base.metadata.create_all(bind=engine)
models.py
from datetime import datetime
from typing import List
from sqlalchemy import Column, Table
from sqlalchemy import Integer, String, DateTime
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship
from src.database import Base
class Answer(Base):
"""
The answer for a question.
"""
__tablename__ = "answer"
answer_id = Column(Integer(), primary_key=True)
answer = Column(String(), nullable=False, index=True)
question_id = Column(Integer(), ForeignKey("question.question_id"))
question = relationship("Question", back_populates="answer")
def __init__(self, answer: str, question_id: int = None):
self.answer = answer
self.question_id = question_id
class Question(Base):
__tablename__ = "question"
question_id = Column(Integer(), primary_key=True)
question = Column(String(), nullable=False, unique=True, index=True)
answer = relationship("Answer", uselist=False, back_populates="question")
def __init__(self, question: str, answer: Answer, options: List[Option] = None):
self.question = question
self.answer = answer
if options:
self.options = options
Create database
>>> from src.database import init_db
>>> init_db()
This creates the database as expected
Add items to the database
>>> from src.models import Question, Answer
>>> a = Answer("Yes")
>>> q = Question("Doctor Who?", a)
>>> from sqlalchemy.orm import Session
>>> s = Session()
>>> s.add(a)
>>> s.add(q)
Until now I did not get an error
>>> s.commit()
Here I get the error:
sqlalchemy.exc.UnboundExecutionError: Could not locate a bind configured on mapper Mapper|Question|question or this Session
What can I do to get this working?

From "Using the Session":
sessionmaker class is normally used to create a top level Session configuration
You shouldn't try and create a session using the Session class directly. Use the scoped_session wrapped sessionmaker defined in your database.py, as it has the correct binding configured etc.:
>>> from src.models import Question, Answer
>>> from src.database import db_session
>>> a = Answer("Yes")
>>> q = Question("Doctor Who?", a)
>>> s = db_session()
>>> s.add(a)
>>> s.add(q)
>>> s.commit()
You could also use a scoped_session as a proxy.

Instantiating object automatically adds to SQLAlchemy Session. Why?

From my understanding of SQLAlchemy, in order to add a model to a session, I need to call session.add(obj). However, for some reason, in my code, SQLAlchemy seems to do this automatically.
Why is it doing this, and how can I stop it? Am I approaching session in the correct way?
example
>>> from database import Session as db
>>> import clients
>>> from instances import Instance
>>> from uuid import uuid4
>>> len(db.query(Instance).all())
>>> 0 # Note, no instances in database/session
>>> i = Instance(str(uuid4()), clients.get_by_code('AAA001'), [str(uuid4())])
>>> len(db.query(Instance).all())
>>> 1 # Why?? I never called db.add(i)!
database.py
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from sqlalchemy.ext.declarative import declarative_base
import config
Base = declarative_base()
class Database():
def __init__(self):
db_url = 'postgresql://{:s}:{:s}#{:s}:{}/{:s}'.format(
config.database['user'],
config.database['password'],
config.database['host'],
config.database['port'],
config.database['dbname']
)
self.engine = create_engine(db_url)
session_factory = sessionmaker(bind=self.engine)
self.session = scoped_session(session_factory)
Database = Database()
Session = Database.session
instance.py
from sqlalchemy import Column, Text, ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.dialects.postgresql import UUID, ARRAY
import database
Base = database.Base
class Instance(Base):
__tablename__ = 'instances'
uuid = Column(UUID, primary_key=True)
client_code = Column(
Text, ForeignKey('clients.code', ondelete='CASCADE'), nullable=False)
mac_addresses = Column(ARRAY(Text, as_tuple=True),
primary_key=True)
client = relationship("Client", back_populates="instances")
def __init__(self, uuid, client, mac_addresses):
self.uuid = uuid
self.client = client
self.mac_addresses = tuple(mac_addresses)
client.py
from sqlalchemy import Column, Text
from sqlalchemy.orm import relationship
import database
from database import Session as db
Base = database.Base
class Client(Base):
__tablename__ = 'clients'
code = Column(Text, primary_key=True)
name = Column(Text)
instances = relationship("Instance", back_populates='client')
def __init__(self, code, name=None):
self.code = code
self.name = name
def get_by_code(code):
client = db.query(Client).filter(Client.code == code).first()
return client

When you create a SQLAlchemy object and link it directly to another SQLAlchemy object, both objects end up in the session.
The reason is that SQLAlchemy needs to make sure you can query these objects.
Take, for example, a user with addresses.
If you create a user in code, with an address, both the user and the address end up in the session, because the address is linked to the user and SQLAlchemy needs to make sure you can query all addresses of a user using user.addresses.all().
In that case all (possibly) existing addresses need to be fetched, as well as the new address you just added. For that purpose the newly added address needs to be saved in the database.
To prevent this from happening (for example if you only need objects to just calculate with), you can link the objects with their IDs/Foreign Keys:
address.user_id = user.user_id
However, if you do this, you won't be able to access the SQLAlchemy properties anymore. So user.addresses or address.user will no longer yield results.
The reverse is also true; I asked a question myself a while back why linking two objects by ID will not result in SQLAlchemy linking these objects in the ORM:
relevant stackoverflow question
another description of this behavior