I am having some real trouble getting a created_date column working with SQLAlchemy 2.0 with the ORM model. The best answer so far I've found is at this comment: https://stackoverflow.com/a/33532154 however I haven't been able to make that function work. In my (simplified) models.py file I have:
import datetime
from sqlalchemy import Integer, String, DateTime
from sqlalchemy.sql import func
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column
class Base(DeclarativeBase):
pass
class MyTable(Base):
__tablename__ = "my_table"
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String, nullable=False)
created_date: Mapped[datetime.datetime] = mapped_column(DateTime(timezone=True), server_default=func.now())
So far, so good, thinks I. In the simplified engine.py I have:
from sqlalchemy import create_engine
from sqlalchemy import select
from sqlalchemy.orm import Session
import models
def add_entry(engine, name_str):
this_row = models.MyTable()
this_row.name = name_str
with Session(engine) as session:
session.add(this_row)
session.commit()
If I'm understanding correctly, the default value for the created_date to be a SQL function, and SQLAlchemy maps now() to SQLite3's datetime(). With the engine set to echo=True, I get the following result when it tries to run this insert command (Please note, this is data from the non-simplified form but it's still pretty simple, had 3 strings instead of the one I described)
2023-02-06 09:47:07,080 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2023-02-06 09:47:07,080 INFO sqlalchemy.engine.Engine INSERT INTO coaches (d_name, bb2_name, bb3_name) VALUES (?, ?, ?) RETURNING id, created_date
2023-02-06 09:47:07,081 INFO sqlalchemy.engine.Engine [generated in 0.00016s] ('andy#1111', 'AndyAnderson', 'Killer Andy')
2023-02-06 09:47:07,081 INFO sqlalchemy.engine.Engine ROLLBACK
This causes an exception when it gets to the time function: IntegrityError: NOT NULL constraint failed: coaches.created_date
Some additional data (I have been using the rich library which produces an enormous amount of debug information so I'm trying to get the best bits:
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │ exc_tb = <traceback object at 0x00000108BD2565C0> │ │
│ │ exc_type = <class 'sqlalchemy.exc.IntegrityError'> │ │
│ │ exc_value = IntegrityError('(sqlite3.IntegrityError) NOT NULL constraint failed: │ │
│ │ coaches.created_date') │ │
│ │ self = <sqlalchemy.util.langhelpers.safe_reraise object at 0x00000108BD1B79A0> │ │
│ │ traceback = None │ │
│ │ type_ = None │ │
│ │ value = None │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
In any event, I feel like I've gotten the wrong end of the stick on the way to make a table column automatically execute a SQL command with the func call. Any notions on this one? I haven't found any direct example in the SQLAlchemy 2.0 docs, and aside from the pretty awesome comment to a similar question, I haven't found any working solutions.
Thanks for considering!
I implemented a SQLAlchemy 2.0 mapped_column with a server_default of func.now() expecting the column to automatically fill during an INSERT operation. During the insert operation, SQLAlchemy threw an exception claiming the column NOT NULLABLE constraint was violated -- thus it was not automatically filling.
Posting an answer to my own question to note what actually did work (actual problem still exists, but a simplified variation does work just dandy the way I expect it to.)
import datetime
from sqlalchemy import Integer, String, DateTime
from sqlalchemy import create_engine
from sqlalchemy.sql import func
from sqlalchemy.orm import DeclarativeBase
from sqlalchemy.orm import Mapped
from sqlalchemy.orm import mapped_column
from sqlalchemy.orm import Session
class Base(DeclarativeBase):
pass
class MyTable(Base):
__tablename__ = "my_table"
id: Mapped[int] = mapped_column(primary_key=True)
name: Mapped[str] = mapped_column(String)
created_date: Mapped[datetime.datetime] = mapped_column(
DateTime(timezone=True), server_default=func.now()
)
def initialize_engine(filename):
return create_engine(f"sqlite+pysqlite:///{filename}", echo=True)
def initialize_tables(engine):
Base.metadata.create_all(engine)
def add_row(engine, name):
this_row = MyTable(name=name)
print(this_row)
with Session(engine) as session:
session.add(this_row)
session.commit()
my_file = "test.db"
my_engine = initialize_engine(my_file)
initialize_tables(my_engine)
add_row(my_engine, "Dave")
This produces the result:
python datetest.py
2023-02-06 11:02:41,157 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2023-02-06 11:02:41,158 INFO sqlalchemy.engine.Engine PRAGMA main.table_info("my_table")
2023-02-06 11:02:41,158 INFO sqlalchemy.engine.Engine [raw sql] ()
2023-02-06 11:02:41,158 INFO sqlalchemy.engine.Engine COMMIT
<__main__.MyTable object at 0x000002CC767ECD50>
2023-02-06 11:02:41,159 INFO sqlalchemy.engine.Engine BEGIN (implicit)
2023-02-06 11:02:41,160 INFO sqlalchemy.engine.Engine INSERT INTO my_table (name) VALUES (?) RETURNING id, created_date
2023-02-06 11:02:41,160 INFO sqlalchemy.engine.Engine [generated in 0.00020s] ('Dave',)
2023-02-06 11:02:41,171 INFO sqlalchemy.engine.Engine COMMIT
The schema in the correctly working database reads:
sqlite> .schema my_table
CREATE TABLE my_table (
id INTEGER NOT NULL,
name VARCHAR NOT NULL,
created_date DATETIME DEFAULT (CURRENT_TIMESTAMP) NOT NULL,
PRIMARY KEY (id)
);
So... all I have to do is figure out why my original code isn't doing the simple variation!
Related
Set RDBMS_URI env var to a connection string like postgresql://username:password#host/database, then on Python 3.9 with PostgreSQL 15 and SQLalchemy 1.14 run:
from os import environ
from sqlalchemy import Boolean, Column, Identity, Integer
from sqlalchemy import create_engine
from sqlalchemy.orm import declarative_base
Base = declarative_base()
class Tbl(Base):
__tablename__ = 'Tbl'
__has_error__ = Column(Boolean)
id = Column(Integer, primary_key=True, server_default=Identity())
engine = create_engine(environ["RDBMS_URI"])
Base.metadata.create_all(engine)
Checking the database:
=> \d "Tbl"
Table "public.Tbl"
Column | Type | Collation | Nullable | Default
--------+---------+-----------+----------+----------------------------------
id | integer | | not null | generated by default as identity
Indexes:
"Tbl_pkey" PRIMARY KEY, btree (id)
How do I force the column names with double underscore to work?
I believe that the declarative machinery explicitly excludes attributes whose names start with a double underscore from the mapping process (based on this and this). Consequently your __has_error__ column is not created in the target table.
There are at least two possible workarounds. Firstly, you could give the model attribute a different name, for example:
_has_error = Column('__has_error__', BOOLEAN)
This will create the database column __has_attr__, accessed through Tbl._has_error*.
If you want the model's attribute to be __has_error__, then you can achieve this by using an imperative mapping.
import sqlalchemy as sa
from sqlalchemy import orm
mapper_registry = orm.registry()
tbl = sa.Table(
'tbl',
mapper_registry.metadata,
sa.Column('__has_error__', sa.Boolean),
sa.Column(
'id', sa.Integer, primary_key=True, server_default=sa.Identity()
),
)
class Tbl:
pass
mapper_registry.map_imperatively(Tbl, tbl)
mapper_registry.metadata.create_all(engine)
* I tried using a synonym to map __has_error__ to _has_error but it didn't seem to work. It probably gets exluded in the mapper as well, but I didn't investigate further.
How can I convert a uuid to text and match using the like operator? E.g I want to do following in sqlalchemy:
SELECT * FROM user
WHERE id::text like '%0c%';
PS: Column definition is
from sqlalchemy.dialects.postgresql import UUID
id = Column(
UUID(as_uuid=True),
primary_key=True,
index=True,
nullable=False,
default=uuid4,
)
Use the cast function
import sqlalchemy as sa
...
with Session() as s:
results = s.scalars(
sa.select(MyModel).where(
sa.cast(MyModel.id, sa.TEXT).like(f'%{fragment}%')
)
)
id::text in Postgresql is shorthand* for cast:
with x as (select gen_random_uuid() as uu)
select uu::text,
cast(uu as text),
uu::text = cast(uu as text) as "Equals?" from x;
uu │ uu │ Equals?
══════════════════════════════════════╪══════════════════════════════════════╪═════════
b23efe1b-fb05-4d8a-92a3-59339d2ea0f4 │ b23efe1b-fb05-4d8a-92a3-59339d2ea0f4 │ t
* Well, mostly...
Envirionment:
SQLAlchemy + PostgreSQL (pg8000)
Situation:
I created two table from psql console.
CREATE TABLE testidtable(
testid BIGSERIAL PRIMARY KEY NOT NULL
);
CREATE TABLE testidtable2(
testid bigint PRIMARY KEY NOT NULL,
CONSTRAINT testid_is_valid FOREIGN KEY(testid) REFERENCES testidtable (testid)
);
Here is the python code:
import sqlalchemy
from sqlalchemy import Table, MetaData
from config import Config
conf = Config()
engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"postgresql+pg8000",
database="databasename",
username=conf.db1_user,
password=conf.db1_pass),
client_encoding='utf-8')
metadata = MetaData()
metadata.reflect(bind=engine)
engine.connect()
table1 = Table("testidtable", metadata, autoload_with=engine)
ins1 = table1.insert().values({})
ret1 = engine.execute(ins1)
testid = ret1.inserted_primary_key[0]
table2 = Table("testidtable2", metadata, autoload_with=engine)
ins2 = table2.insert().values({"testid":testid})
ret2 = engine.execute(ins2)
And the error occurred:
sqlalchemy.exc.CompileError: Unconsumed column names: testid
I tried to show ins2 which is INSERT query for debugging, but faced the same situation.
How can I do INSERT testid into testidtable2 successfully?
Thanks.
I am using sqlalchemy to write to a mysql db where I am indexing some files and storing their contents. I need to write the files and then write the index entries which have a foreign key to the files table. However, sqlalchemy seems to be issuing the INSERT statements out of order.
Here is a minimal functional example illustrating the problem using mock random data (less the configuration file, which contains server specific info):
Index/ORM.py:
#!/bin/env python2.7
from __future__ import print_function
import os
from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.dialects.mysql import LONGBLOB, INTEGER
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import relationship, sessionmaker
from sqlalchemy import create_engine
from Index import load_cfg
class Base(object):
"""
Basic MySQL table settings
"""
__table_args__ = {
'mysql_engine': 'InnoDB',
'mysql_collate': 'latin1_general_cs'
}
Base = declarative_base(cls=Base)
class CoverageIndex(Base):
"""
Class for coverage_index table objects
"""
__tablename__ = 'coverage_index'
filename = Column(String(45), primary_key=True)
#filename = Column(String(45), ForeignKey("files.filename"), primary_key=True)
sequence_id = Column(String(45), primary_key=True, index=True)
def __init__(self, filename, sequence_id):
self.filename = filename
self.sequence_id = sequence_id
class FileRow(Base):
"""
Class for files stored in db
"""
__tablename__ = 'files'
filename = Column(String(45), primary_key=True)
contents = Column(LONGBLOB)
def __init__(self, filename, contents):
self.filename = filename
self.contents = contents
cfg = load_cfg()
db_string = 'mysql://%(user)s:%(passwd)s#%(host)s/%(db)s' % cfg['db_config']
engine = create_engine(db_string, echo=True)
Base.metadata.create_all(engine)
if __name__ == '__main__':
pass
index.py:
#!/usr/bin/env python2.7
from __future__ import print_function
import os
import sys
from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
from sqlalchemy.exc import IntegrityError
from Index.ORM import Base, CoverageIndex, FileRow, engine as db_engine
if __name__ == '__main__':
import string, random
data = {}
for i in range(0,10):
file = 'file' + str(i)
data[file] = {
'seqs': ['seqa' + str(i), 'seqb' + str(i)],
'contents': '\n'.join([''.join([random.choice(string.letters) for x in range (0, 80)]) for y in range (0, 2500)])}
#print (data)
Base.metadata.bind = db_engine
DBSession = sessionmaker(bind=db_engine)
session = DBSession()
for file, datum in data.iteritems():
file_query = session.query(FileRow).filter(FileRow.filename == file)
if file_query.count() > 0:
session.query(CoverageIndex).filter(CoverageIndex.filename == file).delete(synchronize_session='fetch')
file_query.delete(synchronize_session='fetch')
for i in datum['seqs']:
# Write to DB
fqc = file_query.count()
print ("No. of files: " + str(fqc))
if fqc == 0:
print ("Adding: ")
fr = FileRow(
filename = file,
contents = datum['contents']
)
session.add(fr)
cov = CoverageIndex(
filename = file,
sequence_id = i)
session.add(cov)
try:
session.commit()
except:
#print ("SQL Commit Failed: %s" % file)
session.rollback()
session.close()
raise
session.close()
Here is a part of the output of one run. I would like to draw your attention to line 2018-03-13 16:05:40,291 and ...,292:
...
2018-03-13 16:05:40,287 INFO sqlalchemy.engine.base.Engine BEGIN (implicit)
2018-03-13 16:05:40,288 INFO sqlalchemy.engine.base.Engine SELECT count(*) AS count_1
FROM (SELECT files.filename AS files_filename, files.contents AS files_contents
FROM files
WHERE files.filename = %s) AS anon_1
2018-03-13 16:05:40,288 INFO sqlalchemy.engine.base.Engine ('file1',)
2018-03-13 16:05:40,290 INFO sqlalchemy.engine.base.Engine SELECT count(*) AS count_1
FROM (SELECT files.filename AS files_filename, files.contents AS files_contents
FROM files
WHERE files.filename = %s) AS anon_1
2018-03-13 16:05:40,290 INFO sqlalchemy.engine.base.Engine ('file1',)
No. of files: 0
Adding:
2018-03-13 16:05:40,291 INFO sqlalchemy.engine.base.Engine INSERT INTO coverage_index (filename, sequence_id) VALUES (%s, %s)
2018-03-13 16:05:40,291 INFO sqlalchemy.engine.base.Engine ('file1', 'seqa1')
2018-03-13 16:05:40,292 INFO sqlalchemy.engine.base.Engine INSERT INTO files (filename, contents) VALUES (%s, %s)
2018-03-13 16:05:40,292 INFO sqlalchemy.engine.base.Engine ('file1', 'BkTsRJTcNEigPFjofFxDmwVZDXRAsPECawRUjiFZTDGWWoLZzLnGlCwQQeAFyXhLqKjPAJmme
mFNfVzF\nJlZSvwGAdoImTnBAmcrSdMRDvxNYnnMfbQXdfuXulqufiIYpqjFUgfElZSrVkvBvPTg ... (204700 characters truncated) ... trwtYOycEOuDTVxsXeGoNYKAqHlE
LGPqcimwzwAFAEsCZGBBnGzYMHgabgnGZaGmQsn\nSNjYvBwSVdXVKbmJpKdSHSXCDKKvDlkyLxOxsEfOtmlCRruqzaiPhYRocKZQEJSVrtSHncFMBMTEpWUX')
2018-03-13 16:05:40,310 INFO sqlalchemy.engine.base.Engine SELECT count(*) AS count_1
FROM (SELECT files.filename AS files_filename, files.contents AS files_contents
FROM files
WHERE files.filename = %s) AS anon_1
2018-03-13 16:05:40,310 INFO sqlalchemy.engine.base.Engine ('file1',)
No. of files: 1
2018-03-13 16:05:40,311 INFO sqlalchemy.engine.base.Engine INSERT INTO coverage_index (filename, sequence_id) VALUES (%s, %s)
2018-03-13 16:05:40,311 INFO sqlalchemy.engine.base.Engine ('file1', 'seqb1')
2018-03-13 16:05:40,312 INFO sqlalchemy.engine.base.Engine COMMIT
...
Here, you can see that sqlalchemy is inserting the coverage_index before inserting the files object. I presume that this is because the files object is much bigger and taking some time to prepare, and so the engine decides to asynchronously run the later INSERT first.
However, the files entry needs to be inserted first, because filename in coverage_index should be a foreign key to files. (It throws an exception if I do this with the foreign key constraint defined)
I know I can commit after adding to files, but I would prefer that the files and coverage_index INSERT be in the same transaction, so they stay in sync.
So the question is, is there a way to force sqlalchemy to execute synchronously within a transaction?
Not sure if this is the best way, but it seems to achieve what I was going for:
flush(objects=None)
Flush all the object changes to the database.
Writes out all pending object creations, deletions and modifications to the database as INSERTs, DELETEs, UPDATEs, etc. Operations are automatically ordered by the Session’s unit of work dependency solver.
Database operations will be issued in the current transactional context and do not affect the state of the transaction, unless an error occurs, in which case the entire transaction is rolled back. You may flush() as often as you like within a transaction to move changes from Python to the database’s transaction buffer
Thanks to:
Is SQLAlchemy saves order in adding objects to session?
http://www.aosabook.org/en/sqlalchemy.html - Section 20.9 Unit of Work
I do have database table that has an id primary key that is not an auto-increment (sequence). So it's up to the user to create an unique id or the insert will fail.
This table is not under my control, so I cannot change the database structure.
from sqlalchemy import create_engine, Table, MetaData
import psycopg2
db = create_engine('postgresql://...', echo=False).connect()
meta = MetaData()
meta.reflect(bind=db)
t = Table("mytable", meta, autoload=True, autoload_with=db)
values = { "title":"title", "id": ... }# ???
t.insert(bind=db, values=values).execute()
Given this is "single-user" / "single-client" system, you should be able to use the Column defaults: Python-Executed Functions. The example on the documentation linked to is enough to get you started. I would, however, use python function but with proper initialization from the datatabase adn still stored in a global variable:
def new_id_factory():
if not('_MYTABLE_ID_' in globals()):
q = db.execute("select max(mytable.id) as max_id from mytable").fetchone()
_MYTABLE_ID_ = (q and q.max_id) or 0
_MYTABLE_ID_ += 1
return _MYTABLE_ID_
t = Table("mytable", Base.metadata,
Column('id', Integer, primary_key=True, default=new_id_factory), #
autoload=True, autoload_with=db,
)