This is my declarative model:
import datetime
from sqlalchemy import Column, Integer, DateTime
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Test(Base):
__tablename__ = 'test'
id = Column(Integer, primary_key=True)
created_date = DateTime(default=datetime.datetime.utcnow)
However, when I try to import this module, I get this error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "orm/models2.py", line 37, in <module>
class Test(Base):
File "orm/models2.py", line 41, in Test
created_date = sqlalchemy.DateTime(default=datetime.datetime.utcnow)
TypeError: __init__() got an unexpected keyword argument 'default'
If I use an Integer type, I can set a default value. What's going on?
Calculate timestamps within your DB, not your client
For sanity, you probably want to have all datetimes calculated by your DB server, rather than the application server. Calculating the timestamp in the application can lead to problems because network latency is variable, clients experience slightly different clock drift, and different programming languages occasionally calculate time slightly differently.
SQLAlchemy allows you to do this by passing func.now() or func.current_timestamp() (they are aliases of each other) which tells the DB to calculate the timestamp itself.
Use SQLALchemy's server_default
Additionally, for a default where you're already telling the DB to calculate the value, it's generally better to use server_default instead of default. This tells SQLAlchemy to pass the default value as part of the CREATE TABLE statement.
For example, if you write an ad hoc script against this table, using server_default means you won't need to worry about manually adding a timestamp call to your script--the database will set it automatically.
Understanding SQLAlchemy's onupdate/server_onupdate
SQLAlchemy also supports onupdate so that anytime the row is updated it inserts a new timestamp. Again, best to tell the DB to calculate the timestamp itself:
from sqlalchemy.sql import func
time_created = Column(DateTime(timezone=True), server_default=func.now())
time_updated = Column(DateTime(timezone=True), onupdate=func.now())
There is a server_onupdate parameter, but unlike server_default, it doesn't actually set anything serverside. It just tells SQLalchemy that your database will change the column when an update happens (perhaps you created a trigger on the column ), so SQLAlchemy will ask for the return value so it can update the corresponding object.
One other potential gotcha:
You might be surprised to notice that if you make a bunch of changes within a single transaction, they all have the same timestamp. That's because the SQL standard specifies that CURRENT_TIMESTAMP returns values based on the start of the transaction.
PostgreSQL provides the non-SQL-standard statement_timestamp() and clock_timestamp() which do change within a transaction. Docs here: https://www.postgresql.org/docs/current/static/functions-datetime.html#FUNCTIONS-DATETIME-CURRENT
UTC timestamp
If you want to use UTC timestamps, a stub of implementation for func.utcnow() is provided in SQLAlchemy documentation. You need to provide appropriate driver-specific functions on your own though.
DateTime doesn't have a default key as an input. The default key should be an input to the Column function. Try this:
import datetime
from sqlalchemy import Column, Integer, DateTime
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Test(Base):
__tablename__ = 'test'
id = Column(Integer, primary_key=True)
created_date = Column(DateTime, default=datetime.datetime.utcnow)
You can also use sqlalchemy builtin function for default DateTime
from sqlalchemy.sql import func
DT = Column(DateTime(timezone=True), default=func.now())
You likely want to use onupdate=datetime.now so that UPDATEs also change the last_updated field.
SQLAlchemy has two defaults for python executed functions.
default sets the value on INSERT, only once
onupdate sets the value to the callable result on UPDATE as well.
Using the default parameter with datetime.now:
from sqlalchemy import Column, Integer, DateTime
from datetime import datetime
class Test(Base):
__tablename__ = 'test'
id = Column(Integer, primary_key=True)
created_at = Column(DateTime, default=datetime.now)
updated_at = Column(DateTime, default=datetime.now, onupdate=datetime.now)
The default keyword parameter should be given to the Column object.
Example:
Column(u'timestamp', TIMESTAMP(timezone=True), primary_key=False, nullable=False, default=time_now),
The default value can be a callable, which here I defined like the following.
from pytz import timezone
from datetime import datetime
UTC = timezone('UTC')
def time_now():
return datetime.now(UTC)
For mariadb thats worked for me:
from sqlalchemy import Column, Integer, String, DateTime, TIMESTAMP, text
from sqlalchemy.sql import func
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Test(Base):
__tablename__ = "test"
id = Column(Integer, primary_key=True, autoincrement=True)
name = Column(String(255), nullable=False)
email = Column(String(255), nullable=False)
created_at = Column(TIMESTAMP, nullable=False, server_default=func.now())
updated_at = Column(DateTime, server_default=text("CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP"))
In the sqlalchemy documentation for mariadb, it is recommended to import the textfrom sqlalchemy itself and set the server_default with the text, inserting the custom command.
updated_at=Column(DateTime, server_default=text("CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP"))
To understand func.now you can read the sql alchemy documentation.
Hope I helped in some way.
As per PostgreSQL documentation:
now, CURRENT_TIMESTAMP, LOCALTIMESTAMP return the time of transaction
This is considered a feature: the intent is to allow a single
transaction to have a consistent notion of the "current" time, so that
multiple modifications within the same transaction bear the same time stamp.
You might want to use statement_timestamp or clock_timestamp if you don't want transaction timestamp.
statement_timestamp()
returns the start time of the current statement (more specifically,
the time of receipt of the latest command message from the client).
statement_timestamp
clock_timestamp()
returns the actual current time, and therefore its value changes even
within a single SQL command.
Jeff Widman said on his answer that you need to create your own implementation of UTC timestamps for func.utcnow()
As I didnt want to implement it myself, I have searched for and found a python package which already does the job and is maintained by many people.
The package name is spoqa/sqlalchemy-ut.
A summary of what the package does is:
Long story short, UtcDateTime does:
take only aware datetime.datetime,
return only aware datetime.datetime,
never take or return naive datetime.datetime,
ensure timestamps in database always to be encoded in UTC, and
work as you’d expect.
Note that for server_default=func.now() and func.now() to work :
Local_modified = Column(DateTime, server_default=func.now(), onupdate=func.now())
you need to set DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP in your table DDL.
For example
create table test
(
id int auto_increment
primary key,
source varchar(50) null,
Local_modified datetime DEFAULT CURRENT_TIMESTAMP ON UPDATE CURRENT_TIMESTAMP
)
collate=utf8mb4_bin;
Otherwise, server_default=func.now(), onupdate=func.now() makes no effects.
You can use TIMESTAMP with sqlalchemy.
from sqlalchemy import TIMESTAMP, Table, MetaData, Column, ...
... ellipsis ...
def function_name(self) -> Table:
return Table(
"table_name",
self._metadata,
...,
Column("date_time", TIMESTAMP),
)
... ellipsis ...
Related
I'm implementing a restful POST API with Flask, using sqlalchemy to update resource in PostgreSQL, say MyResource:
class MyResource(db.Model):
__tablename__ = 'my_resource'
res_id = Column(Integer, primary_key=True)
<other columns>
time_updated = Column(TIMESTAMP(timezone=True), onupdate=datetime.now(timezone.utc))
There's a MyResource instance derived from API's request payload, let's call it input_instance. Below is my approach for updating the resource:
input_instance_dict = input_instance.__dict__
input_instance_dict.pop('_sa_instance_state', None) # this extra meta field will cause error in update().
update_count = MyResource.query.filter(MyResource.res_id == input_instance.res_id).update(input_instance_dict)
db.session.commit()
With above, columns are updated except time_updated, which remains null; I expect it to be updated with current date time.
If I remove the time_updated field from input prior to calling Query.update(),
input_instance_dict.pop('time_updated', None)
then the null value in time_updated column will be updated with current date time, BUT... on subsequent updates, this column's value remains the same old value.
My doubt is that, even with time_updated field removed from input dict, onupdate only takes effect for first update but not afterwards. Why? Thanks.
--- Update 12/23 10:56am GMT+8
Additional observation, I just re-triggered the same update as last night's twice, time_updated column is getting updated for first retry but not the second ones. Which means, After the very first update, onupdate takes effect on and off for following updates. I can't figure out the pattern, when it'll work and when it won't.
Similar problem is also observed for the other timestamp field to be populated with default, say... a record was inserted yesterday, but all records inserted today end up having the same time_created value as yesterday's value.
time_created = Column(TIMESTAMP(timezone=True), nullable=False, default=datetime.now(timezone.utc))
After changing the argument (for default and onupdate), replacing the python datetime function with sqlalchemy.func.now() resolves the weird behaviour.
time_created = Column(TIMESTAMP(timezone=True), nullable=False, default=func.now())
time_updated = Column(TIMESTAMP(timezone=True), onupdate=func.now())
I'm not sure why the behaviour difference, there are many tutorials that using datetime function as argument, I wonder if those programs having the same problem.
I have a Flask-SQLAlchemy model with an Integer field that I'd like to autoincrement. It's not a primary key; it's a surrogate ID. The model looks like:
class StreetSegment(db.Model):
id = db.Column(db.Integer, autoincrement=True)
seg_id = db.Column(db.Integer, primary_key=True)
When I create the table in my Postgres database, the id field is created as a plain integer. If I insert rows without specifying a value for id, it doesn't get populated. Is there some way I can force SQLAlchemy to use SERIAL even if it isn't the primary key?
Use Sequence instead of autoincrement:
id = db.Column(db.Integer, db.Sequence("seq_street_segment_id"))
SQLAlchemy does not support auto_increment for non-primary-key columns.
If your database supports it, you can setup the same behavior using sequences. PostgreSQL supports this. Sequences actually are not bound to a very specific column. Instead, they exist on the database level and can be reused. Sequences are the exact construct, SQLAlchemy uses for auto incrementing primary-key columns.
To use a sequence as described in the accepted answer, it must exist. Following, I have an example of an alembic migration with SQLAlchemy to achieve that.
You can associate a sequence with a column in the column constructor. The DDL Expression Constructs API helps you creating and dropping the sequence.
An example:
from alembic import op
import sqlalchemy as sa
measurement_id_seq = sa.Sequence('Measurement_MeasurementId_seq') # represents the sequence
def upgrade():
op.execute(sa.schema.CreateSequence(measurement_id_seq)) # create the sequence
op.create_table(
'Measurement',
sa.Column('DataSourceId',
sa.Integer,
sa.ForeignKey('DataSource.DataSourceId'),
nullable=False),
sa.Column('LocationId',
sa.Integer,
sa.ForeignKey('Location.LocationId'),
nullable=False),
sa.Column('MeasurementId',
sa.Integer,
measurement_id_seq, # the sequence as SchemaItem
server_default=measurement_id_seq.next_value())) # next value of the sequence as default
[...]
op.create_primary_key('Measurement_pkey', 'Measurement',
['DataSourceId', 'LocationId', 'Timestamp'])
pass
def downgrade():
op.execute(
sa.schema.DropSequence(sa.Sequence('Measurement_MeasurementId_seq')))
op.drop_constraint('Measurement_pkey', 'Measurement')
op.drop_table('Measurement')
pass
I'm trying to use association proxies to make dealing with tag-style records a little simpler, but I'm running into a problem enforcing uniqueness and getting objects to reuse existing tags rather than always create new ones.
Here is a setup similar to what I have. The examples in the documentation have a few recipes for enforcing uniqueness, but they all rely on having access to a session and usually require a single global session, which I cannot do in my case.
from sqlalchemy import Column, Integer, String, create_engine, ForeignKey
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.ext.associationproxy import association_proxy
Base = declarative_base()
engine = create_engine('sqlite://', echo=True)
Session = sessionmaker(bind=engine)
def _tag_find_or_create(name):
# can't use global objects here, may be multiple sessions and engines
# ?? No access to session here, how to do a query
tag = session.query(Tag).filter_by(name=name).first()
tag = Tag.query.filter_by(name=name).first()
if not tag:
tag = Tag(name=name)
return tag
class Item(Base)
__tablename__ = 'item'
id = Column(Integer, primary_key=True)
tags = relationship('Tag', secondary='itemtag')
tagnames = association_proxy('tags', 'name', creator=_tag_find_or_create)
class ItemTag(Base)
__tablename__ = 'itemtag'
id = Column(Integer, primary_key=True)
item_id = Column(Integer, ForeignKey('item.id'))
tag_id = Column(Integer, ForeignKey('tag.id'))
class Tag(Base)
__tablename__ = 'tag'
id = Column(Integer, primary_key=True)
name = Column(String(50), nullable=False)
# Scenario 1
session = Session()
item = Item()
session.add(item)
item.tagnames.append('red')
# Scenario 2
item2 = Item()
item2.tagnames.append('blue')
item2.tagnames.append('red')
session.add(item2)
Without the creator function, I just get tons of duplicate Tag items. The creator function seems like the most obvious place to put this type of check, but I'm unsure how to do a query from inside the creator function.
Consider the two scenarios provided at the bottom of the example. In the first example, it seems like there should be a way to get access to the session in the creator function, since the object the tags are being added to is already associated with a session.
In the second example, the Item object isn't yet associated with a session, so the validation check can't happen in the creator function. It would have to happen later when the object is actually added to a session.
For the first scenario, how would I go about getting access to the session object in the creator function?
For the second scenario, is there a way to "listen" for when the parent object is added to a session and validate the association proxies at that point?
For the first scenario, you can use object_session.
As for the question overall: true, you need access to the current session; if using scoped_session in your application is appropriate, then the second part of the Recipe you link to should work fine to use. See Contextual/Thread-local Sessions for more info.
Working with events and change objects when they change from transient to persistent state will not make your code pretty or very robust. So I would immediately add new Tag objects to the session, and if the transaction is rolled back, they would not be in the database.
Note that in a multi-user environment you are likely to have race condition: the same tag is new and created in simultaneously by two users. The user who commits last will fail (if you have a unique constraint on the database).
In this case you might consider be without the unique constraint, and have a (daily) procedure to clean those duplicates up (and reassign relations). With time there would be less and less new items, and less possibilities for such clashes.
I'm working on a project with Alembic and SQLAlchemy, but I'm having trouble creating a simple entry in the database as a test. I get the following error:
sqlalchemy.exc.ArgumentError: Mapper Mapper|Sale|sales_cache could not assemble any primary key columns for mapped table 'sales_cache'
I've established the primary key (account_id) in both places below, any idea why SQLAlchemy doesn't recognize that or how to fix it? The other answers I've read have all dealt with exception cases for multiple/no primary keys, and have been solved accordingly, but this is a pretty vanilla model that keeps failing.
I've read up on other answers, most of which deal with the declarative system:
class Sale(Base):
__tablename__ = 'sales_cache'
But I'm required to use the classical mapping system; here's my mapped class and schema, respectively:
class Sale(object):
def __init__(self, notification):
self._sale_id = self._notification.object_id
self._account_id = self._notification.account_id
### schema.py file ###
from sqlalchemy.schema import MetaData, Table, Column
from sqlalchemy.types import (Unicode, Integer)
from database import metadata
metadata = MetaData()
sales_cache = Table('sales_cache', metadata,
Column('account_id', Integer, primary_key=True, autoincrement=False),
Column('sale_id', Integer, nullable=False)
)
And this is the relevant line from my alembic revision:
sa.Column('account_id', sa.Integer(), primary_key=True, autoincrement=False),
I thought it might be failing because I was setting self._sale_id and self._account_id instead of self.sale_id and self.account_id (without the underscore), but nothing changed when I tried it this way too.
Thanks in advance
Should I use () with datetime.now in defining tables? What code wrong 1 or 2?
1:
Base = declarative_base()
class T(Base):
__tablename__ = 't'
created = Column(DateTime, default=datetime.now)
2:
Base = declarative_base()
class T(Base):
__tablename__ = 't'
created = Column(DateTime, default=datetime.now())
You want the first case. What you're doing is telling SqlAlchemy than whenever a row is inserted, run this function (callable) to get the default value. That can be any callable function or a string value.
This way the function is called exactly at insert time and you get the correct date that it was inserted.
If you use the second value, you'll find that all of the default values share the same datetime. That will be the value that occurred the first time the code was processed.
http://www.sqlalchemy.org/docs/core/schema.html?highlight=default#sqlalchemy.schema.ColumnDefault
This could correspond to a constant, a callable function, or a SQL clause.