We Keep Coding

Create a Calculated Column in SQLAlchemy Core Table

I have a SQLAlchemy table defined as following:
user = Table('user', MetaData(),
Column('id', Integer),
Column('first_name', String),
Column('last_name', String))
I often need to refer the full_name of a user during query, for example:
sql = (select([
user.c.id,
(user.c.first_name + user.c.last_name).label('full_name')
]).where(user.c.id == 123))
Since full_name is used in many places, so such kind of code has a lot copies.
I wonder whether there is a way in SQLAlchemy I can create a calculated Column, so that I can conveniently use it just like other normal Column, the SQLAlchemy automatically converts it into (user.c.first_name + user.c.last_name).label('full_name') whenever I refer user.c.full_name
sql = (select([
user.c.id,
user.c.full_name
]).where(user.c.id == 123))
I searched and found there some solution in SQLAlchemy ORM using column_property or hybrid_property. The difference in my case is that I can only use SQLAlchemy Core.

It is not possible to create calculated columns in sqlalchemy core. You don't need to do this however, all that is required is to save your expression in a variable and then use this in your select statements. If you have many of these to store then you could namespace them by storing them all in a collection. In the example below I've used an SQLAlchemy Properties object for this purpose so it will behave in similar manner to the columns collection.
class Table(sqlalchemy.Table):
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
self.d = sqlalchemy.util._collections.Properties({})
user = Table('user', MetaData(),
Column('id', Integer),
Column('first_name', String),
Column('last_name', String))
user.d.full_name = (user.c.first_name + user.c.last_name).label('full_name')
user.d.backwards_name = (user.c.last_name + user.c.first_name).label('backwards_name')
sql = (select([
user.c.id,
user.d.full_name
]).where(user.c.id == 123))

Peewee ORM query result fn.COUNT is type unicode, not integer

Please help me understand behavior of peewee 2.4.5 when talking to MySQL 5.5. I'm running a simple query to count children associated with a parent; in this case documents at a path. As plain SQL it boils down to this:
select p.name, count(d.file) as child_count
from path as p, doc as d
where p.id = d.path_id
group by p.name
The Peewee code uses the fn.COUNT feature, see below for a self-contained example. The result comes back just fine and with the results I expect, with one exception: the query result object attribute "child_count" is of type unicode instead of integer. In this little example there's 1 row and I get back a string (essentially) '1' instead of the number 1.
I'm confused because in other queries I have done with fn.COUNT the result is of type integer. Is this a feature? Am I making a silly python mistake here? Thanks in advance.
'''
Example of accessing MySQL from Python using Peewee.
Developed with peewee 2.4.5, pymysql 0.6.3, MySql 5.5
'''
from __future__ import print_function
from peewee import MySQLDatabase, Model, CharField, ForeignKeyField, fn
db = MySQLDatabase(database="test", host="localhost", user="mumble", password="foo")
class MySQLModel(Model):
'''
Base class to associate the database object
'''
class Meta:
database = db
class Path(MySQLModel):
# peewee adds primary key field 'id'
name = CharField()
class Doc(MySQLModel):
# peewee adds primary key field 'id'
path = ForeignKeyField(Path)
file = CharField()
def main():
db.connect()
db.create_tables([Path, Doc], True)
newpath = Path(name='ab/23')
newpath.save()
newdoc1 = Doc(path=newpath.id, file='file1.txt')
newdoc1.save()
newdoc2 = Doc(path=newpath.id, file='file2.txt')
newdoc2.save()
for row in Path.select():
print("Path: id=%d, name=%s" % (row.id, row.name))
for row in Doc.select():
print("Doc: id=%d, file=%s" % (row.id, row.file))
# query in plain old SQL:
# select p.name, count(d.file) from path as p, doc as d where p.id = d.path_id group by p.name
path_doc_result = (Path
.select(Path.name, fn.COUNT(Doc.file).alias('child_count'))
.join(Doc, on=(Path.id == Doc.path))
.group_by(Path.name))
path_doc_count = len(list(path_doc_result))
print("Path-doc parent-child result count is %d" % path_doc_count)
if path_doc_count == 0:
print("Programmer error, no results!")
else:
# get the first one
d_row = path_doc_result[0]
#### Why is the child_count attribute not integer? ###
print("Type of child_count attribute is %s" % type(d_row.child_count))
print("Path-Doc result: name=%s child_count=%d" % (d_row.name, int(d_row.child_count)))
newdoc1.delete_instance()
newdoc2.delete_instance()
newpath.delete_instance()
# order matters for foreign keys!
db.drop_table(Doc)
db.drop_table(Path)
db.close()
if __name__ == "__main__":
main()

Peewee functions look at the type of the first argument and attempt to coerce the return value to that type. This makes sense in most cases but I can see why it's causing an issue here.
To work around, just call fn.COUNT(Doc.file).coerce(False).alias('child_count')
path_doc_result = (Path
.select(Path.name, fn.COUNT(Doc.file).coerce(False).alias('child_count'))
.join(Doc, on=(Path.id == Doc.path))
.group_by(Path.name))

SQLAlchemy temporary table with Declarative Base

I need a temporary table in my programme. I have seen that this can be achieved with the "mapper" syntax in this way:
t = Table(
't', metadata,
Column('id', Integer, primary_key=True),
# ...
prefixes=['TEMPORARY'],
)
Seen here
But, my whole code is using the declarative base, it is what I understand, and I would like to stick to it. There is the possibility of using a hybrid approach but if possible I'd avoid it.
This is a simplified version of how my declarative class looks like:
import SQLAlchemy as alc
class Tempo(Base):
"""
Class for temporary table used to process data coming from xlsx
#param Base Declarative Base
"""
# TODO: make it completely temporary
__tablename__ = 'tempo'
drw = alc.Column(alc.String)
date = alc.Column(alc.Date)
check_number = alc.Column(alc.Integer)
Thanks in advance!
EDITED WITH THE NEW PROBLEMS:
Now the class looks like this:
import SQLAlchemy as alc
class Tempo(Base):
"""
Class for temporary table used to process data coming from xlsx
#param Base Declarative Base
"""
# TODO: make it completely temporary
__tablename__ = 'tempo'
__table_args__ = {'prefixes': ['TEMPORARY']}
drw = alc.Column(alc.String)
date = alc.Column(alc.Date)
check_number = alc.Column(alc.Integer)
And when I try to insert data in this table, I get the following error message:
sqlalchemy.exc.OperationalError: (OperationalError) no such table:
tempo u'INSERT INTO tempo (...) VALUES (?, ?, ?, ?, ?, ?, ?, ?)' (....)
It seems the table doesn't exist just by declaring it. I have seen something like create_all() that might be the solution for this (it's funny to see how new ideas come while explaining thoroughly)
Then again, thank you very much!

Is it possible to use __table_args__? See https://docs.sqlalchemy.org/en/14/orm/declarative_tables.html#orm-declarative-table-configuration
class Tempo(Base):
"""
Class for temporary table used to process data coming from xlsx
#param Base Declarative Base
"""
# TODO: make it completely temporary
__tablename__ = 'tempo'
__table_args__ = {'prefixes': ['TEMPORARY']}
drw = alc.Column(alc.String)
date = alc.Column(alc.Date)
check_number = alc.Column(alc.Integer)

Old question, but if anyone out there wants to create a temp table from an existing declarative table model on the fly rather than having it always be a part of your model/code, you can try the following approach. Copying __table_args__ is a little tricky since it can have multiple formats and any Index objects need to be recreated so they aren't associated with the old table.
import time
from sqlalchemy.schema import CreateTable
def copy_table_args(model, **kwargs):
"""Try to copy existing __table_args__, override params with kwargs"""
table_args = model.__table_args__
if isinstance(table_args, tuple):
new_args = []
for arg in table_args:
if isinstance(arg, dict):
table_args_dict = arg.copy()
table_args_dict.update(**kwargs)
new_args.append(arg)
elif isinstance(arg, sa.Index):
index = sa.Index(
arg.name,
*[col for col in arg.columns.keys()],
unique=arg.unique,
**arg.kwargs,
)
new_args.append(index)
else:
# TODO: need to handle Constraints
raise Exception(f"Unhandled table arg: {arg}")
table_args = tuple(new_args)
elif isinstance(table_args, dict):
table_args = {
k: (v.copy() if hasattr(v, "copy") else v) for k, v in table_args.items()
}
table_args.update(**kwargs)
else:
raise Exception(f"Unexpected __table_args__ type: {table_args}")
return table_args
def copy_table_from_model(conn, model, **kwargs):
model_name = model.__name__ + "Tmp"
table_name = model.__table__.name + "_" + str(time.time()).replace(".", "_")
table_args = copy_table_args(model, extend_existing=True)
args = {c.name: c.copy() for c in model.__table__.c}
args["__tablename__"] = table_name
args["__table_args__"] = table_args
copy_model = type(model_name, model.__bases__, args)
print(str(CreateTable(copy_model.__table__)))
copy_model.__table__.create(conn)
return copy_model
def temp_table_from_model(conn, model, **kwargs):
return copy_table_from_model(conn, model, prefixes=["TEMPORARY"])
Note: I haven't added logic to handle copying Constraints, and this is lightly tested against MySQL. Also note that if you do this with non-temporary tables and auto-named indexes (i.e. Column(..., index=True)) then this may not play nice with alembic.

How to do an upsert with SqlAlchemy?

I have a record that I want to exist in the database if it is not there, and if it is there already (primary key exists) I want the fields to be updated to the current state. This is often called an upsert.
The following incomplete code snippet demonstrates what will work, but it seems excessively clunky (especially if there were a lot more columns). What is the better/best way?
Base = declarative_base()
class Template(Base):
__tablename__ = 'templates'
id = Column(Integer, primary_key = True)
name = Column(String(80), unique = True, index = True)
template = Column(String(80), unique = True)
description = Column(String(200))
def __init__(self, Name, Template, Desc):
self.name = Name
self.template = Template
self.description = Desc
def UpsertDefaultTemplate():
sess = Session()
desired_default = Template("default", "AABBCC", "This is the default template")
try:
q = sess.query(Template).filter_by(name = desiredDefault.name)
existing_default = q.one()
except sqlalchemy.orm.exc.NoResultFound:
#default does not exist yet, so add it...
sess.add(desired_default)
else:
#default already exists. Make sure the values are what we want...
assert isinstance(existing_default, Template)
existing_default.name = desired_default.name
existing_default.template = desired_default.template
existing_default.description = desired_default.description
sess.flush()
Is there a better or less verbose way of doing this? Something like this would be great:
sess.upsert_this(desired_default, unique_key = "name")
although the unique_key kwarg is obviously unnecessary (the ORM should be able to easily figure this out) I added it just because SQLAlchemy tends to only work with the primary key. eg: I've been looking at whether Session.merge would be applicable, but this works only on primary key, which in this case is an autoincrementing id which is not terribly useful for this purpose.
A sample use case for this is simply when starting up a server application that may have upgraded its default expected data. ie: no concurrency concerns for this upsert.

SQLAlchemy supports ON CONFLICT with two methods on_conflict_do_update() and on_conflict_do_nothing().
Copying from the documentation:
from sqlalchemy.dialects.postgresql import insert
stmt = insert(my_table).values(user_email='a#b.com', data='inserted data')
stmt = stmt.on_conflict_do_update(
index_elements=[my_table.c.user_email],
index_where=my_table.c.user_email.like('%#gmail.com'),
set_=dict(data=stmt.excluded.data)
)
conn.execute(stmt)

SQLAlchemy does have a "save-or-update" behavior, which in recent versions has been built into session.add, but previously was the separate session.saveorupdate call. This is not an "upsert" but it may be good enough for your needs.
It is good that you are asking about a class with multiple unique keys; I believe this is precisely the reason there is no single correct way to do this. The primary key is also a unique key. If there were no unique constraints, only the primary key, it would be a simple enough problem: if nothing with the given ID exists, or if ID is None, create a new record; else update all other fields in the existing record with that primary key.
However, when there are additional unique constraints, there are logical issues with that simple approach. If you want to "upsert" an object, and the primary key of your object matches an existing record, but another unique column matches a different record, then what do you do? Similarly, if the primary key matches no existing record, but another unique column does match an existing record, then what? There may be a correct answer for your particular situation, but in general I would argue there is no single correct answer.
That would be the reason there is no built in "upsert" operation. The application must define what this means in each particular case.

Nowadays, SQLAlchemy provides two helpful functions on_conflict_do_nothing and on_conflict_do_update. Those functions are useful but require you to swich from the ORM interface to the lower-level one - SQLAlchemy Core.
Although those two functions make upserting using SQLAlchemy's syntax not that difficult, these functions are far from providing a complete out-of-the-box solution to upserting.
My common use case is to upsert a big chunk of rows in a single SQL query/session execution. I usually encounter two problems with upserting:
For example, higher level ORM functionalities we've gotten used to are missing. You cannot use ORM objects but instead have to provide ForeignKeys at the time of insertion.
I'm using this following function I wrote to handle both of those issues:
def upsert(session, model, rows):
table = model.__table__
stmt = postgresql.insert(table)
primary_keys = [key.name for key in inspect(table).primary_key]
update_dict = {c.name: c for c in stmt.excluded if not c.primary_key}
if not update_dict:
raise ValueError("insert_or_update resulted in an empty update_dict")
stmt = stmt.on_conflict_do_update(index_elements=primary_keys,
set_=update_dict)
seen = set()
foreign_keys = {col.name: list(col.foreign_keys)[0].column for col in table.columns if col.foreign_keys}
unique_constraints = [c for c in table.constraints if isinstance(c, UniqueConstraint)]
def handle_foreignkeys_constraints(row):
for c_name, c_value in foreign_keys.items():
foreign_obj = row.pop(c_value.table.name, None)
row[c_name] = getattr(foreign_obj, c_value.name) if foreign_obj else None
for const in unique_constraints:
unique = tuple([const,] + [row[col.name] for col in const.columns])
if unique in seen:
return None
seen.add(unique)
return row
rows = list(filter(None, (handle_foreignkeys_constraints(row) for row in rows)))
session.execute(stmt, rows)

I use a "look before you leap" approach:
# first get the object from the database if it exists
# we're guaranteed to only get one or zero results
# because we're filtering by primary key
switch_command = session.query(Switch_Command).\
filter(Switch_Command.switch_id == switch.id).\
filter(Switch_Command.command_id == command.id).first()
# If we didn't get anything, make one
if not switch_command:
switch_command = Switch_Command(switch_id=switch.id, command_id=command.id)
# update the stuff we care about
switch_command.output = 'Hooray!'
switch_command.lastseen = datetime.datetime.utcnow()
session.add(switch_command)
# This will generate either an INSERT or UPDATE
# depending on whether we have a new object or not
session.commit()
The advantage is that this is db-neutral and I think it's clear to read. The disadvantage is that there's a potential race condition in a scenario like the following:
we query the db for a switch_command and don't find one
we create a switch_command
another process or thread creates a switch_command with the same primary key as ours
we try to commit our switch_command

There are multiple answers and here comes yet another answer (YAA). Other answers are not that readable due to the metaprogramming involved. Here is an example that
Uses SQLAlchemy ORM
Shows how to create a row if there are zero rows using on_conflict_do_nothing
Shows how to update the existing row (if any) without creating a new row using on_conflict_do_update
Uses the table primary key as the constraint
A longer example in the original question what this code is related to.
import sqlalchemy as sa
import sqlalchemy.orm as orm
from sqlalchemy import text
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy.orm import Session
class PairState(Base):
__tablename__ = "pair_state"
# This table has 1-to-1 relationship with Pair
pair_id = sa.Column(sa.ForeignKey("pair.id"), nullable=False, primary_key=True, unique=True)
pair = orm.relationship(Pair,
backref=orm.backref("pair_state",
lazy="dynamic",
cascade="all, delete-orphan",
single_parent=True, ), )
# First raw event in data stream
first_event_at = sa.Column(sa.TIMESTAMP(timezone=True), nullable=False, server_default=text("TO_TIMESTAMP(0)"))
# Last raw event in data stream
last_event_at = sa.Column(sa.TIMESTAMP(timezone=True), nullable=False, server_default=text("TO_TIMESTAMP(0)"))
# The last hypertable entry added
last_interval_at = sa.Column(sa.TIMESTAMP(timezone=True), nullable=False, server_default=text("TO_TIMESTAMP(0)"))
#staticmethod
def create_first_event_if_not_exist(dbsession: Session, pair_id: int, ts: datetime.datetime):
"""Sets the first event value if not exist yet."""
dbsession.execute(
insert(PairState).
values(pair_id=pair_id, first_event_at=ts).
on_conflict_do_nothing()
)
#staticmethod
def update_last_event(dbsession: Session, pair_id: int, ts: datetime.datetime):
"""Replaces the the column last_event_at for a named pair."""
# Based on the original example of https://stackoverflow.com/a/49917004/315168
dbsession.execute(
insert(PairState).
values(pair_id=pair_id, last_event_at=ts).
on_conflict_do_update(constraint=PairState.__table__.primary_key, set_={"last_event_at": ts})
)
#staticmethod
def update_last_interval(dbsession: Session, pair_id: int, ts: datetime.datetime):
"""Replaces the the column last_interval_at for a named pair."""
dbsession.execute(
insert(PairState).
values(pair_id=pair_id, last_interval_at=ts).
on_conflict_do_update(constraint=PairState.__table__.primary_key, set_={"last_interval_at": ts})
)

The below works fine for me with redshift database and will also work for combined primary key constraint.
SOURCE : this
Just few modifications required for creating SQLAlchemy engine in the function
def start_engine()
from sqlalchemy import Column, Integer, Date ,Metadata
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.dialects.postgresql import insert
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.dialects import postgresql
Base = declarative_base()
def start_engine():
engine = create_engine(os.getenv('SQLALCHEMY_URI',
'postgresql://localhost:5432/upsert'))
connect = engine.connect()
meta = MetaData(bind=engine)
meta.reflect(bind=engine)
return engine
class DigitalSpend(Base):
__tablename__ = 'digital_spend'
report_date = Column(Date, nullable=False)
day = Column(Date, nullable=False, primary_key=True)
impressions = Column(Integer)
conversions = Column(Integer)
def __repr__(self):
return str([getattr(self, c.name, None) for c in self.__table__.c])
def compile_query(query):
compiler = query.compile if not hasattr(query, 'statement') else
query.statement.compile
return compiler(dialect=postgresql.dialect())
def upsert(session, model, rows, as_of_date_col='report_date', no_update_cols=[]):
table = model.__table__
stmt = insert(table).values(rows)
update_cols = [c.name for c in table.c
if c not in list(table.primary_key.columns)
and c.name not in no_update_cols]
on_conflict_stmt = stmt.on_conflict_do_update(
index_elements=table.primary_key.columns,
set_={k: getattr(stmt.excluded, k) for k in update_cols},
index_where=(getattr(model, as_of_date_col) < getattr(stmt.excluded, as_of_date_col))
)
print(compile_query(on_conflict_stmt))
session.execute(on_conflict_stmt)
session = start_engine()
upsert(session, DigitalSpend, initial_rows, no_update_cols=['conversions'])

This allows access to the underlying models based on string names
def get_class_by_tablename(tablename):
"""Return class reference mapped to table.
https://stackoverflow.com/questions/11668355/sqlalchemy-get-model-from-table-name-this-may-imply-appending-some-function-to
:param tablename: String with name of table.
:return: Class reference or None.
"""
for c in Base._decl_class_registry.values():
if hasattr(c, '__tablename__') and c.__tablename__ == tablename:
return c
sqla_tbl = get_class_by_tablename(table_name)
def handle_upsert(record_dict, table):
"""
handles updates when there are primary key conflicts
"""
try:
self.active_session().add(table(**record_dict))
except:
# Here we'll assume the error is caused by an integrity error
# We do this because the error classes are passed from the
# underlying package (pyodbc / sqllite) SQLAlchemy doesn't mask
# them with it's own code - this should be updated to have
# explicit error handling for each new db engine
# <update>add explicit error handling for each db engine</update>
active_session.rollback()
# Query for conflic class, use update method to change values based on dict
c_tbl_primary_keys = [i.name for i in table.__table__.primary_key] # List of primary key col names
c_tbl_cols = dict(sqla_tbl.__table__.columns) # String:Col Object crosswalk
c_query_dict = {k:record_dict[k] for k in c_tbl_primary_keys if k in record_dict} # sub-dict from data of primary key:values
c_oo_query_dict = {c_tbl_cols[k]:v for (k,v) in c_query_dict.items()} # col-object:query value for primary key cols
c_target_record = session.query(sqla_tbl).filter(*[k==v for (k,v) in oo_query_dict.items()]).first()
# apply new data values to the existing record
for k, v in record_dict.items()
setattr(c_target_record, k, v)

This works for me with sqlite3 and postgres. Albeit it might fail with combined primary key constraints and will most likely fail with additional unique constraints.
try:
t = self._meta.tables[data['table']]
except KeyError:
self._log.error('table "%s" unknown', data['table'])
return
try:
q = insert(t, values=data['values'])
self._log.debug(q)
self._db.execute(q)
except IntegrityError:
self._log.warning('integrity error')
where_clause = [c.__eq__(data['values'][c.name]) for c in t.c if c.primary_key]
update_dict = {c.name: data['values'][c.name] for c in t.c if not c.primary_key}
q = update(t, values=update_dict).where(*where_clause)
self._log.debug(q)
self._db.execute(q)
except Exception as e:
self._log.error('%s: %s', t.name, e)

As we had problems with generated default-ids and references which lead to ForeignKeyViolation-Errors like
update or delete on table "..." violates foreign key constraint
Key (id)=(...) is still referenced from table "...".
we had to exclude the id for the update dict, as otherwise the it will be always generated as new default value.
In addition the method is returning the created/updated entity.
from sqlalchemy.dialects.postgresql import insert # Important to use the postgresql insert
def upsert(session, data, key_columns, model):
stmt = insert(model).values(data)
# Important to exclude the ID for update!
exclude_for_update = [model.id.name, *key_columns]
update_dict = {c.name: c for c in stmt.excluded if c.name not in exclude_for_update}
stmt = stmt.on_conflict_do_update(
index_elements=key_columns,
set_=update_dict
).returning(model)
orm_stmt = (
select(model)
.from_statement(stmt)
.execution_options(populate_existing=True)
)
return session.execute(orm_stmt).scalar()
Example:
class UpsertUser(Base):
__tablename__ = 'upsert_user'
id = Column(Id, primary_key=True, default=uuid.uuid4)
name: str = Column(sa.String, nullable=False)
user_sid: str = Column(sa.String, nullable=False, unique=True)
house_admin = relationship('UpsertHouse', back_populates='admin', uselist=False)
class UpsertHouse(Base):
__tablename__ = 'upsert_house'
id = Column(Id, primary_key=True, default=uuid.uuid4)
admin_id: Id = Column(Id, ForeignKey('upsert_user.id'), nullable=False)
admin: UpsertUser = relationship('UpsertUser', back_populates='house_admin', uselist=False)
# Usage
upserted_user = upsert(session, updated_user, [UpsertUser.user_sid.name], UpsertUser)
Note: Only tested on postgresql but could work also for other DBs which support ON DUPLICATE KEY UPDATE e.g. MySQL

In case of sqlite, the sqlite_on_conflict='REPLACE' option can be used when defining a UniqueConstraint, and sqlite_on_conflict_unique for unique constraint on a single column. Then session.add will work in a way just like upsert. See the official documentation.

I use this code for upsert
Before using this code, you should add primary keys to table in database.
from sqlalchemy import create_engine
from sqlalchemy import MetaData, Table
from sqlalchemy.inspection import inspect
from sqlalchemy.engine.reflection import Inspector
from sqlalchemy.dialects.postgresql import insert
def upsert(df, engine, table_name, schema=None, chunk_size = 1000):
metadata = MetaData(schema=schema)
metadata.bind = engine
table = Table(table_name, metadata, schema=schema, autoload=True)
# olny use common columns between df and table.
table_columns = {column.name for column in table.columns}
df_columns = set(df.columns)
intersection_columns = table_columns.intersection(df_columns)
df1 = df[intersection_columns]
records = df1.to_dict('records')
# get list of fields making up primary key
primary_keys = [key.name for key in inspect(table).primary_key]
with engine.connect() as conn:
chunks = [records[i:i + chunk_size] for i in range(0, len(records), chunk_size)]
for chunk in chunks:
stmt = insert(table).values(chunk)
update_dict = {c.name: c for c in stmt.excluded if not c.primary_key}
s = stmt.on_conflict_do_update(
index_elements= primary_keys,
set_=update_dict)
conn.execute(s)

Is it possible to store Python class objects in SQLite?

I would like to store Python objects into a SQLite database. Is that possible?
If so what would be some links / examples for it?

You can't store the object itself in the DB. What you do is to store the data from the object and reconstruct it later.
A good way is to use the excellent SQLAlchemy library. It lets you map your defined class to a table in the database. Every mapped attribute will be stored, and can be used to reconstruct the object. Querying the database returns instances of your class.
With it you can use not only sqlite, but most databases - It currently also supports Postgres, MySQL, Oracle, MS-SQL, Firebird, MaxDB, MS Access, Sybase, Informix and IBM DB2. And you can have your user choose which one she wants to use, because you can basically switch between those databases without changing the code at all.
There are also a lot of cool features - like automatic JOINs, polymorphing...
A quick, simple example you can run:
from sqlalchemy import Column, Integer, Unicode, UnicodeText, String
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
from random import choice
from string import letters
engine = create_engine('sqlite:////tmp/teste.db', echo=True)
Base = declarative_base(bind=engine)
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(Unicode(40))
address = Column(UnicodeText, nullable=True)
password = Column(String(20))
def __init__(self, name, address=None, password=None):
self.name = name
self.address = address
if password is None:
password = ''.join(choice(letters) for n in xrange(10))
self.password = password
Base.metadata.create_all()
Session = sessionmaker(bind=engine)
s = Session()
Then I can use it like this:
# create instances of my user object
u = User('nosklo')
u.address = '66 Some Street #500'
u2 = User('lakshmipathi')
u2.password = 'ihtapimhskal'
# testing
s.add_all([u, u2])
s.commit()
That would run INSERT statements against the database.
# When you query the data back it returns instances of your class:
for user in s.query(User):
print type(user), user.name, user.password
That query would run SELECT users.id AS users_id, users.name AS users_name, users.address AS users_address, users.password AS users_password.
The printed result would be:
<class '__main__.User'> nosklo aBPDXlTPJs
<class '__main__.User'> lakshmipathi ihtapimhskal
So you're effectively storing your object into the database, the best way.

Yes it's possible but there are different approaches and which one is the suitable one, will depend on your requirements.
Pickling
You can use the pickle module to serialize objects, then store these objects in a blob in sqlite3 (or a textfield, if the dump is e.g. base64 encoded). Be aware of some possible problems: questions/198692/can-i-pickle-a-python-dictionary-into-a-sqlite3-text-field
Object-Relational-Mapping
You can use object relational mapping. This creates, in effect, a "virtual object database" that can be used from within the programming language (Wikipedia). For python, there is a nice toolkit for that: sqlalchemy.

You can use pickle.dumps, its return pickable objects as strings, you would not need to write it to temporary files.
Return the pickled representation of
the object as a string, instead of
writing it to a file.
import pickle
class Foo:
attr = 'a class attr'
picklestring = pickle.dumps(Foo)

SQLite 3's adaptors and converters
I'm surprised by how no one has read the docs for the SQLite 3 library, because it says that you can do this by creating an adaptor and converter. For example, let's say that we have a class called 'Point' and we want to store this and have this returned when selecting it and using the database cursor's fetchone method to return it. Let's make the module know that what you select from the database is a point
from sqlite3 import connect, register_adaptor, register_converter
class Point:
def __init__(self, x, y):
self.x, self.y = x, y
def __repr__(self):
return "(%f;%f)" % (self.x, self.y)
def adapt_point(point):
return ("%f;%f" % (point.x, point.y)).encode('ascii')
def convert_point(s):
x, y = list(map(float, s.split(b";")))
return Point(x, y)
# Register the adapter
register_adapter(Point, adapt_point)
# Register the converter
register_converter("point", convert_point)
p = Point(4.0, -3.2)
# 1) Using declared types
con = connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)
con.execute("create table test(p point)")
con.execute("insert into test(p) values (?)", (p,))
cur = con.execute("select p from test")
print("with declared types:", cur.fetchone()[0])
con.close()
# 1) Using column names
con = connect(":memory:", detect_types=sqlite3.PARSE_COLNAMES)
con.execute("create table test(p)")
con.execute("insert into test(p) values (?)", (p,))
cur = con.execute('select p as "p [point]" from test')
print("with column names:", cur.fetchone()[0])
con.close()

You can use pickle to serialize the object. The serialized object can be inserted to the sqlite DB as a bytearray field.
f=open('object.dump', 'rw')
pickle.dump(obj, f)
Now read object.dump from the file, and write it to the sqlite DB. You might want to write it as a binary data type; read about storing binary data and blob in SQLite here. Note that according to this source, SQLite limits the size of such datafield to 1Mb.
I think that a better option would be serializing your object into a file, and keeping the file name, not contents, in the database.

You other choice instead of pickling is to use an ORM. This lets you map rows in a database to an object. See http://wiki.python.org/moin/HigherLevelDatabaseProgramming for a starting point. I'd recommend SQLAlchemy or SQLObject.

There is relatively simple way to store and compare objects, eaven to index those objects right way and to restrict (with ubique) columns containing objects. And all of that without using ORM engines. Objects mast be stored using pickle dump (so performance might be a issue) Here is example for storing python tuples, indexing restricting and comparing. This method can be easily applied to any other python class. All that is needed is explained in python sqlite3 documentation (somebody already posted the link). Anyway here it is all put together in the following example:
import sqlite3
import pickle
def adapt_tuple(tuple):
return pickle.dumps(tuple)
sqlite3.register_adapter(tuple, adapt_tuple) #cannot use pickle.dumps directly because of inadequate argument signature
sqlite3.register_converter("tuple", pickle.loads)
def collate_tuple(string1, string2):
return cmp(pickle.loads(string1), pickle.loads(string2))
# 1) Using declared types
con = sqlite3.connect(":memory:", detect_types=sqlite3.PARSE_DECLTYPES)
con.create_collation("cmptuple", collate_tuple)
cur = con.cursor()
cur.execute("create table test(p tuple unique collate cmptuple) ")
cur.execute("create index tuple_collated_index on test(p collate cmptuple)")
######################### Test ########################
cur.execute("select name, type from sqlite_master") # where type = 'table'")
print(cur.fetchall())
p = (1,2,3)
p1 = (1,2)
cur.execute("insert into test(p) values (?)", (p,))
cur.execute("insert into test(p) values (?)", (p1,))
cur.execute("insert into test(p) values (?)", ((10, 1),))
cur.execute("insert into test(p) values (?)", (tuple((9, 33)) ,))
cur.execute("insert into test(p) values (?)", (((9, 5), 33) ,))
try:
cur.execute("insert into test(p) values (?)", (tuple((9, 33)) ,))
except Exception as e:
print e
cur.execute("select p from test order by p")
print "\nwith declared types and default collate on column:"
for raw in cur:
print raw
cur.execute("select p from test order by p collate cmptuple")
print "\nwith declared types collate:"
for raw in cur:
print raw
con.create_function('pycmp', 2, cmp)
print "\nselect grater than using cmp function:"
cur.execute("select p from test where pycmp(p,?) >= 0", ((10, ),) )
for raw in cur:
print raw
cur.execute("select p from test where pycmp(p,?) >= 0", ((3,)))
for raw in cur:
print raw
print "\nselect grater than using collate:"
cur.execute("select p from test where p > ?", ((10,),) )
for raw in cur:
print raw
cur.execute("explain query plan select p from test where p > ?", ((3,)))
for raw in cur:
print raw
cur.close()
con.close()

Depending on your exact needs, it could be worth looking into Django (www.djangoproject.com) for this task. Django is actually a web framework, but one of the tasks it handles is to allow you to define Models as python objects (inheriting from a base class provided by the framework). It will then automatically create the database tables required to store those objects, and sqlite is among the supported backends. It also provides handy functions to query the database and return one or more matching objects. See for example the documentation about Models in django:
http://docs.djangoproject.com/en/1.9/topics/db/models/
The drawback is of course that you have to install a full web framework, and (as far as I remember) you can only store objects whose attributes are supported by django. Also, it's made for storing many instances of predefined objects, not for storing one instance each of many different objects. Depending on your needs, this may or may not be impractical.

One option is to use an O/R mapper like SQLObject. It will do most of the plumbing to persist the Python object to a database, and it supports SQLite. As mentioned elsewhere you can also serialise the object using a method such as pickle, which dumps out a representation of the object that it can reconstruct by reading back in and parsing.

As others have mentioned, the answer is yes... but the object needs to be serialized first. I'm the author of a package called klepto that is built to seamlessly store python objects in SQL databases, HDF archives, and other types of key-value stores.
It provides a simple dictionary interface, like this:
>>> from klepto.archives import sqltable_archive as sql_archive
>>> d = sql_archive(cached=False)
>>> d['a'] = 1
>>> d['b'] = '1'
>>> d['c'] = min
>>> squared = lambda x:x*x
>>> d['d'] = squared
>>> class Foo(object):
... def __init__(self, x):
... self.x = x
... def __call__(self):
... return squared(self.x)
...
>>> f = Foo(2)
>>> d['e'] = Foo
>>> d['f'] = f
>>>
>>> d
sqltable_archive('sqlite:///:memory:?table=memo' {'a': 1, 'b': '1', 'c': <built-in function min>, 'd': <function <lambda> at 0x10f631268>, 'e': <class '__main__.Foo'>, 'f': <__main__.Foo object at 0x10f63d908>}, cached=False)
>>>
>>> # min(squared(2), 1)
>>> d['c'](d['f'](), d['a'])
1
>>>
The cached keyword in the archive constructor signifies whether you want to use a local memory cache, with the archive set as the cache backend (cached=True) or just use the archive directly (cached=False). Under the covers, it can use pickle, json, dill, or other serializers to pickle the objects. Looking at the archive's internals, you can see it's leveraging SQLAlchemy:
>>> d._engine
Engine(sqlite://)
>>> d.__state__
{'serialized': True, 'root': 'sqlite:///:memory:', 'id': Table('memo', MetaData(bind=None), Column('Kkey', String(length=255), table=<memo>, primary_key=True, nullable=False), Column('Kval', PickleType(), table=<memo>), schema=None), 'protocol': 3, 'config': {}}