I am creating a sql alchemy table like so:
myEngine = self.get_my_engine() # creates engine
metadata = MetaData(bind=myEngine)
SnapshotTable = Table("mytable", metadata, autoload=False, schema="my schema")
I have to use autoload false because the table may or may not exist (and that code has to run before the table is created)
The problem is, if I use autoload = False, when I try to query the table (after it was created by another process) doing session.query(SnapshotTable) I get an:
InvalidRequestError: Query contains no columns with which to SELECT from.
error; which is understandable because the table wasn't loaded yet.
My question is: how do I "load" the table metadata after it has been defined with autoload = False.
I looked at the schema.py code and it seems that I can do:
SnapshotTable._autoload(metadata, None, None)
but that doesn't look right to me...any other ideas or thoughts?
Thanks
First declare the table model:
class MyTable(Base):
__table__ = Table('mytable', metadata)
Or directly:
MyTable = Table("mytable", metadata)
Then, once you are ready to load it, call this:
Table('mytable', metadata, autoload_with=engine, extend_existing=True)
Where the key to it all is extend_existing=True.
All credit goes to Mike Bayer on the SQLAlchemy mailing list.
I was dealing with this issue just last night, and it turns out that all you need to do is load all available table definitions from the database with the help of metadat.reflect. This is very much similar to #fgblomqvist's solution. The major difference is that you do not have to recreate the table. In essence, the following should help:
SnapshotTable.metadata.reflect(extend_existing=True, only=['mytable'])
The unsung hero here is the extend_existing parameter. It basically makes sure that the schema and other info associated with SnapshotTable are reloaded. The parameter only is used here to limit how much information is retrieved. This will save you a tremendous amount of time, if you are dealing with a large database
I hope this serves a purpose in the future.
I guess that problem is with not reflected metadata.
You could try to load metadata with method this call bevore executing any query :
metadata.reflect()
It will reload definition of table, so framework will know how to build proper SELECT.
And then calling
if SnapshotTable.exists :
SnapshotTable._init_existing()
Related
I am working on a open source persistance layer for a MQTT-Broker https://github.com/volkerjaenisch/amqtt_db
Incoming MQTT messages are irregular blobs of data so usually the DB-Backend is some kind of object storage.
I do it the hard way and deserialize the blobs into typed data colums and store them into a fast relational database. My finally target will be timescaleDB but first I go via SQLAlchemy to access a wide bunch of DBs with one API.
MQTT messages are volatile (think not always complete) so the DB scheme has to adjust dynamically e.g. adding new columns for new information.
First Message:
Time: 1234
Temperature : 23.4
Second Message:
Time: 1245
Temperature : 23.6
Rel Hum : 87 %
I have used SQLalchemy ORM for more than a decade but always for quite static databases. So I am new to work dynmically.
Utilizing the ORM to build DB tables dynamically from the structure of incoming MQTT-Messages was quite doable and worked out perfect.
But currently I am stuck with the case of additional information in the MQTT-Packages that extends the tables with new columns.
What I did so far:
Utilizing sqlalchemy-migration it was quite easy to dynamically add new columns to the existing table in the DB. In the code "topic_cls" is the declarative class and "column_def" a col_name - type mapping.
from migrate.versioning.schema import Table as MiTable, Column as MiColumn
def add_new_colums(self, topic_cls, column_def):
table_name = str(topic_cls.__table__.name)
table = MiTable(table_name, self.metadata)
for col_name, col_type in column_def.items():
col = MiColumn(col_name, col_type)
col.create(table)
Works like a charm. But how to get this changes to the DB reflected back into declarative classes? I tried to get a new inspection of the table:
new_table = Table(topic_cls.__table__.name, self.metadata, autoload_with=self.engine)
This also works but it gives me a new table but not a declarative base.
So my stupid questions are:
Is this the right way to achive my goal?
How can I get a declarative class by inspecting an already existing table in a DB?
"Drop the ORM and use SQL" is not the answer I am looking for.
Cheers,
Volker
Found a solution but it is a bit of a hack.
new_table = Table("test/topic_growth", Base.metadata, autoload_with=self.engine)
Base.metadata.remove(topic_cls.__table__)
new_dcl = type(str(table_name), (Base,), {'__table__': new_table})
Base.metadata._add_table(table_name, None, new_table)
After you obtained the new table via inspection, remove the old table entry from the metadata.
Then generate a new declarative base with the new table and same table name.
At last add the new table to the metadata.
When I try to reflect all tables in my Sybase DB
metadata = MetaData()
metadata.reflect(bind=engine)
SQLAlchemy runs the following query:
SELECT o.name AS name
FROM sysobjects o JOIN sysusers u ON o.uid = u.uid
WHERE u.name = #schema_name
AND o.type = 'U'
I then try to print the contents of metadata.tables, and this yields nothing.
I've tried creating an individual Table object and using the autoload=True option, but this yields a TableDoesNotExist error.
accounts = Table('Accounts', metadata, autoload=True, autoload_with=engine)
I looked into this query and it seems the #schema_name is becoming my username, and none of the tables which come from "sysobjects" appear to have a "name" attribute set to my username. They are all set to "dbo", which means the Database Owner, and thus the query returns nothing, and nothing is ever reflected. Is there any way to force SQLAlchemy to use something different as schema_name?
I've found two questions regarding table reflection using the Sybase dialect. Both were asked 6 years ago and seem to indicate that table reflection with Sybase was unsupported. However, it seems that SQLAlchemy tries to run a genuine sybase reflection query as above, so I don't think this is the case now.
I've solved this by setting the schema parameter on the MetaData object. I had to set it to dbo. You can also specify this in the reflect function.
I want to use SQLAlchemy to create a view in my PostgreSQL database. I'm using the CreateView compiler from sqlalchemy-views. I'm using the answer to this question as a reference:
How to create an SQL View with SQLAlchemy?
My code for creating the view looks like this:
def create_view(self, myparameter):
mytable = Table('mytable', metadata, autoload=True)
myview = Table('myview', metadata)
engine.execute(CreateView(myview, mytable.select().where(mytable.c.mycolumn==myparameter)))
However, when I attempt to run this query, the following exception is thrown:
KeyError: 'mycolumn_1'
Looking at the compiled query, it seems that a placeholder for my parameter value is not being replaced:
'\nCREATE VIEW myview AS SELECT mytable.mycolumn \nFROM mytable \nWHERE mytable.mycolumn = %(mycolumn_1)s\n\n'
Since the placeholder is not being replaced, the query obviously fails. However, I do not understand why the replacement does not happen, since my code does not differ much from the example.
My first suspicion was that maybe the type of the parameter and the column were incompatible. Currently, the parameter comes in as a unicode string, which should be mapped to a text column in my database. I have also tried mapping the parameter as a long to a bigint column with the same (failed) result.
Does anyone have another suggestion?
From the SQLAlchemy documentation, I can see that when one wants to pass the actual value that will be ultimately used at expression time, the bindparam() is used. A nice example is also provided:
from sqlalchemy import bindparam
stmt = select([users_table]).\
where(users_table.c.name == bindparam('username'))
I would like to be able to full text search across several text fields of one of my SQLAlchemy mapped objects. I would also like my mapped object to support foreign keys and transactions.
I plan to use MySQL to run the full text search. However, I understand that MySQL can only run full text search on a MyISAM table, which does not support transactions and foreign keys.
In order to accomplish my objective I plan to create two tables. My code will look something like this:
class User(Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
name = Column(String(50))
description = Column(Text)
users_myisam = Table('users_myisam', Base.metadata,
Column('id', Integer),
Column('name', String(50)),
Column('description', Text),
mysql_engine='MyISAM')
conn = Base.metadata.bind.connect()
conn.execute("CREATE FULLTEXT INDEX idx_users_ftxt \
on users_myisam (name, description)")
Then, to search I will run this:
q = 'monkey'
ft_search = users_myisam.select("MATCH (name,description) AGAINST ('%s')" % q)
result = ft_search.execute()
for row in result: print row
This seems to work, but I have a few questions:
Is my approach of creating two tables to solve my problem reasonable? Is there a standard/better/cleaner way to do this?
Is there a SQLAlchemy way to create the fulltext index, or am I best to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
Looks like I have a SQL injection problem in my search/match against query. How can I do the select the "SQLAlchemy way" to fix this?
Is there a clean way to join the users_myisam select/match against right back to my user table and return actual User instances, since this is what I really want?
In order to keep my users_myisam table in sync with my mapped object user table, does it make sense for me to use a MapperExtension on my User class, and set the before_insert, before_update, and before_delete methods to update the users_myisam table appropriately, or is there some better way to accomplish this?
Thanks,
Michael
Is my approach of creating two tables to solve my problem reasonable?
Is there a standard/better/cleaner way to do this?
I've not seen this use case attempted before, as developers who value transactions and constraints tend to use Postgresql in the first place. I understand that may not be possible in your specific scenario.
Is there a SQLAlchemy way to create the fulltext index, or am I best
to just directly execute "CREATE FULLTEXT INDEX ..." as I did above?
conn.execute() is fine though if you want something slightly more integrated you can use the DDL() construct, read through http://docs.sqlalchemy.org/en/rel_0_8/core/schema.html?highlight=ddl#customizing-ddl for details
Looks like I have a SQL injection problem in my search/match against query. How can I do the
select the "SQLAlchemy way" to fix this?
note: this recipe is only for MATCH against multiple columns simultaneously - if you have just one column, use the match() operator more simply.
most basically you could use the text() construct:
from sqlalchemy import text, bindparam
users_myisam.select(
text("MATCH (name,description) AGAINST (:value)",
bindparams=[bindparam('value', q)])
)
more comprehensively you could define a custom construct:
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.sql.expression import ClauseElement
from sqlalchemy import literal
class Match(ClauseElement):
def __init__(self, columns, value):
self.columns = columns
self.value = literal(value)
#compiles(Match)
def _match(element, compiler, **kw):
return "MATCH (%s) AGAINST (%s)" % (
", ".join(compiler.process(c, **kw) for c in element.columns),
compiler.process(element.value)
)
my_table.select(Match([my_table.c.a, my_table.c.b], "some value"))
docs:
http://docs.sqlalchemy.org/en/rel_0_8/core/compiler.html
Is there a clean way to join the users_myisam select/match against right back
to my user table and return actual User instances, since this is what I really want?
you should probably create a UserMyISAM class, map it just like User, then use relationship() to link the two classes together, then simple operations like this are possible:
query(User).join(User.search_table).\
filter(Match([UserSearch.x, UserSearch.y], "some value"))
In order to keep my users_myisam table in sync with my mapped object
user table, does it make sense for me to use a MapperExtension on my
User class, and set the before_insert, before_update, and
before_delete methods to update the users_myisam table appropriately,
or is there some better way to accomplish this?
MapperExtensions are deprecated, so you'd at least use the event API, and in most cases we want to try applying object mutations outside of the flush process. In this case, I'd be using the constructor for User, or alternatively the init event, as well as a basic #validates decorator which will receive values for the target attributes on User and copy those values into User.search_table.
Overall, if you've been learning SQLAlchemy from another source (like the Oreilly book), its really out of date by many years, and I'd be focusing on the current online documentation.
I'm developing a Pylons app which is based on exisitng database, so I'm using reflection. I have an SQL file with the schema that I used to create my test database. That's why I can't simply use drop_all and create_all.
I would like to write some unit tests and I faced the problem of clearing the database content after each test. I just want to erase all the data but leave the tables intact. Is this possible?
The application uses Postgres and this is what has to be used also for the tests.
I asked about the same thing on the SQLAlchemy Google group, and I got a recipe that appears to work well (all my tables are emptied). See the thread for reference.
My code (excerpt) looks like this:
import contextlib
from sqlalchemy import MetaData
meta = MetaData()
with contextlib.closing(engine.connect()) as con:
trans = con.begin()
for table in reversed(meta.sorted_tables):
con.execute(table.delete())
trans.commit()
Edit: I modified the code to delete tables in reverse order; supposedly this should ensure that children are deleted before parents.
For PostgreSQL using TRUNCATE:
with contextlib.closing(engine.connect()) as con:
trans = con.begin()
con.execute('TRUNCATE {} RESTART IDENTITY;'.format(
','.join(table.name
for table in reversed(Base.metadata.sorted_tables))))
trans.commit()
Note: RESTART IDENTITY; ensures that all sequences are reset as well. However, this is slower than the DELETE recipe by #aknuds1 by 50%.
Another recipe is to drop all tables first and then recreate them. This is slower by another 50%:
Base.metadata.drop_all(bind=engine)
Base.metadata.create_all(bind=engine)
How about using truncate:
TRUNCATE [ TABLE ] name [, ...]
(http://www.postgresql.org/docs/8.4/static/sql-truncate.html)
This will delete all the records in the table, but leave the schema in tact.