I have a table that already exists:
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"), primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255))
)
I created this table by running this:
CONN = create_engine(DB_URL, client_encoding="UTF-8")
META_DATA = MetaData(bind=CONN, reflect=True)
# ... table code
META_DATA.create_all(CONN, checkfirst=True)
the first time it worked and I was able to create the table. However, the 2nd time around I got this error:
sqlalchemy.exc.InvalidRequestError: Table 'users' is already defined for this MetaData instance. Specify 'extend_existing=True' to redefine options and columns on an existing Table object.
which makes sense since the table users already exists. I'm able to see if the table exists like so:
TABLE_EXISTS = CONN.dialect.has_table(CONN, "users")
However, how do I actually get the existing table object? I can't find this anywhere in the documentation. Please help.
We have 3 different approaches here:
assume that required tables have been created already, reflecting them and getting with MetaData.tables dictionary field like
from sqlalchemy import MetaData, create_engine
CONN = create_engine(DB_URL, client_encoding="UTF-8")
META_DATA = MetaData(bind=CONN, reflect=True)
USERS_TABLE = META_DATA.tables['users']
removing reflect flag from MetaData object initialization, because we don't use it and moreover – trying to create tables that've been already reflected:
from sqlalchemy import (MetaData, Table, Column, Integer, String, Sequence,
create_engine)
CONN = create_engine('sqlite:///db.sql')
META_DATA = MetaData(bind=CONN)
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"),
primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255)))
META_DATA.create_all(CONN, checkfirst=True)
assuming that we are keeping reflected table if it was previously created by setting in Table object initializer keep_existing flag to True:
from sqlalchemy import (MetaData, Table, Column, Integer, String, Sequence,
create_engine)
CONN = create_engine('sqlite:///db.sql')
META_DATA = MetaData(bind=CONN, reflect=True)
USERS_TABLE = Table("users", META_DATA,
Column("id", Integer, Sequence("user_id_seq"),
primary_key=True),
Column("first_name", String(255)),
Column("last_name", String(255)),
keep_existing=True)
META_DATA.create_all(CONN, checkfirst=True)
Which one to choose? Depends on your use case, but I prefer second one since it looks like you aren't using reflection, also it is simplest modification: just removing flag from MetaData initializer.
P. S.
we can always make reflection after initialization of MetaData object with MetaData.reflect method:
META_DATA.reflect()
also we can specify which tables to reflect with only parameter (may be any iterable of str objects):
META_DATA.reflect(only=['users'])
and many more.
This works for me pretty well -
import sqlalchemy as db
engine = db.create_engine("your_connection_string")
meta_data = db.MetaData(bind=engine)
db.MetaData.reflect(meta_data)
USERS = meta_data.tables['users']
# View the columns present in the users table
print(USERS.columns)
# You can run sqlalchemy queries
query = db.select([
USERS.c.id,
USERS.c.first_name,
USERS.c.last_name,
])
result = engine.execute(query).fetchall()
Note that using reflect parameter in Metadata(bind=engine, reflect=True) is deprecated and will be removed in a future release. Above code takes care of it.
__table_args__ = {'extend_existing': True}
right below __tablename__
If you're using async Sqlalchemy, you can use
metadata = MetaData()
async with engine.connect() as conn:
await conn.run_sync(metadata.reflect, only=["harshit_table"])
harshit_table = Table("harshit_table", metadata, autoload_with=engine)
print("tables: ", harshit_table, type(harshit_table))
I'm quite new to this, but what worked for me was this (variables are declared in the original question)
USERS_TABLE_NEW = Table("users", META_DATA, autoload_with=CONN)
Related
In my postgres server we have a database database with 2 schemas: public and api.
public has several tables, and I need to create a table in api with a foreign key to a table in public called model.
So it's:
-Schemas
--public
---tables
----models
--api
---tables
Using SQLAlchemy I have the following class:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
However upon launch I receive the following error:
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'training.model_id' could not find table 'public.model' with which to generate a foreign key to target column 'id'
It seems it does look for the correct thing but can't find it? I'm very confused as to why this is happening, especially because the error refers to not finding "public", which is created by default by postgres, rather than "api" which I created myself in pgAdmin.
Am I missing some cruicial config?
The error you are getting means that you are trying to create a foreign key referencing a table that SQLAlchemy does not know about. You can tell sqlalchemy about it by creating a Table associated with the same MetaData describing the referenced table. You can also do this using sqlalchemy's reflection capabilities. For example:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
metadata.reflect(schema="public", only=["model"])
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
By default, MetaData.create_all() will check for the existence of tables first, before creating them, but you can also specify the exact tables to create: metadata.create_all(tables=[self.__table])
I'd like to log whether or not SQLAlchemy has to create any database tables when create_all() is called, however I don't see any documentation on a return value from create_all(). How do I go about this?
I've tried setting up a simple in-memory database using the following code and called create_all(). With echo set to true, I can see that the table is created as expected, but myreturn has a type of NoneType. If I call create_all() a second time, the table isn't created, and myreturn is still NoneType.
from sqlalchemy import create_engine, Column, Integer, String, ForeignKey
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
Base = declarative_base()
class User(Base):
__tablename__ = 'testtable'
id = Column('id', Integer, primary_key=True)
name = Column('name', String, unique=True)
engine = create_engine('sqlite:///:memory:', echo=True)
myreturn = Base.metadata.create_all(bind=engine)
Is there a way to identify whether the tables are created by create_all() or do I need to create additional logic to verify this in the database directly before invoking create_all() ?
You can use the after_create event. The event handler will be passed a keyword arg called tables which is a collection of any tables that were created within the create_all() method call.
from sqlalchemy import event
#event.listens_for(Base.metadata, 'after_create')
def receive_after_create(target, connection, tables, **kw):
"listen for the 'after_create' event"
if tables:
print('A table was created')
else:
print('A table was not created')
I created the table on the server using SQLALchemy:
from sqlalchemy import create_engine, Table, Column, String, MetaData
engine = create_engine('mssql://server/database?driver=SQL+Server&trusted_connection=yes')
meta = MetaData()
table = Table('test17', meta,
Column('id', Integer, primary_key=True),
Column('name', String('255'))
)
metadata.create_all(engine)
Then I connected to this database using SSMS 2012 and added a new column:
ALTER TABLE test17 ADD age INT NULL
How do I tell using SQLALchemy that a new column appears in the table?
I tried to do something like:
meta2 = MetaData()
table = Table('test17', meta, autoload=True, autoload_with=engine)
But in the end I get the same table structure that I defined initially using SQLALchemy.
I think you forgot to bind your MetaData to the engine. Create the engine first, then read the metadata from the db using the engine.
metadata = db.MetaData(bind=engine)
test17 = db.Table('test17', metadata, autoload=True)
Considering Storm, a python ORM, I would like to automatically generate the schema for a (mysql) database. The home page states
"Storm works well with existing database schemas." ( https://storm.canonical.com/FrontPage ),
hence I was hoping to not having to create model classes. However, the 'getting started' tutorial ( https://storm.canonical.com/Tutorial ) suggests that a class, like the one below, needs to be created manually for each table, and each field needs to be manually specified:
class Person(object):
__storm_table__ = "person"
id = Int(primary=True)
name = Unicode()
Alternatively, SQLAlchemy doesn't seem to support a reverse engineering feature either, but does need a schema like this one:
user = Table('user', metadata,
Column('user_id', Integer, primary_key = True),
Column('user_name', String(16), nullable = False),
Column('email_address', String(60)),
Column('password', String(20), nullable = False)
)
Of course, these classes/schemas make sense, since each table will likely represent some 'object of interest' and one can extend them with all kinds of functionality. However, they are tedious to create, and their (initial) content is straight forward when a database already exists.
One ORM that does allow for reverse engineering is:
http://docs.doctrine-project.org/en/2.0.x/reference/tools.html
Are there similar reverse engineering tools for Storm or SQLAlchemy or any python ORM or python database fancyfier?
I am not aware of how Storm manages this process, but you can certainly reflect tables in a database with sqlalchemy. For example, below is a basic example using a SQL Server instance that I have access to at the moment.
AN ENTIRE DATABASE
>>> from sqlalchemy import create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>') # replace <username> with user name etc.
>>> meta = MetaData()
>>> meta.reflect(bind=engine)
>>> funds_table = meta.tables['funds'] # tables are stored in meta.tables dict
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
If you merely want to reflect one table at a time, you can use the following code instead.
ONE TABLE AT A TIME (much faster)
>>> from sqlalchemy import Table, create_engine, MetaData
>>> engine = create_engine('mssql+pyodbc://<username>:<password>#<host>/<database>')
>>> meta = MetaData()
>>> funds_table = Table('funds', meta, autoload=True, autoload_with=engine) # indicate table name (here 'funds') with a string passed to Table as the first argument
>>> funds_table # now stores database schema object
Table(u'funds', MetaData(bind=None), Column(u'fund_token', INTEGER(), table=<funds>, primary_key=True, nullable=False), Column(u'award_year_token', INTEGER(), ForeignKey(u'award_year_defn.award_year_token'), table=<funds>, nullable=False), ... Column(u'fin_aid_disclosure_category', VARCHAR(length=3, collation=u'SQL_Latin1_General_CP1_CI_AS'), table=<funds>), Column(u'report_as_additional_unsub', BIT(), table=<funds>, server_default=DefaultClause(<sqlalchemy.sql.expression.TextClause object at 0x000000000545B6D8>, for_update=False)), schema=None)
As you can probably imagine, you can then save the relevant tables' data for accessing the tables more quickly again in the future.
I have a simple data structure, where a film table has a foreign key to country table.
In order to retrieve all the films that are from the same country, I have this property 'same_country_films', a self-referential relationship.
It almost does the job correctly, however, it also includes the film itself in the list. How can I exclude it and just have other films?
Many thanks!
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import mapper, relationship
metadata = MetaData()
country_table = Table('country', metadata,
Column('id', String, primary_key=True),
Column('name', String),
)
film_table = Table('film', metadata,
Column('id', Integer, primary_key=True),
Column('title', String),
Column('year', Integer),
Column('country_id', Integer, ForeignKey('country.id'))
)
class Country(object):
pass
class Film(object):
pass
mapper(Country, country_table)
mapper(Film, film_table,
properties={
'country':relationship(
Country,
backref='films'),
'same_country_films':relationship(
Film,
primaryjoin=film_table.c.country_id==\
film_table.c.country_id,
foreign_keys=[
film_table.c.country_id,
]
)
}
)
The simplest solution is to code this property yourself instead of relation:
class Film(object):
#property
def same_country_films(self):
return [f for f in self.country.films if f!=self]
This solution won't do separate query for this property when both film.same_country_films and country.films are accessed during session life. The property can't be updated as you can usualy do with relation, but I doubt it's realy needed.
The bad thing is that it's evaluated for each access (not so much work). You can change property decorator to chaching one (like cached_property in werkzeug), but then the property won't reflect changes in country.films after first access to it.
This should do it I think, (though I haven't actually tested it):
primaryjoin=sqlalchemy.and_(
film_table.c.country_id==film_table.c.country_id,
film_table.c.id!=film_table.c.id)