We Keep Coding

SQLAlchemy: ForeignKey across schemas

In my postgres server we have a database database with 2 schemas: public and api.
public has several tables, and I need to create a table in api with a foreign key to a table in public called model.
So it's:
-Schemas
--public
---tables
----models
--api
---tables
Using SQLAlchemy I have the following class:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
However upon launch I receive the following error:
sqlalchemy.exc.NoReferencedTableError: Foreign key associated with column 'training.model_id' could not find table 'public.model' with which to generate a foreign key to target column 'id'
It seems it does look for the correct thing but can't find it? I'm very confused as to why this is happening, especially because the error refers to not finding "public", which is created by default by postgres, rather than "api" which I created myself in pgAdmin.
Am I missing some cruicial config?

The error you are getting means that you are trying to create a foreign key referencing a table that SQLAlchemy does not know about. You can tell sqlalchemy about it by creating a Table associated with the same MetaData describing the referenced table. You can also do this using sqlalchemy's reflection capabilities. For example:
from sqlalchemy import create_engine, MetaData, Table, Column
class __PostgresService:
def __init__(self):
self.__client = create_engine("postgresql://postgres#localhost:5432/database")
metadata = MetaData(self.__client, schema="public")
metadata.reflect(schema="public", only=["model"])
self.__table = Table("training", metadata,
Column("id", String, primary_key=True, nullable=False),
Column("model_id", ForeignKey("model.id"), nullable=False),
schema="api")
metadata.create_all()
postgres_service = __PostgresService()
By default, MetaData.create_all() will check for the existence of tables first, before creating them, but you can also specify the exact tables to create: metadata.create_all(tables=[self.__table])

SqlLite with SqlAlchemy CreateTable result in database is locked

While trying to figure out why my tests fails to create sqlite schema, I tried this very simple example issued from https://pysheeet.readthedocs.io/en/latest/notes/python-sqlalchemy.html
from sqlalchemy import create_engine
from sqlalchemy import MetaData
from sqlalchemy import Table
from sqlalchemy import Column
from sqlalchemy import Integer, String
db_uri = 'sqlite:///db.sqlite'
engine = create_engine(db_uri)
# Create a metadata instance
metadata = MetaData(engine)
# Declare a table
table = Table('Example',metadata,
Column('id',Integer, primary_key=True),
Column('name',String))
# Create all tables
metadata.create_all()
for _t in metadata.tables:
print("Table: ", _t)
But still I get this db locked message:
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) database is locked [SQL: '\nCREATE TABLE "Example" (\n\tid INTEGER NOT NULL, \n\tname VARCHAR, \n\tPRIMARY KEY (id)\n)\n\n'] (Background on this error at: http://sqlalche.me/e/e3q8)
Any idea of how to use create all with non 'in memory' sqlite db ?

ProgrammingError, Flask with postgres and sqlalchemy

I am trying to get this setup to work, the database is created correctly, but trying to insert data I get the following error:
On sqlite:
sqlalchemy.exc.OperationalError
OperationalError: (sqlite3.OperationalError) no such column: Author [SQL: u'SELECT count(*) AS count_1 \nFROM (SELECT Author) AS anon_1']
On postgres:
sqlalchemy.exc.ProgrammingError
ProgrammingError: (psycopg2.ProgrammingError) column "author" does not exist
LINE 2: FROM (SELECT Author) AS anon_1
^
[SQL: 'SELECT count(*) AS count_1 \nFROM (SELECT Author) AS anon_1']
edit: Perhaps this has to do with it: I don't understand why it says "anon_1", as I am using credentials clearly?
I have inspected postgres and sqlite and the tables are created correctly. It seems to be an ORM configuration error, as it only seems to happend on inspecting or creating entries, any suggestion would be welcome!
class Author(CommonColumns):
__tablename__ = 'author'
author = Column(String(200))
author_url = Column(String(2000))
author_icon = Column(String(2000))
comment = Column(String(5000))
registerSchema('author')(Author)
SETTINGS = {
'SQLALCHEMY_TRACK_MODIFICATIONS': True,
'SQLALCHEMY_DATABASE_URI': 'sqlite:////tmp/test.db',
# 'SQLALCHEMY_DATABASE_URI': 'postgresql://xxx:xxx#localhost/test',
}
application = Flask(__name__)
# bind SQLAlchemy
db = application.data.driver
Base.metadata.bind = db.engine
db.Model = Base
db.create_all()
if __name__ == "__main__":
application.run(debug=True)

What is the query you're using to insert data?
I think the error messages may be a bit more opaque than they need to be because you're using Author/author in three very similar contexts:
the Class name
the table name
the column name
For easier debugging, the first thing I'd do is temporarily make each one unique (AuthorClass, author_table, author_column) so you can check which 'Author' is actually being referred to by the error message.
Since you're using the ORM, I suspect the underlying issue is that your insert statement uses Author (the object) when it should actually be using Author.author (the attribute/column name). The SELECT statements are complaining that they can't find the column 'author', but because you use author for both the table and column name, it's unclear what's actually being passed into the SQL statement.

Sqlalchemy drop_all(engine) Oracle sequence is not dropped

With Python + Sqlalchemy + Oracle, trying to drop all tables and recreate them. Using oracle sequence in Id column for autoincreament,but drop all is not dropping sequence.
engine = create_engine('oracle://user:pass#host:port/db',
implicit_returning=False,
echo=False)
Base = declarative_base(bind=engine)
if DROP_AND_CREATE:
Base.metadata.drop_all(bind=engine)
meta_data = MetaData()
meta_data = Base.metadata
from domains import users
meta_data.create_all(engine, checkfirst=False)
domain package sample,
class Users(Base):
__tablename__ = 'users'
id = Column(Integer, Sequence('users_id_seq'), primary_key=True)
name = Column(String(255))
in the above all tables are dropped except I can see the sequences I am using are still present in oracle db. if i manually delete them and run again they are running fine.
The user_id_seq created in oracle is not getting dropped. please help.
Error message:
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-00955: name is already used by an existing object
[SQL: 'CREATE SEQUENCE user_queries_id_seq']

Finally after few hours of googling... still learning the language and framework., but this code works and does exactly what i want in my application...
Thanks all
Base = declarative_base(bind=engine)
from domains import users
if DROP_AND_CREATE:
Base.metadata.drop_all(bind=engine, checkfirst=True)
logger.info('Creating all registered tables.')
Base.metadata.create_all(bind=engine, checkfirst=True)

SQLAlchemy printing raw SQL from create()

I am giving Pylons a try with SQLAlchemy, and I love it, there is just one thing, is it possible to print out the raw SQL CREATE TABLE data generated from Table().create() before it's executed?

from sqlalchemy.schema import CreateTable
print(CreateTable(table))
If you are using declarative syntax:
print(CreateTable(Model.__table__))
Update:
Since I have the accepted answer and there is important information in klenwell answer, I'll also add it here.
You can get the SQL for your specific database (MySQL, Postgresql, etc.) by compiling with your engine.
print(CreateTable(Model.__table__).compile(engine))
Update 2:
#jackotonye Added in the comments a way to do it without an engine.
print(CreateTable(Model.__table__).compile(dialect=postgresql.dialect()))

You can set up you engine to dump the metadata creation sequence, using the following:
def metadata_dump(sql, *multiparams, **params):
# print or write to log or file etc
print(sql.compile(dialect=engine.dialect))
engine = create_engine(myDatabaseURL, strategy='mock', executor=metadata_dump)
metadata.create_all(engine)
One advantage of this approach is that enums and indexes are included in the printout. Using CreateTable leaves this out.
Another advantage is that the order of the schema definitions is correct and (almost) usable as a script.

I needed to get the raw table sql in order to setup tests for some existing models. Here's a successful unit test that I created for SQLAlchemy 0.7.4 based on Antoine's answer as proof of concept:
from sqlalchemy import create_engine
from sqlalchemy.schema import CreateTable
from model import Foo
sql_url = "sqlite:///:memory:"
db_engine = create_engine(sql_url)
table_sql = CreateTable(Foo.table).compile(db_engine)
self.assertTrue("CREATE TABLE foos" in str(table_sql))

Something like this? (from the SQLA FAQ)
http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html

It turns out this is straight-forward:
from sqlalchemy.dialects import postgresql
from sqlalchemy.schema import CreateTable
from sqlalchemy import Table, Column, String, MetaData
metadata = MetaData()
users = Table('users', metadata,
Column('username', String)
)
statement = CreateTable(users)
print(statement.compile(dialect=postgresql.dialect()))
Outputs this:
CREATE TABLE users (
username VARCHAR
)
Going further, it can even support bound parameters in prepared statements.
Reference
How do I render SQL expressions as strings, possibly with bound parameters inlined?
...
or without an Engine:
from sqlalchemy.dialects import postgresql
print(statement.compile(dialect=postgresql.dialect()))
SOURCE: http://docs.sqlalchemy.org/en/latest/faq/sqlexpressions.html#faq-sql-expression-string
Example: Using SQL Alchemy to generate a user rename script
#!/usr/bin/env python
import csv
from sqlalchemy.dialects import postgresql
from sqlalchemy import bindparam, Table, Column, String, MetaData
metadata = MetaData()
users = Table('users', metadata,
Column('username', String)
)
renames = []
with open('users.csv') as csvfile:
for row in csv.DictReader(csvfile):
renames.append({
'from': row['sAMAccountName'],
'to': row['mail']
})
for rename in renames:
stmt = (users.update()
.where(users.c.username == rename['from'])
.values(username=rename['to']))
print(str(stmt.compile(dialect=postgresql.dialect(),
compile_kwargs={"literal_binds": True})) + ';')
When processing this users.csv:
sAMAccountName,mail
bmcboatface,boaty.mcboatface#example.com
ndhyani,naina.dhyani#contoso.com
Gives output like this:
UPDATE users SET username='boaty.mcboatface#example.com' WHERE users.username = 'bmcboatface';
UPDATE users SET username='naina.dhyani#contoso.com' WHERE users.username = 'ndhyani';users.username = 'ndhyani';
Why a research vessel has an email address is yet to be determined. I have been in touch with Example Inc's IT team and have had no response.

May be you mean echo parameter of sqlalchemy.create_engine?
/tmp$ cat test_s.py
import sqlalchemy as sa
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Department(Base):
__tablename__ = "departments"
department_id = sa.Column(sa.types.Integer, primary_key=True)
name = sa.Column(sa.types.Unicode(100), unique=True)
chief_id = sa.Column(sa.types.Integer)
parent_department_id = sa.Column(sa.types.Integer,
sa.ForeignKey("departments.department_id"))
parent_department = sa.orm.relation("Department")
engine = sa.create_engine("sqlite:///:memory:", echo=True)
Base.metadata.create_all(bind=engine)
/tmp$ python test_s.py
2011-03-24 15:09:58,311 INFO sqlalchemy.engine.base.Engine.0x...42cc PRAGMA table_info("departments")
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc ()
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc
CREATE TABLE departments (
department_id INTEGER NOT NULL,
name VARCHAR(100),
chief_id INTEGER,
parent_department_id INTEGER,
PRIMARY KEY (department_id),
UNIQUE (name),
FOREIGN KEY(parent_department_id) REFERENCES departments (department_id)
)
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc ()
2011-03-24 15:09:58,312 INFO sqlalchemy.engine.base.Engine.0x...42cc COMMIT