Here a simple sqlalchemy task, where i try to create instances of each table present in the database:
from sqlalchemy import MetaData, create_engine, Table
engine = create_engine("here my engine details...")
metadata = MetaData()
If i type engine.table_names() , I can see all my tables' names, for instance ['indicators', 'prices', 'scripts'].
I would normally go at creating instances of each of them as follow:
scripts = Table('scripts', metadata, autoload = True, autoload_with=engine)
indicators = Table('indicators', metadata, autoload = True, autoload_with=engine)
prices = Table('prices', metadata, autoload = True, autoload_with=engine)
But is there a way to create the Table instances without coding them explicitely?
Doing this:
tables = engine.table_names()
for table in tables:
table = Table( table , metadata, autoload = True, autoload_with=engine)
obviously doesn't work.
Any suggestion appreciated
You can do just that. This code will get you a list of tables:
my_tables = [Table(table,metadata,autoload=True,autoload_with=engine) for
table in engine.table_names()]
If you prefer a dictionary do this:
my_tables = {table:Table(table,metadata,autoload=True,autoload_with=engine)
for table in engine.table_names()}
With the dictionary you get O(1) lookup of tables when accessing the elements of your dictionary:
my_tables['indicators']
Related
Envirionment:
SQLAlchemy + PostgreSQL (pg8000)
Situation:
I created two table from psql console.
CREATE TABLE testidtable(
testid BIGSERIAL PRIMARY KEY NOT NULL
);
CREATE TABLE testidtable2(
testid bigint PRIMARY KEY NOT NULL,
CONSTRAINT testid_is_valid FOREIGN KEY(testid) REFERENCES testidtable (testid)
);
Here is the python code:
import sqlalchemy
from sqlalchemy import Table, MetaData
from config import Config
conf = Config()
engine = sqlalchemy.create_engine(
sqlalchemy.engine.URL.create(
"postgresql+pg8000",
database="databasename",
username=conf.db1_user,
password=conf.db1_pass),
client_encoding='utf-8')
metadata = MetaData()
metadata.reflect(bind=engine)
engine.connect()
table1 = Table("testidtable", metadata, autoload_with=engine)
ins1 = table1.insert().values({})
ret1 = engine.execute(ins1)
testid = ret1.inserted_primary_key[0]
table2 = Table("testidtable2", metadata, autoload_with=engine)
ins2 = table2.insert().values({"testid":testid})
ret2 = engine.execute(ins2)
And the error occurred:
sqlalchemy.exc.CompileError: Unconsumed column names: testid
I tried to show ins2 which is INSERT query for debugging, but faced the same situation.
How can I do INSERT testid into testidtable2 successfully?
Thanks.
I am trying to upsert into a postgres table where some of the columns have a '.' in their name.
example column names: country.name.
It would be best not to change the column name.
When I try to do this I get an error.
def upsert(df: DataFrame, engine: sql_engine) -> None:
with engine.connect() as conn:
base = automap_base()
base.prepare(engine, reflect=True, schema="some_schema")
table1= Table('table1', base.metadata,
autoload=True, autoload_with=engine, schema="some_schema")
stmt = insert(table1).values(df.to_dict('records'))
conn.execute(stmt.on_conflict_do_update(
constraint='table1_pkey',
set_=dict(country.name=stmt.excluded.country.name
)))
I get the following error:
SyntaxError: expression cannot contain assignment, perhaps you meant "=="?
I was trying to follow this recipe which was working fine until the name of the columns had a '.'
https://docs.sqlalchemy.org/en/14/dialects/postgresql.html#updating-using-the-excluded-insert-values
Any tips?
The sqlalchemy statement contains a excluded field where all the columns are, if you use this then it will work.
I created a "updated_dict" where all the columns names and object are from the excluded column. I filter out the primary keys.
This way it won't matter how the name of the columns are constructed.
def upsert(df: DataFrame, engine: sql_engine) -> None:
with engine.connect() as conn:
base = automap_base()
base.prepare(engine, reflect=True, schema="some_schema")
table1= Table('table1', base.metadata,
autoload=True, autoload_with=engine, schema="some_schema")
stmt = insert(table1).values(df.to_dict('records'))
update_dict = {
c.name: c
for c in stmt.excluded
if not c.primary_key
}
conn.execute(stmt.on_conflict_do_update(
constraint='table1_pkey',
set_=update_dict ))
I have a short SQL script which "copies" selected columns from a SQL table from one id (main_id=1) to two other ids (main_id=3 and 4) of the same table.
There are also some other ids which are part of the primary key of the table.
The script works fine using a PostgreSQL DB.
However, i would like to replace this using SQLAlchemy ORM, but i don't know how to do this.
UPDATE "MyTable" AS T
SET "Varname_1" = Cell."Varname_1",
"Varname_2" = Cell."Varname_2"
FROM "MyTable" AS Cell
WHERE T.id_A = Cell.id_A AND
T.id_B = Cell.id_B AND
Cell.main_id = 1 AND
T.main_id IN (3, 4);
Can anyone help me to "translate" this?
Not sure what you were having problems with, as I was able to do this by following the examples from Multiple Table Updates and Using Aliases and Subqueries sections of the tutorial:
import sqlalchemy
from sqlalchemy import create_engine
engine = create_engine('sqlite:///:memory:', echo=True)
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy import alias
metadata = MetaData()
my_table = Table('MyTable', metadata,
Column('id_A', Integer),
Column('id_B', Integer),
Column('main_id', Integer),
Column('varname_1', String),
Column('varname_2', String),
)
cell = my_table.alias("cell")
stmt = my_table.update(). \
where(my_table.c.id_A == cell.c.id_A). \
where(my_table.c.id_B == cell.c.id_B). \
where(cell.c.main_id == 1). \
where(my_table.c.main_id.in_([3, 4])). \
values(varname_1=cell.c.varname_1,
varname_2=cell.c.varname_2)
print(str(stmt))
print(stmt.compile().params)
I have the following code which throws the following error
engine = create_engine('postgresql+psycopg2:....', convert_unicode=True)
metadata = sqlalchemy.MetaData()
table = sqlalchemy.Table('omni.all_order', metadata,
sqlalchemy.Column('o_id', sqlalchemy.Integer),
sqlalchemy.Column('order', sqlalchemy.String),
)
ins = table.insert().values(all_rows)
engine.execute(ins)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation
"omni.all_order" does not exist
But the following two codes work fine
engine = create_engine('postgresql+psycopg2:....', convert_unicode=True)
result = engine.execute("SELECT * from omni.all_order ")
rows = result.fetchall()
print(rows)
--
engine = create_engine('postgresql+psycopg2:....', convert_unicode=True)
engine.execute("INSERT INTO omni.all_order (o_id) VALUES (1) ")
Creating another table first in the same schema (omni) throws the same error
engine = create_engine('postgresql+psycopg2:....', convert_unicode=True)
result = engine.execute("CREATE TABLE omni.all_order_s(o_id INT, order VARCHAR(80))")
metadata = sqlalchemy.MetaData()
table = sqlalchemy.Table('omni.all_order_s', metadata,
sqlalchemy.Column('o_id', sqlalchemy.Integer),
sqlalchemy.Column('order', sqlalchemy.String),
)
ins = table.insert().values(all_rows)
engine.execute(ins)
sqlalchemy.exc.ProgrammingError: (psycopg2.ProgrammingError) relation
"omni.all_order_s" does not exist
but creating it outside of the schema works fine
engine = create_engine('postgresql+psycopg2:....', convert_unicode=True)
result = engine.execute("CREATE TABLE all_order_s(o_id INT, order VARCHAR(80))")
metadata = sqlalchemy.MetaData()
table = sqlalchemy.Table('all_order_s', metadata,
sqlalchemy.Column('o_id', sqlalchemy.Integer),
sqlalchemy.Column('order', sqlalchemy.String),
)
ins = table.insert().values(all_rows)
engine.execute(ins)
Any ideas why this is?
Pass the table's schema using the schema= keyword argument instead of including it in the table's name:
table = sqlalchemy.Table('all_order', metadata,
sqlalchemy.Column('o_id', sqlalchemy.Integer),
sqlalchemy.Column('order', sqlalchemy.String),
schema='omni',
)
Currently it is quoted as a whole.
I had the same problem and I found the solution in this link: https://dba.stackexchange.com/questions/192897/postgres-relation-does-not-exist-error.
When you create the table name from a variable, the name is passed with quotes, so the name is case sensitive and need the quotes when you called again.
I do have database table that has an id primary key that is not an auto-increment (sequence). So it's up to the user to create an unique id or the insert will fail.
This table is not under my control, so I cannot change the database structure.
from sqlalchemy import create_engine, Table, MetaData
import psycopg2
db = create_engine('postgresql://...', echo=False).connect()
meta = MetaData()
meta.reflect(bind=db)
t = Table("mytable", meta, autoload=True, autoload_with=db)
values = { "title":"title", "id": ... }# ???
t.insert(bind=db, values=values).execute()
Given this is "single-user" / "single-client" system, you should be able to use the Column defaults: Python-Executed Functions. The example on the documentation linked to is enough to get you started. I would, however, use python function but with proper initialization from the datatabase adn still stored in a global variable:
def new_id_factory():
if not('_MYTABLE_ID_' in globals()):
q = db.execute("select max(mytable.id) as max_id from mytable").fetchone()
_MYTABLE_ID_ = (q and q.max_id) or 0
_MYTABLE_ID_ += 1
return _MYTABLE_ID_
t = Table("mytable", Base.metadata,
Column('id', Integer, primary_key=True, default=new_id_factory), #
autoload=True, autoload_with=db,
)