I have a table whose name must be created dynamically. Here is how the table is created:
def create_data_table(table_name):
meta = Base.metadata
my_table= Table(
table_name, meta,
Column('id', BigInteger, primary_key=True, nullable=False),
Column('file_name', String(250), nullable=False)
)
meta.create_all(engine, [my_table])
This works just fine. However, I also need to pass the table name as a variable when I insert rows, and that's where I am running into problems.
Here are the things I have tried:
def insert_data(table_name, data):
meta = Base.metadata
my_table = Table(
table_name, meta,
Column('id', BigInteger, primary_key=True),
Column('file_name', String(250))
)
conn = engine.connect()
conn.execute(my_table.insert(), data)
This results in an error, telling me that my_table is already defined in my metadata.
To avoid this, I have tried creating a class instead:
class my_table(Base):
__tablename__ = ''
id = Column('id', String(50), primary_key=True)
file_name = Column('file_name', String(250))
def __init__(self, table_name, id, file_name):
__tablename__ = table_name
self.id = id
self.file_name = file_name
Again, I was able to create the table without any problems, but SQLAlchemy did not allow me to pass the table name as a parameter when I tried to insert rows.
Ideally, I would like to use the first approach. Any help is appreciated!
The SQLAlchemy MetaData object has a mapping of all of the tables it holds through the MetaData.tables attribute. Which is:
A dictionary of Table objects keyed to their name or “table key”.
You do not want to recreate the Table either directly or through the declarative mapping, just access it by name from MetaData.tables. E.g.:
Base.metadata.tables[table_name]
Your insert_data() implementation can simply be:
def insert_data(table_name, data):
my_table = Base.metadata.tables[table_name]
conn = engine.connect()
conn.execute(my_table.insert(), data)
Related
I have an object
class Summary():
__tablename__ = 'employeenames'
name= Column('employeeName', String(128, collation='utf8_bin'))
date = Column('dateJoined', Date)
I want to patch Summary with a mock object
class Summary():
__tablename__ = 'employeenames'
name= Column('employeeName', String)
date = Column('dateJoined', Date)
or just patch name the field to name= Column('employeeName', String)
The reason I'm doing this is that I'm doing my tests in sqlite and some queries that are only for Mysql are interfering with my tests.
I think it would be difficult to mock the column. However you could instead conditionally compile the String type for Sqlite, removing the collation.
import sqlalchemy as sa
from sqlalchemy import orm
from sqlalchemy.ext.compiler import compiles
from sqlalchemy.types import String
#compiles(String, 'sqlite')
def compile_varchar(element, compiler, **kw):
type_expression = kw['type_expression']
type_expression.type.collation = None
return compiler.visit_VARCHAR(element, **kw)
Base = orm.declarative_base()
class Summary(Base):
__tablename__ = 'employeenames'
id = sa.Column(sa.Integer, primary_key=True)
name = sa.Column('employeeName', sa.String(128, collation='utf8_bin'))
date = sa.Column('dateJoined', sa.Date)
urls = ['mysql:///test', 'sqlite://']
for url in urls:
engine = sa.create_engine(url, echo=True, future=True)
Base.metadata.drop_all(engine)
Base.metadata.create_all(engine)
This script produces the expected output for MySQL:
CREATE TABLE employeenames (
id INTEGER NOT NULL AUTO_INCREMENT,
`employeeName` VARCHAR(128) COLLATE utf8_bin,
`dateJoined` DATE,
PRIMARY KEY (id)
)
but removes the collation for Sqlite:
CREATE TABLE employeenames (
id INTEGER NOT NULL,
"employeeName" VARCHAR(128),
"dateJoined" DATE,
PRIMARY KEY (id)
)
I want to create a column (Id) of type uniqueidentifier in sqlalchemy in a table called Staging.Transactions. Also, I want the column to automatically generate new guids for inserts.
What I want to accomplish is the following (expressed in sql)
ALTER TABLE [Staging].[Transactions] ADD DEFAULT (newid()) FOR [Id]
GO
The code in sqlalchemy is currently:
from sqlalchemy import Column, Float, Date
import uuid
from database.base import base
from sqlalchemy_utils import UUIDType
class Transactions(base):
__tablename__ = 'Transactions'
__table_args__ = {'schema': 'Staging'}
Id = Column(UUIDType, primary_key=True, default=uuid.uuid4)
AccountId = Column(UUIDType)
TransactionAmount = Column(Float)
TransactionDate = Column(Date)
def __init__(self, account_id, transaction_amount, transaction_date):
self.Id = uuid.uuid4()
self.AccountId = account_id
self.TransactionAmount = transaction_amount
self.TransactionDate = transaction_date
When I create the schema from the python code it does not generate the constraint that I want in SQL - that is - to auto generate new guids/uniqueidentifiers for the column [Id].
If I try to make a manual insert I get error message: "Cannot insert the value NULL into column 'Id', table 'my_database.Staging.Transactions'; column does not allow nulls. INSERT fails."
Would appreciate tips on how I can change the python/sqlalchemy code to fix this.
I've found two ways:
1)
Do not use uuid.uuid4() in init of your table class, keep it simple:
class Transactions(base):
__tablename__ = 'Transactions'
__table_args__ = {'schema': 'Staging'}
Id = Column(String, primary_key=True)
...
def __init__(self, Id, account_id, transaction_amount, transaction_date):
self.Id = Id
...
Instead use it in the creation of a new record:
import uuid
...
new_transac = Transacatins(Id = uuid.uuid4(),
...
)
db.session.add(new_transac)
db.session.commit()
Here, db is my SQLAlchemy(app)
2)
Without uuid, you can use raw SQL to do de job (see SQLAlchemy : how can I execute a raw INSERT sql query in a Postgres database?).
Well... session.execute is a Sqlalchemy solution...
In your case, should be something like this:
table = "[Staging].[Transactions]"
columns = ["[Id]", "[AccountId]", "[TransactionAmount]", "[TransactionDate]"]
values = ["(NEWID(), \'"+str(form.AccountId.data) +"\', "+\
str(form.TransactionAmount.data) +", "+\
str(form.TransactionDate.data))"]
s_cols = ', '.join(columns)
s_vals = ', '.join(values)
insSQL = db.session.execute(f"INSERT INTO {table} ({s_cols}) VALUES {s_vals}")
print (insSQL) # to see if SQL command is OK
db.session.commit()
You have to check where single quotes are really needed.
I tired to use the SqlAlchemy orm to build the api to insert the values into database from uploaded excel files. when I tested on the codes it kept showing the error:
TypeError: __init__() missing 1 required positional argument: 'id'
I've updated the id key to primary key, auto increment, unique and unsigned in my local MySql data base. I believe the system cannot insert the primary key automatically because it works if I assign the value to id manually
transaction_obj = Transaction(id=1, name="David", date="2018-03-03",
product="fruit", quantity=20, amount=12.55)
Here is model.py
from sqlalchemy import Table, MetaData, Column, Integer, String, DATE, DECIMAL,ForeignKey, DateTime
from sqlalchemy.orm import mapper
metadata = MetaData()
customers = Table('customers', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(20)),
Column('phone', String(20)),
Column('address', String(45)),
Column('source_from', String(45))
)
class Customers(object):
def __init__(self, name, phone, address, source_from):
self.name = name
self.phone = phone
self.address = address
self.source_from = source_from
def __repr__(self):
return "<Customer(name='%s', phone='%s', address='%s', " \
"source_from='%s')" % (self.name, self.phone, self.address,
self.source_from)
mapper(Customers, customers)
transaction = Table('transaction', metadata,
Column('id', Integer, primary_key=True),
Column('name', String(20)),
Column('date', DateTime),
Column('product', String(20)),
Column('quantity', Integer),
Column('amount',DECIMAL(2))
)
class Transaction(object):
def __index__(self, name, date, product, quantity, amount):
self.name = name
self.date = date
self.product = product
self.quantity = quantity
self.amount = amount
def __repr__(self):
return "<Transaction(name='%s', date='%s', product='%s'," \
"quantity='%s', amount='%s')>" % (self.name, self.date,
self.product, self.quantity,
self.amount)
mapper(Transaction, transaction)
Here is my test coding: test.py
import json
import os
import os
import json
from sqlalchemy import create_engine
import config
import pandas as pd
conn = config.conn_str
def tran_test():
engine = create_engine(conn)
Session_class = sessionmaker(bind=engine)
Session = Session_class
# generate the object for the data we would like to insert
transaction_obj = Transaction(name="David", date="2018-03-03",
product="fruit", quantity=20, amount=12.55)
Session.add(transaction_obj)
Session.commit()
def test_uploaded_file(file):
df = pd.read_excel(file)
return df.info()
if __name__ == '__main__':
# set_env_by_setting('prod')
# conn_str = os.environ.get('ConnectionString')
# print(conn_str)
# test_uploaded_file("-1.xlsx")
tran_test()
I'm using SQLAlchemy==1.2.10, PyMySQL==0.9.2.
I'm doubting if I'm using the wrong format in model.py. Please advise. Thx.
While I'm not sure about the pattern you are using, (manually mapping to your table classes) I think you would have a much easier time making use of declarative_base which does this for you.
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
Then make sure your models inherit Base
from sqlalchemy import (
Column,
Integer,
String
)
class Customers(Base):
__tablename__ = 'customer'
id = Column(Integer, primary_key=True) # Auto-increment should be default
name = Column(String(20))
# Etc.
def __repr__(self):
return "<Customer(name='%s', phone='%s', address='%s', " \
"source_from='%s')" % (self.name, self.phone, self.address,
self.source_from)
And finally use Base to create your table:
Base.metadata.create_all(engine)
Here is a good reference to basic declarative use cases. It gets a little more complicated depending on how you are scaffolding your app but its a great starting point:
http://docs.sqlalchemy.org/en/latest/orm/extensions/declarative/basic_use.html
I'm trying to set up a secondary many-to-many relationship from one table to two others, via a third in the middle that links to all three. I have two files - one for ORM objects (model.py) and one for schema objects (schema.py) They look like this:
model.py
import schema
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import *
class AbstractBase(object):
def __repr__(self):
macros = ["%s=%s" % (key, getattr(self, key, None)) for key in self.__table__.columns.keys()]
rep = "<%s(%s)>" % (self.__class__.__name__, str.join(', ', macros))
return rep
Base = declarative_base(cls=AbstractBase)
class A(Base):
__table__ = schema.a_table
dees = relationship("D",
secondary=schema.b_table,
primaryjoin="A.a_id==b_table.c.a_id",
secondaryjoin="b_table.c.c_id==D.d_id")
cees = relationship("C",
secondary=schema.b_table,
primaryjoin="A.a_id==schema.b_table.c.a_id",
secondaryjoin="b_table.c.d_id==C.c_id",
backref="a_collection")
class C(Base):
__table__ = schema.c_table
class D(Base):
__table__ = schema.d_table
schema.py
from sqlalchemy import *
from sqlalchemy.dialects.mysql import *
metadata = MetaData()
a_table = Table(
'a',
metadata,
Column("a_id", INTEGER(), primary_key=True, nullable=False),
Column("date", DATETIME(timezone=True)),
)
b_table = Table(
'shipment_runs',
metadata,
Column("a_id", ForeignKey("a.a_id"), primary_key=True,),
Column("c_id", ForeignKey("c.c_id"), primary_key=True),
Column("d_id", ForeignKey("d.d_id")),
)
c_table = Table(
'c',
metadata,
Column('c_id', INTEGER(), primary_key=True, nullable=False),
Column('name', VARCHAR(64), unique=True),
)
d_table = Table(
'd',
metadata,
Column('d_id', INTEGER(), primary_key=True, nullable=False)
)
Unfortunately, instantiating this results in the following error:
sqlalchemy.exc.InvalidRequestError: When initializing mapper Mapper|A|a, expression 'A.a_id==b_table.c.a_id' failed to locate a name ("name 'b_table' is not defined"). If this is a class name, consider adding this relationship() to the class after both dependent classes have been defined.
Is there a way I can change my imports or make the mapper be aware of the objects in the schema module somehow?
Was able to get it by doing the following:
class B(Base):
__table__ = schema.b_table
class A(Base):
__table__ = schema.a_table
dees = relationship("D",
secondary=b.__table__,
primaryjoin="A.a_id==B.a_id",
secondaryjoin="B.c_id==D.d_id")
cees = relationship("C",
secondary=B.__table__,
primaryjoin="A.a_id==B.a_id",
secondaryjoin="B.d_id==C.c_id",
backref="a_collection")
All credit goes to this question:
SQLAlchemy Relationship Error: object has no attribute 'c'
In general, you can use the table name in a string, or drop the string and use your actual references.
primaryjoin="A.a_id==shipment_runs.c.a_id",
primaryjoin=schema.a_table.c.a_id==schema.b_table.c.a_id,
That being said, given that you have the ForeignKeys set up in your tables, SQLAlchemy is smart enough that you don't even need the joins for a simple relationship, just secondary.
c_list = relationship("C", secondary=schema.b_table, backref="a_list")
(I think the "C" and "D" are swapped in your example?)
I have a question concerning the mapping of entities in SQLAlchemy.
I have a transient object, which already contains foreign keys to some persistent objects. I want that SQLAlchemy fetches the referenced objects and assigns them to their relationship-attributes. From the SQLAlchemy documentation, I thought that I have to use the merge-operation on the session to achieve this. But in my configuration, it doesn't work.
This is a minimum example demonstrating my problem:
# -*- coding: utf-8 -*-
from sqlalchemy import create_engine
from sqlalchemy.orm import mapper
from sqlalchemy import Table, Column, Integer, String, MetaData, ForeignKey
from sqlalchemy.orm import relationship, sessionmaker
class User(object):
def __init__(self, id, name, fullname, password, best_friend_id=None):
self.id = id
self.name = name
self.fullname = fullname
self.password = password
self.best_friend_id = best_friend_id
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
class Dog(object):
def __init__(self, id, name):
self.id = id
self.name = name
def __repr__(self):
return "<User('%s','%s', '%s')>" % (self.name, self.fullname, self.password)
engine = create_engine('sqlite:///:memory:', echo=True)
Session = sessionmaker(bind=engine)
session = Session()
metadata = MetaData()
dogs_table = Table('dogs', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
)
users_table = Table('users', metadata,
Column('id', Integer, primary_key=True),
Column('name', String),
Column('fullname', String),
Column('password', String),
Column('best_friend_id', Integer, ForeignKey('dogs.id'))
)
metadata.create_all(engine)
mapper(User, users_table, properties={'best_friend': relationship(Dog, uselist=False)})
mapper(Dog, dogs_table)
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret', best_friend_id=1)
session.add(dog)
session.commit()
merged_lordling = session.merge(lordling)
print str(merged_lordling.best_friend.name)
I expect that merged_lordling.best_friend contains the dog 'Hasso'. But it is still None.
I was bit by this same problem recently. Once you established a relationship, you should simply assign your Dog instance to User.best_friend directly, not explicitly using the foreign key. I don't know why exactly that happens, but while investigating a similar problem I realized that if you do that, SQLAlchemy doesn't populate the relationship property until you flushed all the related instances.
So, instead of:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend_id=1)
session.add(dog)
Simply do:
dog = Dog(id=1, name='Hasso')
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=dog)
session.add(lordling)
Or even:
lordling = User(id=2, name='John', fullname='Miller', password='very_secret',
best_friend=Dog(id=1, name='Hasso'))
session.add(lordling)
As a general rule, avoid using the foreign key columns directly when you have a relationship established. Embrace the ORM, and only assign or query directly from foreign keys when you really have no other choice. I learned that the hard way.