i started learning python3 with SQLAlchemy. No i created a quick example Database with a User Table and an Group Table.
CREATE TABLE `USER` (
`ID` int(11) NOT NULL AUTO_INCREMENT,
`NAME` varchar(255) NOT NULL,
`USERNAME` varchar(255) NOT NULL,
`group_id` int(11) NOT NULL,
PRIMARY KEY (`ID`),
KEY `fk_groups_idx` (`group_id`),
CONSTRAINT `fk_groups` FOREIGN KEY (`group_id`) REFERENCES `GROUPS` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB AUTO_INCREMENT=6 DEFAULT CHARSET=latin1;
CREATE TABLE `GROUPS` (
`id` int(11) NOT NULL AUTO_INCREMENT,
`GroupName` varchar(45) NOT NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB AUTO_INCREMENT=2 DEFAULT CHARSET=latin1;
Quick and dirty, just for learning it. Every User is in a group (not multiple) and every Group can relate to many user.
So it is an Many-to-one relationship if i am not wrong.
To build my Relationship bidirectional (with the example from SqlAlchemy) i have to back_populate both classes in my mapping. But when i start my test, i get the following message:
sqlalchemy.exc.NoForeignKeysError: Could not determine join condition between parent/child tables on relationship Groups.users - there are no foreign keys linking these tables. Ensure that referencing columns are associated with a ForeignKey or ForeignKeyConstraint, or specifya 'primaryjoin' expression.
My code is just on script for testing. Maybe someone with more experience could point me to my problem:
from sqlalchemy import ForeignKey
from sqlalchemy.orm import relationship
from sqlalchemy.orm import sessionmaker
from datetime import datetime, timedelta
from sqlalchemy import Table, Column, Integer, String, DateTime, ForeignKey
from sqlalchemy.orm import relationship, backref
from sqlalchemy import create_engine
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import PrimaryKeyConstraint
from sqlalchemy import ForeignKeyConstraint
Base = declarative_base()
class User(Base):
__tablename__ = 'USER'
ID = Column('ID', Integer, primary_key=True)
name = Column('NAME',String)
username = Column('USERNAME', String)
group_id = Column('group_id', Integer, ForeignKey('group_id'))
group = relationship("Groups", back_populates="users")
class Groups(Base):
__tablename__ = 'GROUPS'
id = Column('id', Integer, primary_key=True)
groupName = Column('GroupName', String)
users = relationship("User", back_populates="group")
engine = create_engine('mysql://test:test#localhost/TEST')
Session = sessionmaker(bind=engine)
session = Session()
for instance in session.query(User).order_by(User.ID):
print(instance.ID, instance.username, instance.group_id)
session.close();
Regards
Issue is with foreign key declaration for group_id and it can be done like below to resolve error - sqlalchemy.exc.NoForeignKeysError
group_id = Column('group_id', Integer, ForeignKey('GROUPS.id'))
Related
This question already has answers here:
SQLAlchemy printing raw SQL from create()
(6 answers)
Closed 3 years ago.
I have a very simple One-to-Many database schema (a parent can have many children but a child can only have one parent). My SQLAlchemy models looks like so:
from sqlalchemy import Table, Column, Integer, ForeignKey
from sqlalchemy.orm import relationship
from models import Base
class Parent(Base):
__tablename__ = 'Parent'
id = Column(Integer, primary_key=True)
children = relationship('Child', backref='parent')
class Child(Base):
__tablename__ = 'Child'
id = Column(Integer, primary_key=True)
parent_id = Column(Integer, ForeignKey('Parent.id'))
I'm able to create the tables in the database using the following commands:
engine = create_engine('mysql://localhost:3306/testdb')
Base.metadata.create_all(engine)
I'm curious what the raw SQL looks like to create these tables. I'd imagine it would look something like this, but I would like to be sure:
CREATE TABLE Parent (
id INT NOT NULL AUTO_INCREMENT,
PRIMARY KEY (id)
);
CREATE TABLE Child (
id INT NOT NULL AUTO_INCREMENT,
parent_id int,
PRIMARY KEY (id),
CONSTRAINT FK_Parent FOREIGN KEY (parent_id) REFERENCES Parent(id)
);
Is there anyway to generate the database schema in raw sql using SQLAlchemy? I know I can generate a query in raw sql but I'm wondering how to generate the initial database schema.
On top of that, is there anyway to generate the schema depending on the actual database type (e.g. the raw sql would look slightly different for MySQL and PostgreSQL)?
how-can-i-get-the-create-table-drop-table-output-as-a-string
from sqlalchemy import MetaData, Table, Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.schema import CreateTable
import sqlalchemy
print(sqlalchemy.__version__)
Base = declarative_base()
class Parent(Base):
__tablename__ = 'Parent'
id = Column(Integer, primary_key=True)
children = Column(String(255))
user = Table('users', MetaData(bind=None),
Column('id', Integer(), primary_key=True, nullable=False),
Column('name', String()),
Column('fullname', String()),
Column('password', String()), schema=None)
print(CreateTable(Parent.__table__))
print(CreateTable(user))
Output:
1.3.0b1
CREATE TABLE "Parent" (
id INTEGER NOT NULL,
children VARCHAR(255),
PRIMARY KEY (id)
)
CREATE TABLE users (
id INTEGER NOT NULL,
name VARCHAR,
fullname VARCHAR,
password VARCHAR,
PRIMARY KEY (id)
)
I have a python app and I use sqlalchemy and posgreSQL. I have already some data in the DB and do not want to recreate it, so I want to add a table to the DB and keep all data (earlier I used alembic for that, but I want to use raw SQL this time).
This is my new table, which I want to add to the DB in python:
# User payed data
class UserPayData(Base):
__tablename__ = 'userspaydata'
id = Column(Integer, primary_key=True)
account_owner = Column(Text, nullable=False)
IBAN = Column(Text, nullable=False)
# Foreign Keys
belongs_to_user_id = Column(Integer, ForeignKey('users.id'))
payment_belongs_to_user_relation = relationship("User", back_populates="payment_belongs_to_user_addresses")
def __init__(self, account_owner=None, IBAN=None):
self.account_owner = account_owner
self.IBAN = IBAN
def get(self, id):
if self.id == id:
return self
else:
return None
def __repr__(self):
return '<%s(%r, %r, %r)>' % (self.__class__.__name__, self.id, self.account_owner, self.IBAN)
Here is the relevant part from the users table:
# Project
class User(UserMixin, Base):
__tablename__ = 'users'
id = Column(Integer, primary_key=True)
# etc...
# Foreign Key payment
payment_belongs_to_user_addresses = relationship('UserPayData', back_populates="payment_belongs_to_user_relation")
This is the raw SQL which I want to execute:
CREATE TABLE 'userspaydata'
(
account_owner TEXT NOT NULL,
IBAN TEXT NOT NULL,
belongs_to_user_id INT references users(id)
);
I have found many examples and some are different.
I have a few questions:
Do I have to create a primary key aswell? AFAIK, if the primary key is not defined by me it will be automatically ID INT
Relationships do not need to be defined on the DB level, they are defined on the app level or am I wrong? If I am wrong, how I define the relationship in raw SQL?
I have not submitted the SQL query yet, is the syntax correct?
Do I have to create a primary key aswell
Yes, if you want to work with the tables using sqlalchemy Declarative Mappings, and in general its a great idea to specify the primary key.
Relationships do not need to be defined on the DB level, they are defined on the app level or am I wrong?
Foreign Keys defined in your Sqlalchemy Declarative classes are mapped to database Foreign Key Constraints.
Again, relationships in relational databases are another great idea. Specify foreign key dependencies unless you have a specific reason not to.
If you don't specify constraints in the database you can end up with corrupted data.
If you are emitting DDL statements manually, you can specify the foreign key constraints. Foreign Keys in the Postgres Documentation
I have not submitted the SQL query yet, is the syntax correct?
No, Here's a corrected version of your DDL statement.
CREATE TABLE userspaydata (
id INTEGER PRIMARY KEY,
account_owner TEXT NOT NULL,
IBAN TEXT NOT NULL,
belongs_to_user_id INT references users(id) -- this is a FK constraint
);
Note that id doesn't auto-increment. For auto-increment id
change the id column definition to
-- postgrseql 10 & above
id INTEGER PRIMARY KEY GENERATED BY DEFAULT AS IDENTITY
-- postgresql 9.6 & below
id SERIAL PRIMARY KEY
and change the sqlalchemy mapping for id column to
id = Column(Integer, primary_key=True, auto_increment=True)
This is my solution and it seems to work (I also added datetime):
CREATE TABLE userspaydata (
Id SERIAL primary key,
account_owner TEXT NOT NULL,
IBAN TEXT NOT NULL,
date_added TIMESTAMP WITH TIME ZONE,
belongs_to_user_id INT references users(id)
);
PK must be defined by me.
Relationships are handled on the app level.
I've got an issue with sqlite3 and SQLAlchemy, where my foreign key is created with a ForeignKey with ondelete="RESTRICT", and every session executes "PRAGMA foreign_keys = on", but SQLAlchemy still allows me to orphan a child. Attempting to delete the same row in sqlite3 itself properly throws a foreign key error.
Here's my SQLAlchemy models:
class User(Base):
__tablename__ = 'users'
username = Column(String, nullable=False, unique=True, primary_key=True)
account_type = Column(String, ForeignKey("account_types.name", ondelete="RESTRICT"), nullable=False)
class AccountType(Base):
__tablename__ = 'account_types'
name = Column(String, nullable=False, unique=True, primary_key=True)
description = Column(String)
Now, I open a connection to the database and do the following:
engine = create_engine("sqlite3:///mydatabase")
Session = sessionmaker(bind=engine)
session = Session()
session.execute("PRAGMA foreign_keys = on")
# This gives me the expected result:
# I can't add a user which references an account type I haven't
# created yet
new_user = User(username="abc123",
account_type="admin") # "admin" doesn't exist yet
session.add(new_user)
with pytest.raises(sqlalchemy.exc.IntegrityError):
session.commit()
session.rollback()
new_account_type = AccountType(name="admin",
description="blah blah blah")
session.add(new_account_type)
session.commit()
# Now add our new user (Horray, this now works!)
session.add(new_user)
session.commit()
# Now let's try to delete the account type
# This should raise an integrity error, but it doesn't!
# SELECT * FROM account_types and SELECT * FROM users
# show that the user still exists, but the account type doesn't
session.query(AccountType).filter_by(name="admin").delete()
session.commit()
Now, take SQLAlchemy out of the equation, clear the tables, and manually INSERT/DELETE this data, and sqlite3 is honoring the "ON DELETE RESTRICT":
sqlite> PRAGMA foreign_keys = on;
sqlite> DELETE FROM users;
sqlite> DELETE FROM account_types;
sqlite> .schema users
CREATE TABLE users (
username VARCHAR NOT NULL,
account_type VARCHAR NOT NULL,
PRIMARY KEY (username),
UNIQUE (username),
FOREIGN KEY(account_type) REFERENCES account_types (name) ON DELETE RESTRICT
);
sqlite> .schema account_types
CREATE TABLE account_types (
name VARCHAR NOT NULL,
description VARCHAR,
PRIMARY KEY (name),
UNIQUE (name)
);
sqlite> INSERT INTO account_types VALUES ("admin", "blah blah blah");
sqlite> INSERT INTO users VALUES ("abc123", "admin");
sqlite> DELETE FROM account_types WHERE name = "admin";
Error: FOREIGN KEY constraint failed
=====================================================
Why won't SQLAlchemy (1.0.12) respect the "ON DELETE RESTRICT" constraint in sqlite3?
If it matters, the tables (and the constraints) were all generated through SQLAlchemy:
engine = create_engine("sqlite3:///mydatabase")
Base.metadata.create_all(engine)
I wrestled with this for about two hours before posting, then about an hour after posting, I figured it out. Here's the answer for any other lonely SQLAlchemy newbie who reaches this page now or years down the road. "session.rollback()" was "undoing" my previous execution of "PRAGMA foreign_keys = on".
I had pulled that method of setting up the session from some previous internal source code, so I assumed it was okay. The manual lays out exactly how this should be handled at http://docs.sqlalchemy.org/en/latest/dialects/sqlite.html#foreign-key-support
In short, I added:
from sqlalchemy import event
from sqlalchemy.engine import Engine
#event.listens_for(Engine, "connect")
def set_sqlite_pragma(dbapi_connection, connection_record):
cursor = dbapi_connection.cursor()
cursor.execute("PRAGMA foreign_keys=ON")
cursor.close()
to my module which contains all my models, and it just "works". This method doesn't need to be invoked in any way. Now my foreign key constraints are always being respected, even post-rollback.
I have an existing database that I'm trying to map into SqlAlchemy's ORM. I want it to just figure out the ForiegnKey relations that already exist in the database itself.
Here's the code I have so far:
from sqlalchemy import create_engine, MetaData, Table
from sqlalchemy.orm import create_session
from sqlalchemy.ext.declarative import declarative_base
# connect to database and infer from structure
Base = declarative_base()
engine = create_engine("XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX")
metadata = MetaData(bind=engine)
class Club(Base):
__table__ = Table('clubs', metadata, autoload=True)
def __repr__(self):
return "<Club: %s>" % (self.name,)
class Member(Base):
__table__ = Table('members', metadata, autoload=True)
def __repr__(self):
return "<Member: %s of %d>" % (self.name, self.club_id)
Here's the SQL table dump:
CREATE TABLE `clubs` (
`id` int(11) NOT NULL auto_increment,
`name` varchar(45) collate utf8_bin default NULL,
PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_bin;
CREATE TABLE `members` (
`id` int(11) NOT NULL auto_increment,
`club_id` int(11) default NULL,
`name` VARCHAR(100) default NULL,
PRIMARY KEY (`id`),
KEY `fk_members_club_idx` (`club_id`),
CONSTRAINT `fk_members_club` FOREIGN KEY (`club_id`) REFERENCES `clubs` (`id`) ON DELETE NO ACTION ON UPDATE NO ACTION
) ENGINE=InnoDB DEFAULT CHARSET=utf8;
My problem is that in the Member __repr__ function, for example, instead of printing the club_id (which is useless to human), I'd like to print the Club name. Something like member.club.name would be ideal.
I couldn't find out how to do this on the SqlAlchemy docs unless I defined my own tables, not merely reflecting them in my code, which is what I'm doing here.
Just change your Member class to look like below:
class Member(Base):
__table__ = Table('members', metadata, autoload=True)
club = relationship(Club, backref="members")
def __repr__(self):
return "<Member: %s of %s>" % (self.name, self.club.name)
The point being that the reflection (autoload) will not automatically create relationships between classes, so you have to define them explicitly.
Using the SqlAlchemy ORM, I've been trying to setup a one-to-many relationship between a parent table that has "child" records in a child table. The table and mapper declarations are below. The problem is, only one child record is read from the DB into the "children" attribute when there should be many records.
child_table = Table('child_table', metadata_obj,
Column('field_1', Integer, ForeignKey('parent_table.field_1'),
primary_key=True),
Column('field_2', String, ForeignKey('parent_table.field_2'),
primary_key=True),
autoload=True
mapper(Parent_Class, parent_table, properties={
'children': relation(Child_Class, lazy=False, # Eager load
primaryjoin=and_(child_table.c.field_1==parent_table.c.field_1,
parent_table.c.field_2==parent_table.c.field_2),
backref="parents"),
})
The problem was in the child_table definition. I had only defined the fields with a foreign key relationship as being part of the primary key for the child_table. However, those fields (field_1 & field_2) weren't the complete primary key. So, it appears that SQLAlchemy was only loading one row. I added field_3 and field_4 to fully define the primary key for the child_table and then SQLAlchemy successfully read all records.
child_table = Table('child_table', metadata_obj,
Column('field_1', Integer, ForeignKey('parent_table.field_1'),
primary_key=True),
Column('field_2', String, ForeignKey('parent_table.field_2'),
primary_key=True),
Column('field_3', String, primary_key=True),
Column('field_4', String, primary_key=True),
autoload=True