I succeeded to create a sql table with columns defined dynamically, thanks to python class reflexion.
But I cannot run the code more than one time.
For instance, the following import_file , should create a static table and a dynamic table with specific columns.
It works if I run it one time. But the second time it crashs and returns the following error :
Table 'dynamic' is already defined for this MetaData instance
Code example :
from sqlalchemy import Column, Integer, String, Float, Boolean, Table
from sqlalchemy import create_engine, MetaData
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.orm import clear_mappers
import os
def import_file(filename, columns):
path = filename
if os.path.exists(path):
os.remove(path)
engine = create_engine(f"sqlite:///{path}", echo=True)
clear_mappers()
Base = declarative_base()
class StaticTable(Base):
__tablename__ = "static"
id = Column(Integer, primary_key=True)
name = Column(String)
class DynamicTable(Base):
__tablename__ = "dynamic"
id = Column(Integer, primary_key=True)
for c in columns:
setattr(DynamicTable,c,Column(String))
Base.metadata.create_all(engine)
import_file("test.db", columns = ["age","test"]) # WORKS
import_file("test2.db", columns= ["id","age","foo","bar"]) # NOT WORKING
I try to use sqlalchemy.orm.clear_mappers, but unsucessfully.
Any idea how can I resolve it ?
I think that it is not full you code, because conflicts on Base.metadata
Mappers for link on orm classes and tables, you have defined tables.
You can try somethink like that
import_file("test.db", columns = ["age","test"])
Base.metadata.clear()
sqlalchemy.orm.clear_mappers()
import_file("test2.db", columns= ["id","age","foo","bar"])
Related
I'm having a problem with sqlalchemy in Python.
I have the following files :
base.py:
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
engine = create_engine('postgresql://postgres:mysecretpassword#localhost:5432/postgres',echo=True)
Base = declarative_base(engine)
Product.py:
from sqlalchemy import Table,Date,TEXT,Column,BIGINT,Integer,Boolean
from base import Base
class Product(Base):
__tablename__ = 'products'
id = Column('id',BIGINT, primary_key=True)
barcode = Column('barcode' ,BIGINT)
productName = Column('name', TEXT)
productType = Column('type', Integer)
maufactureName=Column('maufacture_name',TEXT,nullable=True)
manufactureCountry = Column('manufacture_country', TEXT)
manufacturerItemDescription = Column('manufacture_description',TEXT)
unitQuantity=Column('uniq_quantity',Integer)
quantity=Column('quantity',Integer)
quanityInPackage=Column('quantity_in_package',Integer)
isWeighted=Column('is_weighted',Integer)
picture=Column('picture_url',TEXT)
def __init__(self,args...):
.....
main.py:
from Product import Product
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
from base import Base
Base.metadata.create_all()
Session = sessionmaker()
session=Session()
session.add(Product(...))
session.commit()
When I run the main I keep getting an error that the products relation doesn't exist :
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "products" does not exist
Any idea why ? From the sqlalchemy logs it doesn't seems like it even tries to create the table.
Not sure what was the original root cause, but by following the help in the comments I was able to solve the problem. I'm leaving the updated code in the post so that others who face the same problem can see the solution
I have read in the following link:
Sqlalchemy adding multiple records and potential constraint violation
That using SQLAlchemy core library to perform the inserts is much faster option, rather than the ORM's session.add() method:
i.e:
session.add()
should be replaced with:
session.execute(Entry.__table__.insert(), params=inserts)
In the following code I have tried to replace .add with .insert:
from sqlalchemy import Column, DateTime, String, Integer, ForeignKey, func
from sqlalchemy.orm import relationship, backref
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
class Department(Base):
__tablename__ = 'department'
id = Column(Integer, primary_key=True)
name = Column(String)
class Employee(Base):
__tablename__ = 'employee'
id = Column(Integer, primary_key=True)
name = Column(String)
# Use default=func.now() to set the default hiring time
# of an Employee to be the current time when an
# Employee record was created
hired_on = Column(DateTime, default=func.now())
department_id = Column(Integer, ForeignKey('department.id'))
# Use cascade='delete,all' to propagate the deletion of a Department onto its Employees
department = relationship(
Department,
backref=backref('employees',
uselist=True,
cascade='delete,all'))
from sqlalchemy import create_engine
engine = create_engine('postgres://blah:blah#blah:blah/blah')
from sqlalchemy.orm import sessionmaker
session = sessionmaker()
session.configure(bind=engine)
Base.metadata.create_all(engine)
d = Department(name="IT")
emp1 = Employee(name="John", department=d)
s = session()
s.add(d)
s.add(emp1)
s.commit()
s.delete(d) # Deleting the department also deletes all of its employees.
s.commit()
s.query(Employee).all()
# Insert Option Attempt
from sqlalchemy.dialects.postgresql import insert
d = insert(Department).values(name="IT")
d1 = d.on_conflict_do_nothing()
s.execute(d1)
emp1 = insert(Employee).values(name="John", department=d1)
emp1 = emp1.on_conflict_do_nothing()
s.execute(emp1)
The error I receive:
sqlalchemy.exc.CompileError: Unconsumed column names: department
I can't quite understand the syntax and how to do it in the right way, I'm new to the SQLAlchemy.
It looks my question is similar to How to get primary key columns in pd.DataFrame.to_sql insertion method for PostgreSQL "upsert"
, so potentially by answering either of our questions, you could help two people at the same time ;-)
I am new to SQLAlchemy as well, but this is what I found :
Using your exact code, adding department only didn't work using "s.execute(d1)", so I changed it to the below and it does work :
with engine.connect() as conn:
d = insert(Department).values(name="IT")
d1 = d.on_conflict_do_nothing()
conn.execute(d1)
I found on SQLAlchemy documentation that in the past it was just a warning when you try to use a virtual column that doesn't really exist. But from version 0.8, it has been changed to an exception.
As a result, I am not sure if you can do that using the insert. I think that SQLAlchemy does it behind the scene in some other way when using session.add(). Maybe some experts can elaborate here.
I hope that will help.
I create a very simple database with sqlalchemy as follows:
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
Base = declarative_base()
class Person(Base):
__tablename__ = 'person'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
engine = create_engine('sqlite:///sqlalchemy_example.db')
# Create all tables in the engine. This is equivalent to "Create Table"
# statements in raw SQL.
Base.metadata.create_all(engine)
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
# Insert a Person in the person table
new_person = Person(name='new person')
session.add(new_person)
session.commit()
and then I tried to read it using pyDAL reference.
from pydal import DAL, Field
db = DAL('sqlite://sqlalchemy_example.db', auto_import=True)
db.tables
>> []
db.define_table('person', Field('name'))
>> OperationalError: table "person" already exists
How do I access the table using pyDAL?
thank you
First, do not set auto_import=True, as that is only relevant if pyDAL *.table migration metadata files exist for the tables, which will not be the case here.
Second, pyDAL does not know the table already exists, and because migrations are enabled by default, it attempts to create the table. To prevent this, you can simply disable migrations:
# Applies to all tables.
db = DAL('sqlite://sqlalchemy_example.db', migrate_enabled=False)
or:
# Applies to this table only.
db.define_table('person', Field('name'), migrate=False)
If you would like pyDAL to take over migrations for future changes to this table, then you should run a "fake migration", which will cause pyDAL to generate a *.table migration metadata file for this table without actually running the migration. To do this, temporarily make the following change:
db.define_table('person', Field('name'), fake_migrate=True)
After leaving the above in place for a single request, the *.table file will be generated, and you can remove the fake_migrate=True argument.
Finally, note that pyDAL expects the id field to be an auto-incrementing integer primary key field.
I have a little problem regarding "arranging of code, which depends on each other" when setting an sqlalchemy mapping to a sqlite database in python.
The goal is to write a script, whcih satisfies the following conditions:
Gets a filename parameter as command line argument.
Based on the filename it should create an absolute path to the SQLite database.
It should connect to the database and create an engine
It shall reflect the tables in this databases.
It should monkey patch the column id in the table mytable as a primary key column, since the table doesn't habe a primary key and sqlalchemy requires one.
So I came up with this:
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def create_path(file_name):
# generate absolute path to file_name
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
Base = declarative_base()
Base.metadata.create_all(engine)
class MyTable(Base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key.
def main(argv):
# parse file_name here from argv
Session = sessionmaker(bind=engine)
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
if __name__ == '__main__':
sys.exit(main())
But this is a doomed construction and I don't see how I could rearrange my code without breaking the dependencies.
To be able to create MyClass I need Base to be defined before MyClass.
To be able to create Base I need engine to be defined before Base.
To be able to create engine I need path_to_file to be defined before engine.
To be able to create path_to_file outside of main() I need create_file() to be defined before path_to_file.
And so on...
Hopefully you see where I am stuck...
Any suggestions?
Edit: By the way, the code works, but only with a hardcoded filename in the top of the script.
I do not see why you could not use the declarative mapping still complety. I think the key to the proper dependencies is to know what to put in the module and what in the script/function. With python, you can easily define the class inside the run_script function.
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def create_path(filename):
import os
fn = os.path.abspath(os.path.join(os.path.dirname(__file__), 'sqlite_files', filename))
return fn
def run_script(filename):
path_to_file = create_path(filename)
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=True)
Session = sessionmaker(bind=engine)
Base = declarative_base(bind=engine)
# object model
class MyTable(Base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key
# script itself
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
def main(argv):
assert argv, "must specify a database file name"
run_script(argv[0])
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
I solved my problem using the classical mapping approach of SQLAlchemy:
from sqlalchemy import create_engine, Column, Integer, MetaData, Table
from sqlalchemy.orm import sessionmaker, mapper
class MyTable(object):
pass
def main():
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
metadata = MetaData()
metadata.bind = engine
my_table = Table('mytable',
metadata,
Column('id', Integer, primary_key=True), # Monkey patching the id column as primary key.
autoload=True,
autoload_with=engine)
mapper(MyTable, my_table)
Session = sessionmaker(bind=engine)
session = Session()
# Do Stuff!
I don't have the capability to do any testing at my current location. But I suggest moving all of that code to main() as follows. Also, MyTable subclasses declarative_base.
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker from sqlalchemy.ext.declarative import declarative_base
class MyTable(declarative_base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key.
def create_path(file_name):
# generate absolute path to file_name
def main(argv): # parse file_name here from argv
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
Base = MyTable()
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
if __name__ == '__main__':
sys.exit(main())
The docs for Camelot say that it uses Elixir models. Since SQLAlchemy has included declarative_base for a while, I had used that instead of Elixir for another app. Now I would like to use the SQLAlchemy/declarative models directly in Camelot.
There is a post on Stackoverflow that says Camelot is not tied to Elixir and that using different models would be possible but it doesn't say how.
Camelot's original model.py only has this content:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, Integer, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
__metadata__ = metadata
I added my SQLAlchemy model and changed model.py to this:
import camelot.types
from camelot.model import metadata, Entity, Field, ManyToOne, OneToMany, Unicode, Date, using_options
from camelot.view.elixir_admin import EntityAdmin
from camelot.view.forms import *
from sqlalchemy import Column, Integer, String
from sqlalchemy.ext.declarative import declarative_base
Base = declarative_base()
__metadata__ = metadata
Base = declarative_base()
class Test(Base):
__tablename__ = "test"
id = Column(Integer, primary_key=True)
text = Column(String)
It didn't work. When I start main.py, I can see the GUI and Test in the sidebar, but can't see any rows. This is the tail of the traceback:
File "/usr/lib/python2.6/dist-packages/camelot/view/elixir_admin.py", line 52, in get_query
return self.entity.query
AttributeError: type object 'Test' has no attribute 'query'
This is the elixir_admin.py code for line 46-52:
#model_function
def get_query(self):
""":return: an sqlalchemy query for all the objects that should be
displayed in the table or the selection view. Overwrite this method to
change the default query, which selects all rows in the database.
"""
return self.entity.query
If this code is causing the problem, how do I overwrite the method to change the default query to make it work?
How can you use SQLAlchemy/declarative models in Camelot?
Here is some sample code on using Declarative to define a Movie model for Camelot, some explanation can be found here.
import sqlalchemy.types
from sqlalchemy import Column
from sqlalchemy.ext.declarative import ( declarative_base,
_declarative_constructor )
from camelot.admin.entity_admin import EntityAdmin
from camelot.model import metadata
import camelot.types
from elixir import session
class Entity( object ):
def __init__( self, **kwargs ):
_declarative_constructor( self, **kwargs )
session.add( self )
Entity = declarative_base( cls = Entity,
metadata = metadata,
constructor = None )
class Movie( Entity ):
__tablename__ = 'movie'
id = Column( sqlalchemy.types.Integer, primary_key = True )
name = Column( sqlalchemy.types.Unicode(50), nullable = False )
cover = Column( camelot.types.Image(), nullable = True )
class Admin( EntityAdmin ):
list_display = ['name']
form_display = ['name', 'cover']
Which version of Camelot are you using ?
With the current version of Camelot (11.12.30) it is possible to use Declarative through some
hacks. The upcoming version will make it much easier, while after this, the examples will be
ported to Declarative as well.