I have a little problem regarding "arranging of code, which depends on each other" when setting an sqlalchemy mapping to a sqlite database in python.
The goal is to write a script, whcih satisfies the following conditions:
Gets a filename parameter as command line argument.
Based on the filename it should create an absolute path to the SQLite database.
It should connect to the database and create an engine
It shall reflect the tables in this databases.
It should monkey patch the column id in the table mytable as a primary key column, since the table doesn't habe a primary key and sqlalchemy requires one.
So I came up with this:
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def create_path(file_name):
# generate absolute path to file_name
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
Base = declarative_base()
Base.metadata.create_all(engine)
class MyTable(Base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key.
def main(argv):
# parse file_name here from argv
Session = sessionmaker(bind=engine)
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
if __name__ == '__main__':
sys.exit(main())
But this is a doomed construction and I don't see how I could rearrange my code without breaking the dependencies.
To be able to create MyClass I need Base to be defined before MyClass.
To be able to create Base I need engine to be defined before Base.
To be able to create engine I need path_to_file to be defined before engine.
To be able to create path_to_file outside of main() I need create_file() to be defined before path_to_file.
And so on...
Hopefully you see where I am stuck...
Any suggestions?
Edit: By the way, the code works, but only with a hardcoded filename in the top of the script.
I do not see why you could not use the declarative mapping still complety. I think the key to the proper dependencies is to know what to put in the module and what in the script/function. With python, you can easily define the class inside the run_script function.
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker
from sqlalchemy.ext.declarative import declarative_base
def create_path(filename):
import os
fn = os.path.abspath(os.path.join(os.path.dirname(__file__), 'sqlite_files', filename))
return fn
def run_script(filename):
path_to_file = create_path(filename)
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=True)
Session = sessionmaker(bind=engine)
Base = declarative_base(bind=engine)
# object model
class MyTable(Base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key
# script itself
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
def main(argv):
assert argv, "must specify a database file name"
run_script(argv[0])
if __name__ == '__main__':
import sys
sys.exit(main(sys.argv))
I solved my problem using the classical mapping approach of SQLAlchemy:
from sqlalchemy import create_engine, Column, Integer, MetaData, Table
from sqlalchemy.orm import sessionmaker, mapper
class MyTable(object):
pass
def main():
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
metadata = MetaData()
metadata.bind = engine
my_table = Table('mytable',
metadata,
Column('id', Integer, primary_key=True), # Monkey patching the id column as primary key.
autoload=True,
autoload_with=engine)
mapper(MyTable, my_table)
Session = sessionmaker(bind=engine)
session = Session()
# Do Stuff!
I don't have the capability to do any testing at my current location. But I suggest moving all of that code to main() as follows. Also, MyTable subclasses declarative_base.
from sqlalchemy import create_engine, Column, Integer
from sqlalchemy.orm import sessionmaker from sqlalchemy.ext.declarative import declarative_base
class MyTable(declarative_base):
__tablename__ = 'my_table'
__table_args__ = {'autoload': True}
id = Column(Integer, primary_key=True) # Monkey patching the id column as primary key.
def create_path(file_name):
# generate absolute path to file_name
def main(argv): # parse file_name here from argv
path_to_file = create_path("my_file_name.sqlite")
engine = create_engine('sqlite:///{path}'.format(path=path_to_file), echo=False)
Base = MyTable()
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
for row in session.query(MyTable).all():
print row
return "Stop!"
if __name__ == '__main__':
sys.exit(main())
Related
I have a sqlite database with sqlalchemy that stores information about specific files on my filesystem. Something like this:
class FilePath(BaseModel):
id = Column(Integer, primary_key=True)
filepath = Column(String, nullable=False)
whatever = Column(String)
I store absolute paths in the filepath column, however not always the real path (i.e. there might be some symlinks somewhere in it. For example:
$ tree
.
└── mydata
├── linkdir -> realdir/
└── realdir
└── file.txt
The database could contain either the filepath string "mydata/realdir/file.txt", or "mydata/linkdir/file.txt" - no way to know which.
Is there a way to map the filepath column contents to their real paths at the time of the query?
# I have a path, I want to see if it's in the db
real_path = "mydata/realdir/file.txt"
# Run a query
fetched = session.query(FilePath).filter(FilePath.filepath == real_path)
# If no path was found, I can't say whether it's because the file isn't in db, or
# if it's present but recorded under a different logical path "mydata/linkdir/file.txt"
Is it possible to query this model, using the real paths as they exist at the time of the query?
You can use the create_function method in Python's sqlite3 module to create a custom realpath function in SQLite and then use it with func in a query:
import os
from sqlalchemy import create_engine, Column, String, func
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
engine = create_engine("sqlite:///:memory:")
def _realpath(virtpath):
return os.path.realpath(virtpath)
cnxn = engine.raw_connection()
cnxn.create_function("realpath", 1, _realpath)
Base = declarative_base()
class FileLocation(Base):
__tablename__ = "file_location"
path = Column(String, primary_key=True)
comment = Column(String)
def __repr__(self):
return f"<FileLocation(path='{self.path}')>"
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
session = Session()
thing = FileLocation(
path="/usr/bin/python3", comment="symlink to /usr/bin/python3.6"
)
session.add(thing)
session.commit()
result = (
session.query(FileLocation)
.filter(func.realpath(FileLocation.path) == "/usr/bin/python3.6")
.first()
)
print(result)
"""console output:
<FileLocation(path='/usr/bin/python3')>
"""
I'm using SQLAlchemy to connect to an in-memory SQLite database.
In my main.py file I have
engine = create_engine('sqlite:///:memory:', echo = True)
Base.metadata.create_all(engine)
Session = sessionmaker(bind=engine)
sess = Session()
After this I create a new thread, where I pass the session to
thread = MyThread(sess)
thread.start()
I have created my tables in tables.py like this
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import Column, Integer, String, Float
Base = declarative_base()
class My_Entry(Base):
__tablename__ = 'my_entry'
id = Column(Integer, primary_key=True)
name = Column(String)
url = Column(String)
My_Thread looks like this
class MyThread(Thread):
def __init__(sess):
Thread.__init__(self)
self.sess = sess
def run(self):
last_entry_url = self.sess.query(My_Entry).filter_by(name=key).all()[-1]
I try to query the database in the run function of my thread
last_entry_url = self.sess.query(My_Entry).filter_by(name=key).all()[-1]
but this throws an error
(sqlite3.OperationalError) no such table: my_entry
But creating the tables is one of the first things I do?
Indeed, using the echo=True gives me
CREATE TABLE my_entry (
id INTEGER NOT NULL,
name VARCHAR,
url VARCHAR,
PRIMARY KEY (id)
)
I have a sleep statement before running MyThread. My other tables work just fine, it's only this one table that doesn't.
bonus question: is there another way to get the last entry instead of all()[-1]? I assume there is but I didn't find anything
I don't understand why I have this error. Please explain the error.
I used official documentation.
I run Pipenv virtual env:
python 3.8.2
sqlalchemy 1.3.16
You can try run this code too.
import enum
from sqlalchemy import create_engine, Column, Integer, String, Enum
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
Session = sessionmaker(bind=engine)
session = Session()
class Type(str, enum.Enum):
ONE = "one"
TWO = "two"
class Item(Base):
__tablename__ = 'items'
id = Column(Integer, primary_key=True, index=True)
name = Column(String, unique=True, index=True)
type = Column(Enum(Type), default=Type.ONE, nullable=False)
item = Item(name="item_name", type="one")
session.add(item)
print(Item.__table__)
session.commit()
for name in session.query(Item.name):
print(name)
I added:
engine = create_engine('sqlite:///:memory:', echo=True)
Base = declarative_base()
Base.metadata.create_all(bind=engine)
Session = sessionmaker(bind=engine)
session = Session()
It creates the tables in the database (there are ways of using SQLAlchemy with preexisting tables, so an explicit instruction isd necessary).
In case it ends up helping someone, my issue was due to using an in memory sqlite db to unittest an API. Pytest would set up the database and tables with one connection while the API would create its own separate connection when the test were hitting endpoints. This ultimately solved it for me.
engine = create_engine("sqlite:///:memory:", poolclass=StaticPool, connect_args={'check_same_thread': False})
Ref: https://docs.sqlalchemy.org/en/14/dialects/sqlite.html#using-a-memory-database-in-multiple-threads
I succeeded to create a sql table with columns defined dynamically, thanks to python class reflexion.
But I cannot run the code more than one time.
For instance, the following import_file , should create a static table and a dynamic table with specific columns.
It works if I run it one time. But the second time it crashs and returns the following error :
Table 'dynamic' is already defined for this MetaData instance
Code example :
from sqlalchemy import Column, Integer, String, Float, Boolean, Table
from sqlalchemy import create_engine, MetaData
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker, relationship
from sqlalchemy.orm import clear_mappers
import os
def import_file(filename, columns):
path = filename
if os.path.exists(path):
os.remove(path)
engine = create_engine(f"sqlite:///{path}", echo=True)
clear_mappers()
Base = declarative_base()
class StaticTable(Base):
__tablename__ = "static"
id = Column(Integer, primary_key=True)
name = Column(String)
class DynamicTable(Base):
__tablename__ = "dynamic"
id = Column(Integer, primary_key=True)
for c in columns:
setattr(DynamicTable,c,Column(String))
Base.metadata.create_all(engine)
import_file("test.db", columns = ["age","test"]) # WORKS
import_file("test2.db", columns= ["id","age","foo","bar"]) # NOT WORKING
I try to use sqlalchemy.orm.clear_mappers, but unsucessfully.
Any idea how can I resolve it ?
I think that it is not full you code, because conflicts on Base.metadata
Mappers for link on orm classes and tables, you have defined tables.
You can try somethink like that
import_file("test.db", columns = ["age","test"])
Base.metadata.clear()
sqlalchemy.orm.clear_mappers()
import_file("test2.db", columns= ["id","age","foo","bar"])
I'm newer in SQLAlchemy I use some examples to create table and insert information to it and it's working 100% .
But what I didn't find is some example for how can I update & delete some information from the database.
What I'm doing is :
from sqlalchemy import Column, ForeignKey, Integer, String
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
Base = declarative_base()
## create
class Person(Base):
__tablename__ = 'person'
id = Column(Integer, primary_key=True)
name = Column(String(250), nullable=False)
engine = create_engine('sqlite:///database.db')
Base.metadata.create_all(engine)
## insert
Base.metadata.bind = engine
DBSession = sessionmaker(bind=engine)
session = DBSession()
new_person = Person(name='new person')
session.add(new_person)
session.commit()
## fetch
getperson = session.query(Person).first()
print getperson.name
# this will print : new person
# I need some example to how can I update and delete this : new person
So in this code it'll print "new person" my question is how can I update or delete it ?
Here's some example on each CRUD operation in sqlalchemy (ommiting Create, Read as you already know how to perform those):
First, necessary imports and configs for any operation:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker
# Category, Item, User are my tables
from database_setup import Base, Category, Item, User
# Generating session to connect to the db's ORM
engine = create_engine('sqlite:///catalogwithusers.db') # my db
Base.metadata.bind = engine
DBSession = sessionmaker(bind = engine)
session = DBSession()
Then peforming an update:
# Get the item filtering by it's id using a one() query on Item table
# If query is not empty, update the attributes, add query to session and commit
q = session.query(Item).filter_by(id=item_id).one()
if q != []:
q.name = edited_name
q.description = edited_description
session.add(q)
session.commit()
Finally, performing a deletion:
# Again get the item similarly to the example above
# Then if query returned results, use the delete method and commit
q = session.query(Item).filter_by(id=item_id).one()
if q != []:
session.delete(q)
session.commit()
These examples are taken from here. I suggest you have a look. ORM Creation is inside database_setup.py and CRUD ops are performed inside project.py and populatecatalog.py.