We Keep Coding

How to automap the result set of a custom SQL query in SQLAlchemy

I'd like to run raw SQL queries through SQLAlchemy and have the resulting rows use python types which are automatically mapped from the database type. This AutoMap functionality is available for tables in the database. Is it available for any arbitrary resultset?
As an example, we build small sqlite database:
import sqlite3
con = sqlite3.connect('test.db')
cur = con.cursor()
cur.execute("CREATE TABLE Trainer (id INTEGER PRIMARY KEY, first_name VARCHAR(50), last_name VARCHAR(50), dob DATE, tiger_skill FLOAT);")
cur.execute("INSERT INTO Trainer VALUES (1, 'Joe', 'Exotic', '1963-03-05', 0.6)")
cur.execute("INSERT INTO Trainer VALUES (2, 'Carole', 'Baskin', '1961-06-06', 0.3)")
cur.close()
con.commit()
con.close()
And uing SQLAlchemy, I query the newly created database "test.db":
from sqlalchemy import create_engine
engine = create_engine("sqlite:///test.db")
connection = engine.connect()
CUSTOM_SQL_QUERY = "SELECT count(*) as total_trainers, min(dob) as first_dob from Trainer"
result = connection.execute(CUSTOM_SQL_QUERY)
for r in result:
print(r)
>>> (2, '1961-06-06')
Notice that the second column in the result set is a python string, not a python datetime.date object. Is there a way for sqlalchemy to automap an arbitrary result set? Or is this automap reflection capability limited to just actual tables in the database?

sqlite3 insert not committing

When trying to insert rows into a table with a unique index, it appears to simply silently not insert.
I've captured the behaviour in the following program: on the second call to test_insert I should get an integrity violation on the unique key. But nothing. Also, if I take the c.execute(query, [id_to_test]) line and duplicate itself below it, I do receive the proper integrity constraint as expected. What's happening here?
import sqlite3
def test_insert(id_to_test):
conn = sqlite3.connect('test.db')
c = conn.cursor()
query = '''INSERT INTO test(unique_id)
VALUES(?)'''
c.execute(query, [id_to_test])
def setup_table():
conn = sqlite3.connect('test.db')
c = conn.cursor()
c.execute('''DROP TABLE IF EXISTS test''')
c.execute('''CREATE TABLE test (unique_id text)''')
c.execute('''CREATE UNIQUE INDEX test_unique_id ON test (unique_id)''')
if __name__ == '__main__':
setup_table()
test_insert('test_id')
test_insert('test_id')
test_insert('test_id')

At the end of database operations, commit the changes to the database:
conn.commit()

Flask-SQLAlchemy check if table exists in database

Flask-SQLAlchemy check if table exists in database.
I see similar problems, but I try not to succeed.
Flask-SQLAlchemy check if row exists in table
I have create a table object ,like this:
<class'flask_sqlalchemy.XXX'>,
now how to check the object if exists in database.
I do many try:
eg:
for t in db.metadata.sorted_tables:
print("tablename",t.name)
some table object is created before,but it doesnt exists in database,and now they. all print.
eg:print content is
tablename： table_1
tablename： table_2
tablename： table_3
but only table_1 is exist datable，table_2 and table_3 is dynamica create，now I only want use the table_1.
very thanks.

I used these methods. Looking at the model like you did only tells you what SHOULD be in the database.
import sqlalchemy as sa
def database_is_empty():
table_names = sa.inspect(engine).get_table_names()
is_empty = table_names == []
print('Db is empty: {}'.format(is_empty))
return is_empty
def table_exists(name):
ret = engine.dialect.has_table(engine, name)
print('Table "{}" exists: {}'.format(name, ret))
return ret
There may be a simpler method than this:
def model_exists(model_class):
engine = db.get_engine(bind=model_class.__bind_key__)
return model_class.metadata.tables[model_class.__tablename__].exists(engine)

the solution is too easy, just write this two rows in you code and it should work fine for you
from flask_sqlalchemy import SQLAlchemy, inspect
...
inspector = inspect(db.engine)
print(inspector.has_table("user")) # output: Boolean
have a nice day

SQL Alchemy's recommended way to check for the presence of a table is to create an inspector object and use its has_table() method.
The following example was copied from sqlalchemy.engine.reflection.Inspector.has_table, with the addition of an SQLite engine (in memory) to make it reproducible:
In [17]: from sqlalchemy import create_engine, inspect
...: from sqlalchemy import MetaData, Table, Column, Text
...: engine = create_engine('sqlite://')
...: meta = MetaData()
...: meta.bind = engine
...: user_table = Table('user', meta, Column("first_name", Text))
...: user_table.create()
...: inspector = inspect(engine)
...: inspector.has_table('user')
Out[17]: True
You can also use the user_table metadata element name to check if it exists as such:
inspector.has_table(user_table.name)

creating a table in sqlite3 python

I apologize in advance for asking such a basic question but I am new to SQlite3 and having trouble starting. I am trying to build a database with one table. I used the following code to build a table.
import sqlite3
conn = sqlite3.connect('example.db')
c = conn.cursor()
c.execute('''CREATE TABLE mytable
(start, end, score)''')
but whenever I try to update or access the table it seems that it doesnt exist or maybe it exists in a different database. I also tried creating a table called example.mytable but I got the error:
sqlite3.OperationalError: unknown database example
What am I missing?
Thanks

I think that a commit is needed after inserts (schema changes such as new tables should automatically commit). I would suggest adding the full path to your database as well to make sure you are accessing the same location next time round.
Here is an extension on your code:
import sqlite3
def create():
try:
c.execute("""CREATE TABLE mytable
(start, end, score)""")
except:
pass
def insert():
c.execute("""INSERT INTO mytable (start, end, score)
values(1, 99, 123)""")
def select(verbose=True):
sql = "SELECT * FROM mytable"
recs = c.execute(sql)
if verbose:
for row in recs:
print row
db_path = r'C:\Users\Prosserc\Documents\Geocoding\test.db'
conn = sqlite3.connect(db_path)
c = conn.cursor()
create()
insert()
conn.commit() #commit needed
select()
c.close()
Output:
(1, 99, 123)
After closing the program if I log onto the SQLite database the data is still there.

import sqlite3;
import pandas as pd;
con=None
def getConnection():
databaseFile="./test.db"
global con
if con == None:
con=sqlite3.connect(databaseFile)
return con
def createTable(con):
try:
c = con.cursor()
c.execute("""CREATE TABLE IF NOT EXISTS Movie
(start, end, score)""")
except Exception as e:
pass
def insert(con):
c = con.cursor()
c.execute("""INSERT INTO Movie (start, end, score)
values(1, 99, 123)""")
def queryExec():
con=getConnection()
createTable(con)
insert(con)
# r = con.execute("""SELECT * FROM Movie""")
result=pd.read_sql_query("select * from Movie;",con)
return result
r = queryExec()
print(r)

Getting the id of the last record inserted for Postgresql SERIAL KEY with Python

I am using SQLAlchemy without the ORM, i.e. using hand-crafted SQL statements to directly interact with the backend database. I am using PG as my backend database (psycopg2 as DB driver) in this instance - I don't know if that affects the answer.
I have statements like this,for brevity, assume that conn is a valid connection to the database:
conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
Assume also that the user table consists of the columns (id [SERIAL PRIMARY KEY], name, country_id)
How may I obtain the id of the new user, ideally, without hitting the database again?

You might be able to use the RETURNING clause of the INSERT statement like this:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING *")
If you only want the resulting id:
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)
RETURNING id")
[new_id] = result.fetchone()

User lastrowid
result = conn.execute("INSERT INTO user (name, country_id) VALUES ('Homer', 123)")
result.lastrowid

Current SQLAlchemy documentation suggests
result.inserted_primary_key should work!

Python + SQLAlchemy
after commit, you get the primary_key column id (autoincremeted) updated in your object.
db.session.add(new_usr)
db.session.commit() #will insert the new_usr data into database AND retrieve id
idd = new_usr.usrID # usrID is the autoincremented primary_key column.
return jsonify(idd),201 #usrID = 12, correct id from table User in Database.

this question has been asked many times on stackoverflow and no answer I have seen is comprehensive. Googling 'sqlalchemy insert get id of new row' brings up a lot of them.
There are three levels to SQLAlchemy.
Top: the ORM.
Middle: Database abstraction (DBA) with Table classes etc.
Bottom: SQL using the text function.
To an OO programmer the ORM level looks natural, but to a database programmer it looks ugly and the ORM gets in the way. The DBA layer is an OK compromise. The SQL layer looks natural to database programmers and would look alien to an OO-only programmer.
Each level has it own syntax, similar but different enough to be frustrating. On top of this there is almost too much documentation online, very hard to find the answer.
I will describe how to get the inserted id AT THE SQL LAYER for the RDBMS I use.
Table: User(user_id integer primary autoincrement key, user_name string)
conn: Is a Connection obtained within SQLAlchemy to the DBMS you are using.
SQLite
======
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
# Execute within a transaction (optional)
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.lastrowid
txn.commit()
MS SQL Server
=============
insstmt = text(
'''INSERT INTO user (user_name)
OUTPUT inserted.record_id
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()
MariaDB/MySQL
=============
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm) ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = conn.execute(text('SELECT LAST_INSERT_ID()')).fetchone()[0]
txn.commit()
Postgres
========
insstmt = text(
'''INSERT INTO user (user_name)
VALUES (:usernm)
RETURNING user_id ''' )
txn = conn.begin()
result = conn.execute(insstmt, usernm='Jane Doe')
# The id!
recid = result.fetchone()[0]
txn.commit()

result.inserted_primary_key
Worked for me. The only thing to note is that this returns a list that contains that last_insert_id.

Make sure you use fetchrow/fetch to receive the returning object
insert_stmt = user.insert().values(name="homer", country_id="123").returning(user.c.id)
row_id = await conn.fetchrow(insert_stmt)

For Postgress inserts from python code is simple to use "RETURNING" keyword with the "col_id" (name of the column which you want to get the last inserted row id) in insert statement at end
syntax -
from sqlalchemy import create_engine
conn_string = "postgresql://USERNAME:PSWD#HOSTNAME/DATABASE_NAME"
db = create_engine(conn_string)
conn = db.connect()
INSERT INTO emp_table (col_id, Name ,Age)
VALUES(3,'xyz',30) RETURNING col_id;
or
(if col_id column is auto increment)
insert_sql = (INSERT INTO emp_table (Name ,Age)
VALUES('xyz',30) RETURNING col_id;)
result = conn.execute(insert_sql)
[last_row_id] = result.fetchone()
print(last_row_id)
#output = 3
ex -