We Keep Coding

Multithreading with SQLAlchemy ORM and in-memory SQLite

I am trying to write to an in-memory SQLite database in multiple threads. According to SQLAlchemy docs this is possible. However, in practice I cannot reliably get it to work. I have tried with both scoped_session and without, yielding the same results. Strangely enough, if I pass echo=True to create_engine the code works every time, maybe because it forces the engine to synchronously print all statements to the screen. Code sample and exceptions are below.
EDIT: The more I look into this, the more I realize that SQLite is just not very good at concurrency for write operations. The section on High Concurrency says that while an unlimited amount of readers are allowed, there can only be one writer at any given time. A better approach here would be to queue all write operations instead of trying to achieve any level of concurrency. However I am still open to suggestions if anyone has managed to find a solution.
import threading
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy.pool import StaticPool
from sqlalchemy.orm import sessionmaker, scoped_session
in_memory_engine = create_engine('sqlite://', connect_args={'check_same_thread':False}, poolclass=StaticPool)
Base = declarative_base()
class Item(Base):
__tablename__ = 'items'
id = Column(Integer, primary_key=True, autoincrement=True)
data = Column(Integer)
Base.metadata.create_all(bind=in_memory_engine)
session_factory = sessionmaker(bind=in_memory_engine)
Session = scoped_session(session_factory)
def insert_items():
session = Session()
for i in range(1000):
session.add(Item(data=i))
session.commit()
Session.remove()
for i in range(10):
t = threading.Thread(target=insert_items)
t.start()
Exception in thread Thread-6:
Traceback (most recent call last):
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: cannot start a transaction within a transaction
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "c:\python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-1-e1040f59c94e>", line 26, in insert_items
session.commit()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 1027, in commit
self.transaction.commit()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 494, in commit
self._prepare_impl()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 473, in _prepare_impl
self.session.flush()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2459, in flush
self._flush(objects)
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2597, in _flush
transaction.rollback(_capture_exception=True)
File "c:\python\lib\site-packages\sqlalchemy\util\langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2557, in _flush
flush_context.execute()
File "c:\python\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 422, in execute
rec.execute(self)
File "c:\python\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 589, in execute
uow,
File "c:\python\lib\site-packages\sqlalchemy\orm\persistence.py", line 245, in save_obj
insert,
File "c:\python\lib\site-packages\sqlalchemy\orm\persistence.py", line 1138, in _emit_insert_statements
statement, params
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 988, in execute
return meth(self, multiparams, params)
File "c:\python\lib\site-packages\sqlalchemy\sql\elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1107, in _execute_clauseelement
distilled_params,
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1253, in _execute_context
e, statement, parameters, cursor, context
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1473, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) cannot start a transaction within a transaction
[SQL: INSERT INTO items (data) VALUES (?)]
[parameters: (780,)]
(Background on this error at: http://sqlalche.me/e/e3q8)
Exception in thread Thread-7:
Traceback (most recent call last):
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.

"psycopg2.errors.UndefinedTable: relation "flights" does not exist"

import os
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
engine = create_engine("postgres://kali:kali#localhost/mydb")
db = scoped_session(sessionmaker(bind=engine))
def main():
flights = db.execute ("SELECT origin, destination, duration FROM flights;").fetchall()
for flight in flights:
print(f"{flight.origin} to {flight.destination}, {flight.duration} minutes.")
if __name__ == "__main__":
main()
After i execute python , he send for me:
I do not know the reason
Error:
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "flights" does not exist
LINE 1: SELECT origin, destination, duration FROM flights
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "list.py", line 18, in <module>
main()
File "list.py", line 13, in main
flights = db.execute("SELECT origin, destination, duration FROM flights").fetchall()
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/session.py", line 1278, in execute
clause, params or {}
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1103, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1288, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1482, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "flights" does not exist
LINE 1: SELECT origin, destination, duration FROM flights
^
[SQL: SELECT origin, destination, duration FROM flights]
(Background on this error at: http://sqlalche.me/e/f405)
root#ubuntu:~/lecture3# python3 list2.py
Traceback (most recent call last):
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedTable: relation "flights" does not exist
LINE 1: SELECT origin, destination, duration FROM flights;
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "list2.py", line 15, in <module>
main()
File "list2.py", line 10, in main
flights = db.execute ("SELECT origin, destination, duration FROM flights;").fetchall()
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/orm/session.py", line 1278, in execute
clause, params or {}
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 984, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/sql/elements.py", line 293, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1103, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1288, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1482, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.7/dist-packages/sqlalchemy/engine/default.py", line 588, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "flights" does not exist
LINE 1: SELECT origin, destination, duration FROM flights;
^
[SQL: SELECT origin, destination, duration FROM flights;]
(Background on this error at: http://sqlalche.me/e/f405)
I try change "engine = create_engine("postgres://kali:kali#localhost/mydb")" to "engine = create_engine(os.getenv("postgres://kali:kali#localhost:5432/mydb"))" and:
" AttributeError: 'NoneType' object has no attribute '_instantiate_plugins' "

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "flights" does not exist
Means that you don't have a table by the name of(flights) in your database.
Create the table like this:
CREATE TABLE flights(id SERIAL PRIMARY KEY NOT NULL, origin VARCHAR NOT NULL, destination VARCHAR NOT NULL, duration INTEGER NOT NULL);
Then try running your application, it should work.

sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "users" does not exist

Im currently taking the cs50web course and i have to connect my flask app to heroku postgres database, i have already created the tables using pgadmin 4 but im not able to access the data.
import os
from flask import Flask, session
from flask_session import Session
from sqlalchemy import create_engine
from sqlalchemy.orm import scoped_session, sessionmaker
from dotenv import load_dotenv
load_dotenv()
# Check for environment variable
if not os.getenv("DATABASE_URL"):
raise RuntimeError("DATABASE_URL is not set")
# Set up database
engine = create_engine(os.getenv("DATABASE_URL"))
db = scoped_session(sessionmaker(bind=engine))
rows = db.execute("SELECT * FROM users;")
print(rows)
This is the error message i got back
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/Bryan/Downloads/project1/tester.py", line 21, in <module>
rows = db.execute("SELECT * FROM users;")
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/orm/session.py", line 1268, in execute
clause, params or {}
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "/Users/Bryan/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedTable) relation "users" does not exist
LINE 1: SELECT * FROM users;
^
[SQL: SELECT * FROM users;]
(Background on this error at: http://sqlalche.me/e/f405)
[Finished in 6.69s]
pgadmin4 screenshot

It is just saying that there is no such thing as a "users" table.

In SQL, the table name should be in double quotes, since Python also passes the string to execute function. You need to send users inside double quotes. Hence, try this :
rows = db.execute("SELECT * FROM "Users"")

Sqlalchemy: OperationalError with pycharm but not at command line

I'm working with scrapy and have had a functioning pipeline which uses sql-alchemy to insert records into a sqllite db. This was working but then I added 2 new fields: name and address. Now I'm getting:
2016-06-14 10:12:23 [scrapy] ERROR: Error processing {'account': u'X6',
'address': u' the address',
'name': u'my name'}
Traceback (most recent call last):
File "C:\envs\virtalenvs\teat\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "C:\envs\r2\tutorial\tutorial\pipelines.py", line 54, in process_item
self.connection.execute(ins_query)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 914, in execute
return meth(self, multiparams, params)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\sql\elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\util\compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) table accounts has no column named name [SQL: u'INSERT INTO accounts (account, name, address) VALUES (?, ?, ?)'] [parameters: (u'X6', u' my name', u' the address')]
Here's the pipeline code:
class MYPipeline(object):
def __init__(self):
_engine = create_engine("sqlite:///data.db")
_connection = _engine.connect()
_metadata = MetaData()
_stack_items = Table("accounts", _metadata,
Column("id", Integer, primary_key=True),
Column("account", Text),
Column("name", Text),
Column("address", Text))
_metadata.create_all(_engine)
self.connection = _connection
self.stack_items = _stack_items
def process_item(self, item, spider):
is_valid = True
for data in item:
if not data:
is_valid = False
raise DropItem("Missing %s!" % data)
if is_valid:
ins_query = self.stack_items.insert().values(
account=item["account"],
name=item["name"],
address=item["address"])
self.connection.execute(ins_query)
return item
Whats weird is that if I run this using :
scrapy crawl myspider
using GIT-Bash (I'm using win7) this works normally. However with pycharm I get the errors above.
for pycharm the run config is in the screenshot, and man.py:
from scrapy import cmdline
cmdline.execute("scrapy crawl myspider".split())
What am I doing wrong?

I suspect you're running with two different working directories. Try configuring PyCharm with the working directory you're using on the command line. Alternatively, specify an absolute path for the SQLAlchemy URL.

sqlalchemy with postgres: insert into a table whose columns have parentheses

Suppose you have a table "foo" in postgres with column name "col (parens) name". The psql command
INSERT INTO "foo" ("col (parens) name") VALUES ('bar');
works just fine. However, if I try to do the same using sqlalchemy (version 0.9.7), the resulting python code fails:
conn = sqlalchemy.create_engine('postgresql://name:password#host:port/database')
meta = sqlalchemy.schema.MetaData()
meta.reflect(bind=conn)
foo = meta.tables['foo']
vals = [{'col (parens) name': 'hi'}, {'col (parens) name': 'bye'}]
conn.execute(foo.insert(values=vals))
This does not work, giving the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "sqlalchemy/engine/base.py", line 957, in _execute_context
context)
File "sqlalchemy/engine/base.py", line 1162, in _handle_dbapi_exception
util.reraise(*exc_info)
File "sqlalchemy/engine/base.py", line 950, in _execute_context
context)
File "sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
KeyError: 'col (parens'
Apparently the sqlalchemy method to bind db parameters is running into trouble with python string interpolation. Any suggestions for a workaround?

Add paramstyle="format" to the create_engine call. It will change the way the query values are inserted to the query in a way that it won't crash on closing brackets.
conn = sqlalchemy.create_engine(
'postgresql://name:password#host:port/database',
paramstyle="format"
)