I'm working with scrapy and have had a functioning pipeline which uses sql-alchemy to insert records into a sqllite db. This was working but then I added 2 new fields: name and address. Now I'm getting:
2016-06-14 10:12:23 [scrapy] ERROR: Error processing {'account': u'X6',
'address': u' the address',
'name': u'my name'}
Traceback (most recent call last):
File "C:\envs\virtalenvs\teat\lib\site-packages\twisted\internet\defer.py", line 588, in _runCallbacks
current.result = callback(current.result, *args, **kw)
File "C:\envs\r2\tutorial\tutorial\pipelines.py", line 54, in process_item
self.connection.execute(ins_query)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 914, in execute
return meth(self, multiparams, params)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\sql\elements.py", line 323, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1010, in _execute_clauseelement
compiled_sql, distilled_params
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1146, in _execute_context
context)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1341, in _handle_dbapi_exception
exc_info
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\util\compat.py", line 202, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\base.py", line 1139, in _execute_context
context)
File "C:\envs\virtalenvs\teat\lib\site-packages\sqlalchemy\engine\default.py", line 450, in do_execute
cursor.execute(statement, parameters)
OperationalError: (sqlite3.OperationalError) table accounts has no column named name [SQL: u'INSERT INTO accounts (account, name, address) VALUES (?, ?, ?)'] [parameters: (u'X6', u' my name', u' the address')]
Here's the pipeline code:
class MYPipeline(object):
def __init__(self):
_engine = create_engine("sqlite:///data.db")
_connection = _engine.connect()
_metadata = MetaData()
_stack_items = Table("accounts", _metadata,
Column("id", Integer, primary_key=True),
Column("account", Text),
Column("name", Text),
Column("address", Text))
_metadata.create_all(_engine)
self.connection = _connection
self.stack_items = _stack_items
def process_item(self, item, spider):
is_valid = True
for data in item:
if not data:
is_valid = False
raise DropItem("Missing %s!" % data)
if is_valid:
ins_query = self.stack_items.insert().values(
account=item["account"],
name=item["name"],
address=item["address"])
self.connection.execute(ins_query)
return item
Whats weird is that if I run this using :
scrapy crawl myspider
using GIT-Bash (I'm using win7) this works normally. However with pycharm I get the errors above.
for pycharm the run config is in the screenshot, and man.py:
from scrapy import cmdline
cmdline.execute("scrapy crawl myspider".split())
What am I doing wrong?
I suspect you're running with two different working directories. Try configuring PyCharm with the working directory you're using on the command line. Alternatively, specify an absolute path for the SQLAlchemy URL.
Related
I am learning about Rest APIs and Flask. Following a simple tutorial, something doesn't work and I can't get it fixed. I create a database and I want to add something to it. My file is called "API.py" and I am using PyCharm. After running the file, I type into the Python Console in Python:
from API import db
from API import Drink
Drink.query.all() (returns an empty list, all good)
drink = Drink(name="x", description="y")
Drink.query.all() (returns an empty list, all good)
db.session.add(drink)
Drink.query.all() (returns the drink, all good)
db.session.commit()
Drink.query.all() - ERROR
I tried several things, but I can't figure it out. In the tutorial, everything is working just fine. Would you please help me fix this? Below you will find the error message and the code:
Error message:
Traceback (most recent call last):
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 1800, in _execute_context
context = constructor(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1015, in _init_compiled
self.cursor = self.create_cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1386, in create_cursor
return self.create_default_cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1389, in create_default_cursor
return self._dbapi_connection.cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\pool\base.py", line 1083, in cursor
return self.dbapi_connection.cursor(*args, **kwargs)
sqlite3.ProgrammingError: SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 5244 and this is thread id 9008.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\IPython\core\interactiveshell.py", line 3397, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-7-87e83f2944ed>", line 1, in <cell line: 1>
Drink.query.all()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\orm\query.py", line 2772, in all
return self._iter().all()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\orm\query.py", line 2907, in _iter
result = self.session.execute(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\orm\session.py", line 1712, in execute
result = conn._execute_20(statement, params or {}, execution_options)
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 1705, in _execute_20
return meth(self, args_10style, kwargs_10style, execution_options)
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\sql\elements.py", line 333, in _execute_on_connection
return connection._execute_clauseelement(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 1572, in _execute_clauseelement
ret = self._execute_context(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 1806, in _execute_context
self._handle_dbapi_exception(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 2124, in _handle_dbapi_exception
util.raise_(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\util\compat.py", line 208, in raise_
raise exception
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\base.py", line 1800, in _execute_context
context = constructor(
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1015, in _init_compiled
self.cursor = self.create_cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1386, in create_cursor
return self.create_default_cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\engine\default.py", line 1389, in create_default_cursor
return self._dbapi_connection.cursor()
File "C:\Users\larsw\anaconda3\envs\useenv\lib\site-packages\sqlalchemy\pool\base.py", line 1083, in cursor
return self.dbapi_connection.cursor(*args, **kwargs)
sqlalchemy.exc.ProgrammingError: (sqlite3.ProgrammingError) SQLite objects created in a thread can only be used in that same thread. The object was created in thread id 5244 and this is thread id 9008.
[SQL: SELECT drink.iden AS drink_iden, drink.name AS drink_name, drink.description AS drink_description
FROM drink]
[parameters: [{}]]
(Background on this error at: https://sqlalche.me/e/14/f405)
Code:
from flask import Flask
from flask_sqlalchemy import SQLAlchemy
app = Flask(__name__)
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///data.db"
db = SQLAlchemy(app)
class Drink(db.Model):
iden = db.Column(db.Integer, primary_key=True)
name = db.Column(db.String(80), unique=True, nullable=False)
description = db.Column(db.String(120))
def __repr__(self):
return f"{self.name} - {self.description}"
db.create_all()
#app.route('/')
def index():
return "Hey"
#app.route('/drinks')
def get_drinks():
return {"drinks": "drink data"}
if __name__ == "__main__":
app.run(debug=True, port=8000)
Thanks to snakecharmerb's comment, I replaced
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///data.db"
by
app.config["SQLALCHEMY_DATABASE_URI"] = "sqlite:///data.db?check_same_thread=False"
and it works just fine now.
I am trying to write to an in-memory SQLite database in multiple threads. According to SQLAlchemy docs this is possible. However, in practice I cannot reliably get it to work. I have tried with both scoped_session and without, yielding the same results. Strangely enough, if I pass echo=True to create_engine the code works every time, maybe because it forces the engine to synchronously print all statements to the screen. Code sample and exceptions are below.
EDIT: The more I look into this, the more I realize that SQLite is just not very good at concurrency for write operations. The section on High Concurrency says that while an unlimited amount of readers are allowed, there can only be one writer at any given time. A better approach here would be to queue all write operations instead of trying to achieve any level of concurrency. However I am still open to suggestions if anyone has managed to find a solution.
import threading
from sqlalchemy import *
from sqlalchemy.ext.declarative import declarative_base
from sqlalchemy import create_engine
from sqlalchemy.pool import StaticPool
from sqlalchemy.orm import sessionmaker, scoped_session
in_memory_engine = create_engine('sqlite://', connect_args={'check_same_thread':False}, poolclass=StaticPool)
Base = declarative_base()
class Item(Base):
__tablename__ = 'items'
id = Column(Integer, primary_key=True, autoincrement=True)
data = Column(Integer)
Base.metadata.create_all(bind=in_memory_engine)
session_factory = sessionmaker(bind=in_memory_engine)
Session = scoped_session(session_factory)
def insert_items():
session = Session()
for i in range(1000):
session.add(Item(data=i))
session.commit()
Session.remove()
for i in range(10):
t = threading.Thread(target=insert_items)
t.start()
Exception in thread Thread-6:
Traceback (most recent call last):
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: cannot start a transaction within a transaction
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "c:\python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "c:\python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-1-e1040f59c94e>", line 26, in insert_items
session.commit()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 1027, in commit
self.transaction.commit()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 494, in commit
self._prepare_impl()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 473, in _prepare_impl
self.session.flush()
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2459, in flush
self._flush(objects)
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2597, in _flush
transaction.rollback(_capture_exception=True)
File "c:\python\lib\site-packages\sqlalchemy\util\langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "c:\python\lib\site-packages\sqlalchemy\orm\session.py", line 2557, in _flush
flush_context.execute()
File "c:\python\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 422, in execute
rec.execute(self)
File "c:\python\lib\site-packages\sqlalchemy\orm\unitofwork.py", line 589, in execute
uow,
File "c:\python\lib\site-packages\sqlalchemy\orm\persistence.py", line 245, in save_obj
insert,
File "c:\python\lib\site-packages\sqlalchemy\orm\persistence.py", line 1138, in _emit_insert_statements
statement, params
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 988, in execute
return meth(self, multiparams, params)
File "c:\python\lib\site-packages\sqlalchemy\sql\elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1107, in _execute_clauseelement
distilled_params,
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1253, in _execute_context
e, statement, parameters, cursor, context
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1473, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "c:\python\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) cannot start a transaction within a transaction
[SQL: INSERT INTO items (data) VALUES (?)]
[parameters: (780,)]
(Background on this error at: http://sqlalche.me/e/e3q8)
Exception in thread Thread-7:
Traceback (most recent call last):
File "c:\python\lib\site-packages\sqlalchemy\engine\base.py", line 1249, in _execute_context
cursor, statement, parameters, context
File "c:\python\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
sqlite3.InterfaceError: Error binding parameter 0 - probably unsupported type.
I am writing on a Dash app and using SQL alchemy for the database. I want to make the creation of the database automated so I have a few lines of code to create the database if it does not exist.
from sqlalchemy import create_engine, MetaData, Table, Column, Integer, String, Float
from sqlalchemy_utils import create_database, database_exists
engine = create_engine(SQLALCHEMY_DATABASE_URI, echo = True)
create_database(SQLALCHEMY_DATABASE_URI)
meta = MetaData()
flagged_vendors_table = Table(
'flagged_vendors', meta,
Column(vendor_col, String, primary_key = True),
Column(flag_col, Integer)
)
meta.create_all(engine)
This seems to work well as per the echo:
2020-01-21 09:28:30,604 INFO sqlalchemy.engine.base.Engine
CREATE TABLE flagged_vendors (
"Vendor PRS_description" VARCHAR NOT NULL,
"Flag" INTEGER,
PRIMARY KEY ("Vendor PRS_description")
)
2020-01-21 09:28:30,608 INFO sqlalchemy.engine.base.Engine ()
2020-01-21 09:28:30,639 INFO sqlalchemy.engine.base.Engine COMMIT
However when I execute the app I get the following error:
Traceback (most recent call last):
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\default.py", line 581, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: no such table: flagged_vendors
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\pandas\io\sql.py", line 518, in to_sql
method=method,
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\pandas\io\sql.py", line 1319, in to_sql
table.create()
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\pandas\io\sql.py", line 647, in create
self.pd_sql.drop_table(self.name, self.schema)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\pandas\io\sql.py", line 1367, in drop_table
self.get_table(table_name, schema).drop()
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\sql\schema.py", line 884, in drop
bind._run_visitor(ddl.SchemaDropper, self, checkfirst=checkfirst)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 2049, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1618, in _run_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\sql\visitors.py", line 138, in traverse_single
return meth(obj, **kw)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 1002, in visit_table
self.connection.execute(DropTable(table))
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 982, in execute
return meth(self, multiparams, params)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\sql\ddl.py", line 72, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1044, in _execute_ddl
compiled,
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1250, in _execute_context
e, statement, parameters, cursor, context
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1476, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\util\compat.py", line 398, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\util\compat.py", line 152, in reraise
raise value.with_traceback(tb)
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\base.py", line 1246, in _execute_context
cursor, statement, parameters, context
File "C:\Users\LIAG8802\Documents\Procurement_analytics\venv\lib\site-packages\sqlalchemy\engine\default.py", line 581, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: flagged_vendors
[SQL:
DROP TABLE flagged_vendors]
(Background on this error at: http://sqlalche.me/e/e3q8)
The traceback is not leading to any line in my code, which is an issue that I am trying to investigate here, but I believe there is only one part in my code that could cause the error:
def to_storage(df, storage_info):
storage_type = storage_info["storage_type"]
if storage_type == "sql":
db_uri = storage_info["db_uri"]
engine = create_engine(db_uri, echo = False, pool_pre_ping=True)
table_name = storage_info["table_name"]
df.to_sql(table_name, engine, if_exists = "replace")
It is very strange because the code is supposed to drop the table only if it exists. One last element: this issue seems to appear only if there is no existing database at the specified URI when I launch the app.
I have followed the CKAN install guide from source (http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html) and managed to get all the way until point 6. Create database tables.
At that stage, running paster db init -c /etc/ckan/default/development.ini gives the following stack trace:
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 208, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 164, in _load_config
self.site_user = logic.get_action('get_site_user')({'ignore_auth': True}, {})
File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 424, in wrapped
result = _action(context, data_dict, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py", line 2209, in get_site_user
user = model.User.get(site_id)
File "/usr/lib/ckan/default/src/ckan/ckan/model/user.py", line 64, in get
return query.first()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2334, in first
ret = list(self[0:1])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2201, in __getitem__
return list(res)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2405, in __iter__
return self._execute_and_instances(context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2420, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 727, in execute
return meth(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 954, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception
exc_info
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 435, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
^
'SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".openid AS user_openid, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state \nFROM "user" \nWHERE "user".name = %(name_1)s OR "user".openid = %(openid_1)s OR "user".id = %(id_1)s ORDER BY "user".name \n LIMIT %(param_1)s' {'param_1': 1, 'id_1': 'default', 'name_1': 'default', 'openid_1': 'default'}
I did not make any modifications to the installation or the schema. The installation is under a proxy, but that did not turn out to be a problem thus far, since there were many posts giving help in that regard.
Did anybody come across this error before and managed to resolve it? I don't suppose it is a proxy-related problem... though it seems to be related to the schema.
I ran into the same issue but dropping the DB and creating and initialising it again afterwards produced the same error. Also running paster db clear -c /etc/ckan/default/development.ini failed with said error.
It worked for me, when I deactivated all ckan.plugins in development.ini before running paster db init -c /etc/ckan/default/development.ini.
Managed to solve this one after all, so I'll answer my own question... :)
The problem was with the setup of the database. Once I purged the database
and completely reinstalled it, everything worked like a charm!
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
This error usually means that the CKAN database exists but is empty - the CKAN tables have not been created in it.
To solve it:
cd /usr/lib/ckan/default/src/ckan
paster db init -c /etc/ckan/default/development.ini
There's more about this in the docs:
http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html#create-database-tables
Suppose you have a table "foo" in postgres with column name "col (parens) name". The psql command
INSERT INTO "foo" ("col (parens) name") VALUES ('bar');
works just fine. However, if I try to do the same using sqlalchemy (version 0.9.7), the resulting python code fails:
conn = sqlalchemy.create_engine('postgresql://name:password#host:port/database')
meta = sqlalchemy.schema.MetaData()
meta.reflect(bind=conn)
foo = meta.tables['foo']
vals = [{'col (parens) name': 'hi'}, {'col (parens) name': 'bye'}]
conn.execute(foo.insert(values=vals))
This does not work, giving the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "sqlalchemy/engine/base.py", line 957, in _execute_context
context)
File "sqlalchemy/engine/base.py", line 1162, in _handle_dbapi_exception
util.reraise(*exc_info)
File "sqlalchemy/engine/base.py", line 950, in _execute_context
context)
File "sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
KeyError: 'col (parens'
Apparently the sqlalchemy method to bind db parameters is running into trouble with python string interpolation. Any suggestions for a workaround?
Add paramstyle="format" to the create_engine call. It will change the way the query values are inserted to the query in a way that it won't crash on closing brackets.
conn = sqlalchemy.create_engine(
'postgresql://name:password#host:port/database',
paramstyle="format"
)