I have followed the CKAN install guide from source (http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html) and managed to get all the way until point 6. Create database tables.
At that stage, running paster db init -c /etc/ckan/default/development.ini gives the following stack trace:
Traceback (most recent call last):
File "/usr/lib/ckan/default/bin/paster", line 9, in <module>
load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 104, in run
invoke(command, command_name, options, args[1:])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 143, in invoke
exit_code = runner.run(args)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/paste/script/command.py", line 238, in run
result = self.command()
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 208, in command
self._load_config(cmd!='upgrade')
File "/usr/lib/ckan/default/src/ckan/ckan/lib/cli.py", line 164, in _load_config
self.site_user = logic.get_action('get_site_user')({'ignore_auth': True}, {})
File "/usr/lib/ckan/default/src/ckan/ckan/logic/__init__.py", line 424, in wrapped
result = _action(context, data_dict, **kw)
File "/usr/lib/ckan/default/src/ckan/ckan/logic/action/get.py", line 2209, in get_site_user
user = model.User.get(site_id)
File "/usr/lib/ckan/default/src/ckan/ckan/model/user.py", line 64, in get
return query.first()
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2334, in first
ret = list(self[0:1])
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2201, in __getitem__
return list(res)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2405, in __iter__
return self._execute_and_instances(context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2420, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 727, in execute
return meth(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 824, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 954, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1116, in _handle_dbapi_exception
exc_info
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 189, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 947, in _execute_context
context)
File "/usr/lib/ckan/default/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 435, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
^
'SELECT "user".password AS user_password, "user".id AS user_id, "user".name AS user_name, "user".openid AS user_openid, "user".fullname AS user_fullname, "user".email AS user_email, "user".apikey AS user_apikey, "user".created AS user_created, "user".reset_key AS user_reset_key, "user".about AS user_about, "user".activity_streams_email_notifications AS user_activity_streams_email_notifications, "user".sysadmin AS user_sysadmin, "user".state AS user_state \nFROM "user" \nWHERE "user".name = %(name_1)s OR "user".openid = %(openid_1)s OR "user".id = %(id_1)s ORDER BY "user".name \n LIMIT %(param_1)s' {'param_1': 1, 'id_1': 'default', 'name_1': 'default', 'openid_1': 'default'}
I did not make any modifications to the installation or the schema. The installation is under a proxy, but that did not turn out to be a problem thus far, since there were many posts giving help in that regard.
Did anybody come across this error before and managed to resolve it? I don't suppose it is a proxy-related problem... though it seems to be related to the schema.
I ran into the same issue but dropping the DB and creating and initialising it again afterwards produced the same error. Also running paster db clear -c /etc/ckan/default/development.ini failed with said error.
It worked for me, when I deactivated all ckan.plugins in development.ini before running paster db init -c /etc/ckan/default/development.ini.
Managed to solve this one after all, so I'll answer my own question... :)
The problem was with the setup of the database. Once I purged the database
and completely reinstalled it, everything worked like a charm!
sqlalchemy.exc.ProgrammingError: (ProgrammingError) column user.password does not exist
LINE 1: SELECT "user".password AS user_password, "user".id AS user_i...
This error usually means that the CKAN database exists but is empty - the CKAN tables have not been created in it.
To solve it:
cd /usr/lib/ckan/default/src/ckan
paster db init -c /etc/ckan/default/development.ini
There's more about this in the docs:
http://docs.ckan.org/en/latest/maintaining/installing/install-from-source.html#create-database-tables
Related
Ubuntu 20.04.3 LTS (WSL2)
fiona 1.8.21
geopandas 0.10.2
geopandas-base 0.10.2
geopy 2.2.0
numpy 1.22.3
pandas 1.4.1
psycopg2 2.9.3
pyproj 3.3.0
rtree 0.9.7
shapely 1.8.0
postgresql-14 14.2-1.pgdg20.04+1
postgresql-14-postgis-3 3.2.1+dfsg-1.pgdg20.04+1
trying to send a couple of polygons to postgresql database (postgis). Having no luck -.-' Always getting the same error message. Can someone help me?
Exception in thread Thread-2 (_create_searchwindows):
Traceback (most recent call last):
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
self.dialect.do_execute(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
psycopg2.errors.UndefinedObject: type "geometry" does not exist
LINE 4: geometry geometry(POLYGON,25832)
^
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/user/miniconda3/envs/geo/lib/python3.10/threading.py", line 1009, in _bootstrap_inner
self.run()
File "/home/user/miniconda3/envs/geo/lib/python3.10/threading.py", line 946, in run
self._target(*self._args, **self._kwargs)
File "/home/user/github/create_searchwindows/searchwindow_creator.py", line 143, in _create_searchwindows
grid.to_postgis(self.args.searchwindows_tablename,self.engine,if_exists="append",index=True,index_label="Index")
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/geopandas/geodataframe.py", line 1808, in to_postgis
geopandas.io.sql._write_postgis(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/geopandas/io/sql.py", line 431, in _write_postgis
gdf.to_sql(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/core/generic.py", line 2963, in to_sql
return sql.to_sql(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/io/sql.py", line 697, in to_sql
return pandas_sql.to_sql(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/io/sql.py", line 1729, in to_sql
table = self.prep_table(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/io/sql.py", line 1628, in prep_table
table.create()
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/io/sql.py", line 842, in create
self._execute_create()
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/pandas/io/sql.py", line 828, in _execute_create
self.table.create(bind=self.pd_sql.connectable)
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/sql/schema.py", line 950, in create
bind._run_ddl_visitor(ddl.SchemaGenerator, self, checkfirst=checkfirst)
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2119, in _run_ddl_visitor
visitorcallable(self.dialect, self, **kwargs).traverse_single(element)
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/sql/visitors.py", line 524, in traverse_single
return meth(obj, **kw)
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 893, in visit_table
self.connection.execute(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1295, in execute
return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/sql/ddl.py", line 80, in _execute_on_connection
return connection._execute_ddl(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1387, in _execute_ddl
ret = self._execute_context(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1851, in _execute_context
self._handle_dbapi_exception(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 2032, in _handle_dbapi_exception
util.raise_(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
raise exception
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/base.py", line 1808, in _execute_context
self.dialect.do_execute(
File "/home/user/miniconda3/envs/geo/lib/python3.10/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.ProgrammingError: (psycopg2.errors.UndefinedObject) type "geometry" does not exist
LINE 4: geometry geometry(POLYGON,25832)
^
[SQL:
CREATE TABLE public.sw32 (
"Index" BIGINT,
geometry geometry(POLYGON,25832)
)
]
(Background on this error at: https://sqlalche.me/e/14/f405)
I made sure, that the postgis extension is activated in the database. If I create the table with a geometry column in the database using psql its working perfectly fine. Something seems to be not correct with the python packages installed. Did someone came up with the same trouble?
code looks like this:
import geopandas as gpd
from sqlalchemy import create_engine
engine = create_engine(f'postgresql://{db_user}:{db_password}#{db_host}:{db_port}/')
grid = gpd.GeoDataFrame(polygons,geometry="geometry",crs="EPSG:25832")
grid.to_postgis("testtable",engine,if_exists="append",index=True,index_label="Index")
EDIT 1:
My current workaround is to store the polygons in a *.shp file and execute shp2psql to send the polygons to the database. The code looks as follows:
grid.to_file("tmp_name.shp")
cmd = f"shp2pgsql -s 25832 tmp_name.shp public.testtable |psql postgresql://{db_user}:{db_password}#{db_host}:{db_port}/{database}"
os.system(cmd)
Indeed my connection string was not correct. forgot to mention the database. with the adjustment of the connection string as follows:
engine = create_engine(f'postgresql://{db_user}:{db_password}#{db_host}:{db_port}/{database}')
everything works fine!
Probably it is related to your "geoalchemy2" version. Your geoalchemy2 version should be "0.9.2".
Problem applying just created migration (Added db.Model) through Flask-Migrate (SQLAlchemy) for PostgresSQL DB.
The error itself:
sqlalchemy.exc.InternalError: (psycopg2.InternalError) cannot drop table parameter_subtype because other objects depend on it
Full error stack trace is:
INFO [alembic.autogenerate.compare] Detected removed foreign key (event_id) (id) on table stage_has_event
INFO [alembic.autogenerate.compare] Detected removed column 'stage_has_event.event_id'
Generating /app/migrations/versions/e224df1a4818_.py ... done
(venv) $ ./manage db upgrade
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade cadf22871ae0 -> e224df1a4818, empty message
Traceback (most recent call last):
File "/app/venv/bin/flask", line 11, in <module>
sys.exit(main())
File "/app/venv/lib/python2.7/site-packages/flask_cli/cli.py", line 502, in main
cli.main(args=args, prog_name=name)
File "/app/venv/lib/python2.7/site-packages/flask_cli/cli.py", line 369, in main
return AppGroup.main(self, *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/flask/cli.py", line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/flask_migrate/cli.py", line 134, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/app/venv/lib/python2.7/site-packages/flask_migrate/__init__.py", line 259, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/app/venv/lib/python2.7/site-packages/alembic/command.py", line 254, in upgrade
script.run_env()
File "/app/venv/lib/python2.7/site-packages/alembic/script/base.py", line 427, in run_env
util.load_python_file(self.dir, 'env.py')
File "/app/venv/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
module = load_module_py(module_id, path)
File "/app/venv/lib/python2.7/site-packages/alembic/util/compat.py", line 141, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 87, in <module>
run_migrations_online()
File "migrations/env.py", line 80, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/app/venv/lib/python2.7/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
self.get_context().run_migrations(**kw)
File "/app/venv/lib/python2.7/site-packages/alembic/runtime/migration.py", line 330, in run_migrations
step.migration_fn(**kw)
File "/app/migrations/versions/e224df1a4818_.py", line 21, in upgrade
op.drop_table('parameter_subtype')
File "<string>", line 8, in drop_table
File "<string>", line 3, in drop_table
File "/app/venv/lib/python2.7/site-packages/alembic/operations/ops.py", line 1187, in drop_table
operations.invoke(op)
File "/app/venv/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
return fn(self, operation)
File "/app/venv/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 70, in drop_table
operation.to_table(operations.migration_context)
File "/app/venv/lib/python2.7/site-packages/alembic/ddl/impl.py", line 203, in drop_table
self._exec(schema.DropTable(table))
File "/app/venv/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
return conn.execute(construct, *multiparams, **params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/app`enter code here`/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.InternalError) cannot drop table parameter_subtype because other objects depend on it
DETAIL: constraint parameter_parameter_subtype_id_fkey on table parameter depends on table parameter_subtype
HINT: Use DROP ... CASCADE to drop the dependent objects too.
[SQL: '\nDROP TABLE parameter_subtype']
This leads to table not created in PostgreSQL db and columns either.
Unsure what may have caused this problem and how to understand this Error message.
I did manually drop the tables and it helped. However how to be with migration.
Any Suggestions?
Flask-Migrate does not read in the database to see the dependencies between the objects. You can achieve a successful migration by reordering the drop_table in your migration file.
What you can do is modify the migration file like so:
op.drop_table("parameter_subtype")
op.drop_table("parameter")
And then run the upgrade command. Normally ordering the drop in this order should solve the problem.
I got the same error and what worked for me was to first drop the table in the database using the command:
$ DROP TABLE table_name;
And then in the migration file in the upgrade() function delete or comment out the line where it says:
op.drop_table('table_name')
There are a couple of related questions regarding this, but in my case, all those solutions is not working out. Thats why I thought of asking again. I am getting this error while I am firing below query using sqlalchemy orm.
Traceback (most recent call last):
File "MyFile.py", line 1010, in <module>
handler.handle(line.split('\t'))
File "MyFile.py", line 849, in handle
self.getRecord(whatIfFlag, id)
File "MyFile.py", line 143, in getRecord
newRecord = self.recordSearcher.getRecordByParams(name, pId)
File "abc.py", line 67, in getRecord
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2361, in one
ret = list(self)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2404, in __iter__
return self._execute_and_instances(context)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2419, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 720, in execute
return meth(self, multiparams, params)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/sql/elements.py", line 317, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 817, in _execute_clauseelement
compiled_sql, distilled_params
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 947, in _execute_context
context)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1108, in _handle_dbapi_exception
exc_info
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/util/compat.py", line 185, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 940, in _execute_context
context)
File "/opt/product/python/prod/bin/2.7.6/lib/python2.7/site-packages/SQLAlchemy-0.9.4-py2.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 435, in do_execute
cursor.execute(statement, parameters)
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 205, in execute
File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
sqlalchemy.exc.OperationalError: (OperationalError) (2013, 'Lost connection to MySQL server during query') ....
query = self.__session.query(MyTable).filter(and_(MyTable.NAME == name,MyTable.P_ID == p_id)
try:
record = query.one()
except NoResultFound:
new_record = MyTable(params)
self.__session.add(new_record)
self.__session.commit()
self.__session.close()
It is expected to return only one record. This is how I create my session.
sqlEngine = sqlalchemy.create_engine(self.getMySQLURI(), pool_recycle=10800, echo=False, echo_pool=False)
session = scoped_session(sessionmaker(autoflush=True,
autocommit=False,
bind=sqlEngine,
expire_on_commit=False))
These are my mysql configurations: interactive_timeout and wait_timeout is set to 28800 ~ 8 hours. net_write_timeout is set to 3600 ~ 60 mins and net_read_timeout is set 300 ~ 5 mins.
Any help is highly appreciated.
It is turned out to be problem with the tcp_connect_timeout between the application server and the database server. The tcp connect timeout was default of 1 hour and my pool recycle settings was 3 hrs. So anything between 1 and 3 were failing. Posting the answer to help others who might face this later.
I'm trying to use LOAD LOCAL DATA INFILE with sqlalchemy to upload batch files to my server. I have edited my /etc/mysql/my.conf to contain local-infile == 1 under both [mysql] and [mysqld], yet I'm getting this traceback when I run the program.
The relevant line of code is:
Traceback (most recent call last):
File "main.py", line 48, in <module>
con.execute(sql)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 906, in execute
return self._execute_text(object, multiparams, params)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1054, in _execute_text
statement, parameters
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1146, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1332, in _handle_dbapi_exception
exc_info
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context
context)
File "/Users/eoddata/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 442, in do_execute
cursor.execute(statement, parameters)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
self.errorhandler(self, exc, value)
File "/Users/eoddata/venv/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
sqlalchemy.exc.OperationalError: (_mysql_exceptions.OperationalError) (1148, 'The used command is not allowed with
this MySQL version') [SQL: "LOAD DATA LOCAL
INFILE 'stocks/AMEX_20150420.txt' INTO TABLE database.new_table\n
FIELDS TERMINATED BY ',' (Symbol,#Date,Open,High,Low,Close,Volume)\n
SET Date = STR_TO_DATE(#Date,'%%Y%%m%%d');"]
I've done some research from the docs and can't find a way to fix this error. With python MySQLdb (from pip install MySQL-Python) will allow me to pass in a local-infile paramter. I can't find an equivalent in SQLAlchemy. Any help is appreciated.
EDIT: I've worked on this all night and still haven't made any progress - a lot of dead ends. Any ideas?
Thanks, Jared
You should add local_infile=1 to the connection string.
SQLALCHEMY_DATABASE_URI = "mysql://user:pass#localhost/dbname?charset=utf8&local_infile=1"
Suppose you have a table "foo" in postgres with column name "col (parens) name". The psql command
INSERT INTO "foo" ("col (parens) name") VALUES ('bar');
works just fine. However, if I try to do the same using sqlalchemy (version 0.9.7), the resulting python code fails:
conn = sqlalchemy.create_engine('postgresql://name:password#host:port/database')
meta = sqlalchemy.schema.MetaData()
meta.reflect(bind=conn)
foo = meta.tables['foo']
vals = [{'col (parens) name': 'hi'}, {'col (parens) name': 'bye'}]
conn.execute(foo.insert(values=vals))
This does not work, giving the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "sqlalchemy/sql/elements.py", line 321, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "sqlalchemy/engine/base.py", line 957, in _execute_context
context)
File "sqlalchemy/engine/base.py", line 1162, in _handle_dbapi_exception
util.reraise(*exc_info)
File "sqlalchemy/engine/base.py", line 950, in _execute_context
context)
File "sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
KeyError: 'col (parens'
Apparently the sqlalchemy method to bind db parameters is running into trouble with python string interpolation. Any suggestions for a workaround?
Add paramstyle="format" to the create_engine call. It will change the way the query values are inserted to the query in a way that it won't crash on closing brackets.
conn = sqlalchemy.create_engine(
'postgresql://name:password#host:port/database',
paramstyle="format"
)