I am following the install instructions for the security utility opencve which imports data into a postgresql database:
https://docs.opencve.io/installation/manual/
I have followed all of the instructions including creating the 5G swap file. All tables create, no problems, however, the data import fails at this step:
opencve import-data
I receive the below errors referring to a timeout time and time again. Despite straw clutching which includes disabling the firewall, searching for any postgresql timeout parameters, creating a large swap file and attempting the install on another server, as well as verifying that psycopg2-binary is installed, I still hit the same problem in the same spot.
The full errors displayed are below but I believe the main part is:
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 725, in do_executem
cursor.executemany(statement, parameters)
psycopg2.DatabaseError: could not receive data from server: Connection timed out
If I look at that psycopg2.py file, this is the block of code with line 725 being the last line:
def do_executemany(self, cursor, statement, parameters, context=None):
if self.psycopg2_batch_mode:
extras = self._psycopg2_extras()
extras.execute_batch(cursor, statement, parameters)
else:
cursor.executemany(statement, parameters)
If this is a genuine timeout somewhere I can't see where. The connection is good as the tables get created in the earlier part of the import-data command.
This is the section that is repeatedly failing.
#####################
Importing CPE list...
#####################
[*] Downloading https://nvd.nist.gov/feeds/xml/cpe/dictionary/official-cpe-dictionary_v2.3.xml.gz... (done in 1.31s).
[*] Parsing XML elements... (done in 269.254s).
[*] Creating list of mappings... (done in 330.919s).
[*] Inserting Vendors and Products...Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1224, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 725, in do_executem
cursor.executemany(statement, parameters)
psycopg2.DatabaseError: could not receive data from server: Connection timed out
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/usr/local/bin/opencve", line 11, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/opencve/commands/__init__.py", line 39, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/flask/cli.py", line 412, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/opencve/commands/imports/__init__.py", line 42, in import_data
cpe.run(vendors)
File "/usr/local/lib/python3.6/site-packages/opencve/commands/imports/cpe.py", line 63, in run
db.session.bulk_insert_mappings(Vendor, mappings["vendors"].values())
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/scoping.py", line 162, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2790, in bulk_insert_mappings
render_nulls,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2880, in _bulk_save_mappings
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 129, in reraise
raise value
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2874, in _bulk_save_mappings
render_nulls,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 102, in _bulk_insert
bookkeeping=return_defaults,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1066, in _emit_insert_statements
c = cached_connections[connection].execute(statement, multiparams)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 988, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 287, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1107, in _execute_clauseelement
distilled_params,
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1466, in _handle_dbapi_exception
util.raise_from_cause(sqlalchemy_exception, exc_info)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 383, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 128, in reraise
raise value.with_traceback(tb)
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1224, in _execute_context
cursor, statement, parameters, context
File "/usr/local/lib/python3.6/site-packages/sqlalchemy/dialects/postgresql/psycopg2.py", line 725, in do_executem
cursor.executemany(statement, parameters)
sqlalchemy.exc.DatabaseError: (psycopg2.DatabaseError) could not receive data from server: Connection timed out
[SQL: INSERT INTO vendors (id, created_at, updated_at, name) VALUES (%(id)s, now(), now(), %(name)s)]
[parameters: ({'id': UUID('a53ecce1-d16d-4ae5-a20d-3c93034b8d32'), 'name': 'openbsd'}, {'id': UUID('98176222-30ad-498c5a-4a3981bc068e'), 'name': 'freebsd'}, {'id': UUID('f734d65f-f55d-4a3d-a099-189d120bfa99'), 'name': 'bsdi'}, {'id'UID('eadb7068-c668-4981-9190-5aba1ce58d6c'), 'name': 'redhat'}, {'id': UUID('ec37138d-9dc9-432f-a6a4-a4d2e80770b1'),ame': 'caldera'}, {'id': UUID('523ea159-56bc-48c6-914f-1ebecd0f1120'), 'name': 'tritreal'}, {'id': UUID('287ac9fb-654751-a005-625b43059719'), 'name': 'sun'}, {'id': UUID('167edc67-f559-42fb-a501-f91e4114faed'), 'name': 'hp'} ... diaying 10 of 28136 total bound parameter sets ... {'id': UUID('2121a912-76a3-453f-8963-09c537f4bf78'), 'name': 'zend_feedback_tab_project'}, {'id': UUID('5f742c4f-3e37-486d-b5aa-80a92e9f0559'), 'name': 'zyzel'})]
(Background on this error at: http://sqlalche.me/e/4xp6)
I have opened a case for comment on the opencve git page but there has been no response so, hoping that someone may have some advice here please.
Related
I would like to serialize many Gensim library Word2Vec models in MySQL database. For the purpose of my project I am utilizing: Python, Flask, SQLAlchemy, and MySQL+PyMySQL. Below you can see my Word2VecModel class which I am using in order to create word2vec table in MySQL DB:
from db.db import db
import db_models.influencer
class Word2VecModel(db.Model):
__tablename__ = "word2vec"
m_id = db.Column(db.Integer, primary_key=True)
m_username = db.Column(db.String(20), db.ForeignKey("influencer.m_username"))
m_binary = db.Column(db.LargeBinary)
influencer = db.relationship("InfluencerModel")
def __init__(self, m_binary, m_username):
self.m_binary = m_binary
self.m_username = m_username
def __repr__(self):
return '<Word2Vec {}>'.format(self.m_name)
#classmethod
def find_by_username(cls, m_username):
return cls.query.filter_by(name=m_username).first()
def save_to_db(self):
db.session.add(self)
db.session.commit()
def delete_from_db(self):
db.session.delete(self)
db.session.commit()
Whenever I try to execute:
word2vec_model = Word2VecModel(pickle.dumps(word2vec), username)
word2vec_model .save_to_db()
I get the following error message:
Traceback (most recent call last):
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 705, in _write_bytes
self._sock.sendall(data)
BrokenPipeError: [Errno 32] Broken pipe
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
cursor, statement, parameters, context
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 504, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 773, in _execute_command
self.write_packet(sql[:packet_size])
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 634, in write_packet
self._write_bytes(data)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 710, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
pymysql.err.OperationalError: (2006, "MySQL server has gone away (BrokenPipeError(32, 'Broken pipe'))")
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "app.py", line 117, in <module>
word2vec.save_to_db()
File "/home/stefan/PycharmProjects/NLP - Influencer Text Analysis/src/flask_api/db_models/word2vec.py", line 25, in save_to_db
db.session.commit()
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/scoping.py", line 163, in do
return getattr(self.registry(), name)(*args, **kwargs)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 1042, in commit
self.transaction.commit()
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 504, in commit
self._prepare_impl()
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 483, in _prepare_impl
self.session.flush()
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2523, in flush
self._flush(objects)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2664, in _flush
transaction.rollback(_capture_exception=True)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/util/langhelpers.py", line 69, in __exit__
exc_value, with_traceback=exc_tb,
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 2624, in _flush
flush_context.execute()
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 422, in execute
rec.execute(self)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/unitofwork.py", line 589, in execute
uow,
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 245, in save_obj
insert,
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/orm/persistence.py", line 1136, in _emit_insert_statements
statement, params
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1014, in execute
return meth(self, multiparams, params)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/sql/elements.py", line 298, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1133, in _execute_clauseelement
distilled_params,
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1318, in _execute_context
e, statement, parameters, cursor, context
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1512, in _handle_dbapi_exception
sqlalchemy_exception, with_traceback=exc_info[2], from_=e
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/util/compat.py", line 178, in raise_
raise exception
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/base.py", line 1278, in _execute_context
cursor, statement, parameters, context
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/sqlalchemy/engine/default.py", line 593, in do_execute
cursor.execute(statement, parameters)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/cursors.py", line 163, in execute
result = self._query(query)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/cursors.py", line 321, in _query
conn.query(q)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 504, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 773, in _execute_command
self.write_packet(sql[:packet_size])
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 634, in write_packet
self._write_bytes(data)
File "/home/stefan/.virtualenvs/dl4cv/lib/python3.6/site-packages/pymysql/connections.py", line 710, in _write_bytes
"MySQL server has gone away (%r)" % (e,))
Killed
I've read somewhere that increasing MySQL connection timeout might help, but not in my case. Here is my DB engine configuration:
app.config['SQLALCHEMY_TRACK_MODIFICATIONS'] = False
app.config['SQLALCHEMY_DATABASE_URI'] = 'mysql+pymysql://root:''#localhost/********'
app.config['PROPAGATE_EXCEPTIONS'] = True
app.config["SQLALCHEMY_ENGINE_OPTIONS"] = {"pool_recycle": 120}
I want to access an oracle database through sqlalchemy in flask development. An engine is created by sqlalchemy create_engine()
MemoryError is reported when engine.connect() is called.
If setting a wrong password in the linkword, it would report:
sqlalchemy.exc.DatabaseError: (cx_Oracle.DatabaseError) ORA-01017: invalid username/password; logon denied.
With the correct password and user name, MemoryError would appear.
It is very strange when i copy the codes into jupyter or a new project (not a flask project)in pycharm, the codes work well. I wonder if it is possible the flask has problems.
from sqlalchemy import create_engine
#test_bp.route('/testtd')
def testtd():
#the linkword has no problem because these codes works in Jupyter or a new project
linkword = 'blablabla'
engine = create_engine(linkword)
#here i got a problem. memoryError was reported.
conn = engine.connect()
ret = conn.execute('select table_name from user_tables').fetchall()
print(ret)
conn.close()
Errors reported by pycharm are as follows:
File "E:\Projects\Python\DAM2\blueprints\test.py", line 19, in testtd
conn = engine.connect()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 2196, in connect
return self._connection_cls(self, **kwargs)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 103, in __init__
else engine.raw_connection()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 2296, in raw_connection
self.pool.unique_connection, _connection
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 2265, in _wrap_pool_connect
return fn()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 303, in unique_connection
return _ConnectionFairy._checkout(self)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 760, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 492, in checkout
rec = pool._do_get()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\impl.py", line 139, in _do_get
self._dec_overflow()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\util\langhelpers.py", line 68, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\impl.py", line 136, in _do_get
return self._create_connection()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 308, in _create_connection
return _ConnectionRecord(self)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 437, in __init__
self.__connect(first_connect_check=True)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\pool\base.py", line 649, in __connect
).exec_once(self.connection, self)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\event\attr.py", line 287, in exec_once
self(*args, **kw)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\event\attr.py", line 297, in __call__
fn(*args, **kw)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1443, in go
return once_fn(*arg, **kw)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\strategies.py", line 199, in first_connect
dialect.initialize(c)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\dialects\oracle\cx_oracle.py", line 832, in initialize
super(OracleDialect_cx_oracle, self).initialize(connection)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\dialects\oracle\base.py", line 1140, in initialize
super(OracleDialect, self).initialize(connection)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\default.py", line 297, in initialize
connection
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\dialects\oracle\base.py", line 1239, in _get_default_schema_name
connection.execute("SELECT USER FROM DUAL").scalar()
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 982, in execute
return self._execute_text(object_, multiparams, params)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 1155, in _execute_text
parameters,
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 1248, in _execute_context
e, statement, parameters, cursor, context
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 1468, in _handle_dbapi_exception
util.reraise(*exc_info)
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\util\compat.py", line 153, in reraise
raise value
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\base.py", line 1244, in _execute_context
cursor, statement, parameters, context
File "C:\Users\GAC\.virtualenvs\DAM2-SwV_IMXP\lib\site-packages\sqlalchemy\engine\default.py", line 552, in do_execute
cursor.execute(statement, parameters)
MemoryError
how can i fix the codes?
Problem applying just created migration (Added db.Model) through Flask-Migrate (SQLAlchemy) for PostgresSQL DB.
The error itself:
sqlalchemy.exc.InternalError: (psycopg2.InternalError) cannot drop table parameter_subtype because other objects depend on it
Full error stack trace is:
INFO [alembic.autogenerate.compare] Detected removed foreign key (event_id) (id) on table stage_has_event
INFO [alembic.autogenerate.compare] Detected removed column 'stage_has_event.event_id'
Generating /app/migrations/versions/e224df1a4818_.py ... done
(venv) $ ./manage db upgrade
INFO [alembic.runtime.migration] Context impl PostgresqlImpl.
INFO [alembic.runtime.migration] Will assume transactional DDL.
INFO [alembic.runtime.migration] Running upgrade cadf22871ae0 -> e224df1a4818, empty message
Traceback (most recent call last):
File "/app/venv/bin/flask", line 11, in <module>
sys.exit(main())
File "/app/venv/lib/python2.7/site-packages/flask_cli/cli.py", line 502, in main
cli.main(args=args, prog_name=name)
File "/app/venv/lib/python2.7/site-packages/flask_cli/cli.py", line 369, in main
return AppGroup.main(self, *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 697, in main
rv = self.invoke(ctx)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 1066, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 895, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/decorators.py", line 17, in new_func
return f(get_current_context(), *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/flask/cli.py", line 257, in decorator
return __ctx.invoke(f, *args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/click/core.py", line 535, in invoke
return callback(*args, **kwargs)
File "/app/venv/lib/python2.7/site-packages/flask_migrate/cli.py", line 134, in upgrade
_upgrade(directory, revision, sql, tag, x_arg)
File "/app/venv/lib/python2.7/site-packages/flask_migrate/__init__.py", line 259, in upgrade
command.upgrade(config, revision, sql=sql, tag=tag)
File "/app/venv/lib/python2.7/site-packages/alembic/command.py", line 254, in upgrade
script.run_env()
File "/app/venv/lib/python2.7/site-packages/alembic/script/base.py", line 427, in run_env
util.load_python_file(self.dir, 'env.py')
File "/app/venv/lib/python2.7/site-packages/alembic/util/pyfiles.py", line 81, in load_python_file
module = load_module_py(module_id, path)
File "/app/venv/lib/python2.7/site-packages/alembic/util/compat.py", line 141, in load_module_py
mod = imp.load_source(module_id, path, fp)
File "migrations/env.py", line 87, in <module>
run_migrations_online()
File "migrations/env.py", line 80, in run_migrations_online
context.run_migrations()
File "<string>", line 8, in run_migrations
File "/app/venv/lib/python2.7/site-packages/alembic/runtime/environment.py", line 836, in run_migrations
self.get_context().run_migrations(**kw)
File "/app/venv/lib/python2.7/site-packages/alembic/runtime/migration.py", line 330, in run_migrations
step.migration_fn(**kw)
File "/app/migrations/versions/e224df1a4818_.py", line 21, in upgrade
op.drop_table('parameter_subtype')
File "<string>", line 8, in drop_table
File "<string>", line 3, in drop_table
File "/app/venv/lib/python2.7/site-packages/alembic/operations/ops.py", line 1187, in drop_table
operations.invoke(op)
File "/app/venv/lib/python2.7/site-packages/alembic/operations/base.py", line 319, in invoke
return fn(self, operation)
File "/app/venv/lib/python2.7/site-packages/alembic/operations/toimpl.py", line 70, in drop_table
operation.to_table(operations.migration_context)
File "/app/venv/lib/python2.7/site-packages/alembic/ddl/impl.py", line 203, in drop_table
self._exec(schema.DropTable(table))
File "/app/venv/lib/python2.7/site-packages/alembic/ddl/impl.py", line 118, in _exec
return conn.execute(construct, *multiparams, **params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/app/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/app`enter code here`/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.InternalError: (psycopg2.InternalError) cannot drop table parameter_subtype because other objects depend on it
DETAIL: constraint parameter_parameter_subtype_id_fkey on table parameter depends on table parameter_subtype
HINT: Use DROP ... CASCADE to drop the dependent objects too.
[SQL: '\nDROP TABLE parameter_subtype']
This leads to table not created in PostgreSQL db and columns either.
Unsure what may have caused this problem and how to understand this Error message.
I did manually drop the tables and it helped. However how to be with migration.
Any Suggestions?
Flask-Migrate does not read in the database to see the dependencies between the objects. You can achieve a successful migration by reordering the drop_table in your migration file.
What you can do is modify the migration file like so:
op.drop_table("parameter_subtype")
op.drop_table("parameter")
And then run the upgrade command. Normally ordering the drop in this order should solve the problem.
I got the same error and what worked for me was to first drop the table in the database using the command:
$ DROP TABLE table_name;
And then in the migration file in the upgrade() function delete or comment out the line where it says:
op.drop_table('table_name')
The issue I am having is with the mixing of sqlalchemy sessions with flask-sqlalchemy session. So for context, I am running a flask-restful app on the background and I have a file (runnable.py) with a main method which is supposed to run periodically for an average period of 4 weeks.
This file uses its own session (pure sqlalchemy session) and connects to this app (which uses flask-sqlalchemy session).
After a few hours (sometimes 1 day), I get this error
Traceback (most recent call last):
File "/home/ubuntu/cron/runnable.py", line 781, in <module>
main()
File "/home/ubuntu/cron/runnable.py", line 677, in main
db_process_attendee(attendee, eid, event_id)
File "/home/ubuntu/cron/runnable.py", line 78, in db_process_attendee
update_attendee_information(attendee, attributes, eid, event_id)
File "/home/ubuntu/cron/runnable.py", line 117, in update_attendee_information
update_attendee_profile(attendee, id_attendee_login, eid, event_id)
File "/home/ubuntu/cron/runnable.py", line 291, in update_attendee_profile
generate_company_tags_lower.generate_company_tags(id_attendee_login)
File "/home/ubuntu/cron/app/util/generate_company_tags_lower.py", line 28, in generate_company_tags
filter(a_p.id_attendee_login == id_attendee_login).\
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2426, in scalar
ret = self.one()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2395, in one
ret = list(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/query.py", line 2453, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 322, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
context)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 436, in do_execute
cursor.execute(statement, parameters)
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 134, in execute
result = self._query(query)
File "/usr/local/lib/python2.7/dist-packages/pymysql/cursors.py", line 282, in _query
conn.query(q)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 767, in query
self._execute_command(COMMAND.COM_QUERY, sql)
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 957, in _execute_command
self._write_bytes(prelude + sql[:chunk_size-1])
File "/usr/local/lib/python2.7/dist-packages/pymysql/connections.py", line 916, in _write_bytes
raise err.OperationalError(2006, "MySQL server has gone away (%r)" % (e,))
sqlalchemy.exc.OperationalError: (OperationalError) (2006, "MySQL server has gone away (error(110, 'Connection timed out'))") 'SELECT a_p.id_attendee_profile AS a_p_id_attendee_profile, a_p.id_attendee_login AS a_p_id_attendee_login \nFROM a_p \nWHERE a_p.id_attendee_login = %s' (128368,)
I tried the following suggestions but they did not work:
• In my app config, adding "pymysql" to the SQLALCHEMY_DATABASE_URI
• Doing a db.session.commit() when I enter generate_company_tags(id_attendee_login) before writing a sql select query. I thought it would flush the session.
Btw, the sqlalchemy pool recycle value is set to less than the max in AWS RDS.
Any suggestions on how I can go solving this?
I have a very confusing problem with sqlalchemy. This isn't the first time I've used sqlalchemy. I'm working on a new project we just stood up from scratch, so it is possible there is a configuration error. I include the python connector package below from our pip reqs file in case it is of interest:
mysql-connector-repackaged==0.3.1
I just created a basic unit test to evaluate our sql wrapper class. The method under evaluation is an add_user class, which simply adds a user to the database. The class first runs a query to see if the user exists. This query is not succeeding.
Here is the code for the query:
q = self.session.query(User).\
filter_by(name=name, email=email)
result = q.all()
Seems simple enough, right? I'll include my connection string below in case anyone is interested:
db =create_engine('mysql+mysqlconnector://{user}:{password}#{host}:{port}/{database}'.format(user=conn.USER, password=conn.PASS, host=conn.HOST, port=conn.PORT, database=database, poolclass=NullPool))
The database is empty, although the table structure does exist. When this query is run it throws an exception. The exception reads as below:
StatementError: expected string or buffer (original cause: TypeError: expected string or buffer) u"SHOW VARIABLES LIKE 'sql_mode'" []
This exception comes from the python regular expression class re.py. The end of the stack trace looks like this:
File "/home/vagrant/GitRepos/SqlInteraction/venv/lib/python2.7/re.py", line 137, in match
return _compile(pattern, flags).match(string)
If I break on that line I can see the problem. The string object which the _compile method is running is not a string at all. It is a tuple. The tuple looks like this: (0, 3, 1, '', '')
Now, while it is obvious why that method is breaking, it is not at all obvious why a tuple is being passed in to that method. All of that happens internally to third party libraries. I include the full stack trace below for reference:
Traceback (most recent call last):
File "/home/vagrant/GitRepos/SqlInteraction/test/databasetest.py", line 14, in setUp
userId = self.add_user(Constants.TestObjects.USER_ID, Constants.TestObjects.USER_EMAIL, Constants.TestObjects.USER_PHONE)
File "/home/vagrant/GitRepos/SqlInteraction/test/databasetest.py", line 27, in add_user
result = q.all()
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2320, in all
return list(self)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2438, in __iter__
return self._execute_and_instances(context)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2451, in _execute_and_instances
close_with_result=True)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2442, in _connection_from_session
**kw)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 854, in connection
close_with_result=close_with_result)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 858, in _connection_for_bind
return self.transaction._connection_for_bind(engine)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 322, in _connection_for_bind
conn = bind.contextual_connect()
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1798, in contextual_connect
self.pool.connect(),
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 338, in connect
return _ConnectionFairy._checkout(self)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 644, in _checkout
fairy = _ConnectionRecord.checkout(pool)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 440, in checkout
rec = pool._do_get()
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 963, in _do_get
return self._create_connection()
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 285, in _create_connection
return _ConnectionRecord(self)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 416, in __init__
exec_once(self.connection, self)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/event/attr.py", line 250, in exec_once
self(*args, **kw)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/event/attr.py", line 260, in __call__
fn(*args, **kw)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 1219, in go
return once_fn(*arg, **kw)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 166, in first_connect
dialect.initialize(c)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2453, in initialize
self._detect_ansiquotes(connection)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/base.py", line 2718, in _detect_ansiquotes
connection.execute("SHOW VARIABLES LIKE 'sql_mode'"),
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
return self._execute_text(object, multiparams, params)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
statement, parameters
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 893, in _execute_context
None, None)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1159, in _handle_dbapi_exception
exc_info
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 889, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 635, in _init_statement
if not dialect.supports_unicode_statements and \
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 725, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqlconnector.py", line 97, in supports_unicode_statements
return util.py3k or self._mysqlconnector_version_info > (2, 0)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 725, in __get__
obj.__dict__[self.__name__] = result = self.fget(obj)
File "/home/vagrant/GitRepos/SqlInteraction/venv/local/lib/python2.7/site-packages/sqlalchemy/dialects/mysql/mysqlconnector.py", line 131, in _mysqlconnector_version_info
self.dbapi.__version__)
File "/home/vagrant/GitRepos/SqlInteraction/venv/lib/python2.7/re.py", line 137, in match
return _compile(pattern, flags).match(string)
StatementError: expected string or buffer (original cause: TypeError: expected string or buffer) u"SHOW VARIABLES LIKE 'sql_mode'" []
I have no idea why this problem is occurring. I've bounced around the net with no useful results. I'm hoping someone here at SO might have some idea what is going on.