I'm currently learning SQLAlchemy, and i found this strange thing. I was experimenting with a table which stores a person's name and address, and to get them i use this:
session.query(User)
And to get the first item, i tried:
session.query(User).first()
Which throws a DatabaseError:
Traceback (most recent call last):
File "<pyshell#4>", line 1, in <module>
session.query(User).first()
File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 2275, in first
ret = list(self[0:1])
File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 2142, in __getitem__
return list(res)
File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 2346, in __iter__
return self._execute_and_instances(context)
File "build\bdist.win32\egg\sqlalchemy\orm\query.py", line 2361, in _execute_and_instances
result = conn.execute(querycontext.statement, self._params)
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 664, in execute
return meth(self, multiparams, params)
File "build\bdist.win32\egg\sqlalchemy\sql\elements.py", line 272, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 761, in _execute_clauseelement
compiled_sql, distilled_params
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 874, in _execute_context
context)
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 1023, in _handle_dbapi_exception
exc_info
File "build\bdist.win32\egg\sqlalchemy\util\compat.py", line 185, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb)
File "build\bdist.win32\egg\sqlalchemy\engine\base.py", line 867, in _execute_context
context)
File "build\bdist.win32\egg\sqlalchemy\engine\default.py", line 376, in do_execute
cursor.execute(statement, parameters)
DatabaseError: (DatabaseError) ORA-01036: illegal variable name/number
'SELECT test_user_uid, test_user_name, test_user_address \nFROM (SELECT test_user."uid" AS test_user_uid, test_user.name AS test_user_name, test_user.address AS test_user_address \nFROM test_user) \nWHERE ROWNUM <= :ROWNUM_1' {'ROWNUM_1': 1}
However, i was able to retrieve what i wanted if i select all the rows, and loop through the query object:
users = [user for user in session.query(User)]
user1 = users[0]
That's all, i thought it's strange. Here's my mapping class:
class User(Base):
__tablename__ = 'test_user'
uid = Column(Integer, primary_key = True)
name = Column(String(50))
address = Column(String(100))
def __repr__(self):
return "<User (%s, %s)"%(self.name, self.address)
My best guess is that Session.query().first() is looking for the first row, with the generated query. However, the working method retrieves all the rows, and select the first one in Python. The problem is clearly from the generated query (invalid query). The main question is, what caused SQLAlchemy to create an invalid query?
Also, i noticed that SQLAlchemy makes things more difficult by making a query with sub-query. Is that behavior intended?
I hope i can get a satisfying answer, thanks!
Well, it didn't take me long to realize this. It turns out that it's a version problem, i was previously using cx_Oracle version 5.0.2 10g, i tried to upgrade it to version 5.1.2 10g, and things works fine.
This is probably an undocumented bug in SQLAlchemy, i can't find a place where they mention it.
Conclusion: If you want to use the latest version of SQLAlchemy (0.9.0b1) with Oracle 10g, you shouldn't use cx_Oracle older than version 5.1.2 10g.
Hope this helps, and thanks for reading the question!
Related
I have a user named mi_abc in oracle 11g.
The user do not have any table in the database but has access to all the tables in another schema sch_abc.
When I run a normal select query from sqldeveloper on the sch_abc schema from mi_abc, it works perfectly fine, but when I use django, I am always getting the error:-
django.db.utils.DatabaseError: ORA-00942: table or view does not exist
I tried to set the db_table = sch_abc.tableName and also set db_table = tableName but both gives me the same error. Any clue how to resolve this?
TRACE:-
Traceback (most recent call last):
File "C:\xxx\xxx\xxx\xxx\xxx\xxx\xxxx\lib\site-packages\django\core\handlers\exception.py", line 41, in inner
response = get_response(request)
File "C:\xxx\xxx\xxxx\xxxx\xxxx\Python\Python37\lib\site-packages\django\utils\deprecation.py", line 142, in __call__
response = self.process_response(request, response)
File "C:\xxxx\xxxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\contrib\sessions\middleware.py", line 58, in process_response
request.session.save()
File "C:\xxx\xxxx\xxxx\xxxx\xxxxx\Python\Python37\lib\site-packages\django\contrib\sessions\backends\db.py", line 81, in save
return self.create()
File "C:\xxxx\xxxxx\xxxx\xxxx\xxxxx\Python\Python37\lib\site-packages\django\contrib\sessions\backends\db.py", line 50, in create
self._session_key = self._get_new_session_key()
File "C:\xxxx\xxxxx\xxxxx\xxxxx\xxxx\Python\Python37\lib\site-packages\django\contrib\sessions\backends\base.py", line 164, in _get_new_session_key
if not self.exists(session_key):
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\contrib\sessions\backends\db.py", line 46, in exists
return self.model.objects.filter(session_key=session_key).exists()
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\models\query.py", line 673, in exists
return self.query.has_results(using=self.db)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\models\sql\query.py", line 517, in has_results
return compiler.has_results()
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\models\sql\compiler.py", line 858, in has_results
return bool(self.execute_sql(SINGLE))
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\models\sql\compiler.py", line 899, in execute_sql
raise original_exception
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\models\sql\compiler.py", line 889, in execute_sql
cursor.execute(sql, params)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\backends\utils.py", line 79, in execute
return super(CursorDebugWrapper, self).execute(sql, params)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\utils.py", line 94, in __exit__
six.reraise(dj_exc_type, dj_exc_value, traceback)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\utils\six.py", line 685, in reraise
raise value.with_traceback(tb)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\backends\utils.py", line 64, in execute
return self.cursor.execute(sql, params)
File "C:\xxx\xxx\xxx\xxx\xxx\Python\Python37\lib\site-packages\django\db\backends\oracle\base.py", line 497, in execute
return self.cursor.execute(query, self._param_generator(params))
django.db.utils.DatabaseError: ORA-00942: table or view does not exist
Well, I resolved the issue and let me tell you there is no straight way to do it in django. The problem with my application was, I was using the authetication features of django and also session handling. All these tables are created by django directly on the initial migration. So, there is no existence of them in the models.py file that you can simply append the schema name and ask your application to connect to the table of that schema.
What I ended up doing is, I created private synonyms to all the tables of the other schema which actually contained those tables. If you do this, you don't have to change anything in your django code. Your application will simply work because oracle will do the dirty work of actually connecting to the proper table. You will merely call the table in your application as if its your own. In this way when django looks for tables like django_session, auth_user etc, it simply queries it like it always does and oracle redirects it to the actual tables present in another schema.
Hope this helps people who face this issue in the future.
This is by no means officially supported, but this works in Postgres:
class Meta:
managed = False
db_table = 'schema\".\"table'
It took some trial and error for Postgres, but you can probably do something similar for Oracle. This is because the Postgres engine quotes object names, and this fakes the quoting mechanism out.
UPDATE:
After doing some digging, I found this for Oracle (modified for Python 3):
class Meta:
db_table = '"SCHEMA"."TABLE_NAME"'
Source: https://code.djangoproject.com/ticket/14136
I would recommend keeping managed = False unless you really know what you're doing. Good luck!
You can set the required schema, before executing the command. and then go back to public schema once the queryset is processed.
from django.db import connection
connection.set_schema(schema_name)
I'm using Doubles to track time in my database (as Unix timestamps with fractional portions). For that, I've created the following user type (that takes pytz-aware datetimes from the application and converts them to UTC timestamps).
from sqlalchemy.types import TypeDecorator, Float
class DoubleDatetime(TypeDecorator):
impl = Float
def __init__(self):
TypeDecorator.__init__(self, as_decimal=False)
def process_bind_param(self, value, dialect):
return value.timestamp()
def process_result_value(self, value, dialect):
return datetime.datetime.utcfromtimestamp(value)
This works fine for all datetimes in my system. I'm using the following column definition to have the database (in my case, SQLite3) insert the transaction time:
from sqlalchemy import Column
from sqlalchemy.sql.functions import now
column = Column("transaction_begin", DoubleDatetime, server_default=now())
Despite defining the transaction_begin column as a DoubleDatetime, my database still stores the records with a string.
How can I force the server transaction timestamp to use fractional Unix epochs?
What have I tried?
Reading the SQLite3 docs, I know I need to do some Julian Day trickery to do this.
SELECT (julianday('now') - 2440587.5) * 86400.0;
But I can't find julianday in sqlalchemy.sql.functions, to wrap it in a zero-argument lambda (to pass as the server_default arg to Column).
I tried the following, but it doesn't seem to work.
from sqlalchemy import text
server_default = text("SELECT (julianday('now') - 2440587.5) * 86400.0;")
Nor does the following:
from sqlalchemy.sql import select, text
server_default = select([text("(julianday('now') - 2440587.5) * 86400.0;")])
Traceback:
Traceback (most recent call last):
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlite3.OperationalError: near "SELECT": syntax error
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "tasks.py", line 81, in <module>
metadata.create_all(engine)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/schema.py", line 4005, in create_all
tables=tables)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1940, in _run_visitor
conn._run_visitor(visitorcallable, element, **kwargs)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1549, in _run_visitor
**kwargs).traverse_single(element)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 757, in visit_metadata
_is_metadata_operation=True)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/visitors.py", line 121, in traverse_single
return meth(obj, **kw)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 791, in visit_table
include_foreign_key_constraints=include_foreign_key_constraints
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
return meth(self, multiparams, params)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/sql/ddl.py", line 68, in _execute_on_connection
return connection._execute_ddl(self, multiparams, params)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1009, in _execute_ddl
compiled
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
context)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1413, in _handle_dbapi_exception
exc_info
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 265, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/util/compat.py", line 248, in reraise
raise value.with_traceback(tb)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
context)
File "/home/randm/Libraries/anaconda3/lib/python3.7/site-packages/sqlalchemy/engine/default.py", line 509, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) near "SELECT": syntax error [SQL: "\nCREATE TABLE barchartbarmessage (\n\tsymbol VARCHAR NOT NULL, \n\tdate DATE NOT NULL, \n\topen NUMERIC, \n\thigh NUMERIC, \n\tlow NUMERIC, \n\tclose NUMERIC, \n\tvolume INTEGER, \n\tprevious_day_open_interest INTEGER, \n\tprevious_day_volume INTEGER, \n\tmessage_time FLOAT, \n\ttransaction_begin FLOAT DEFAULT SELECT (julianday('now') - 2440587.5) * 86400.0; NOT NULL, \n\ttransaction_end FLOAT, \n\tPRIMARY KEY (symbol, date, transaction_begin)\n)\n\n"] (Background on this error at: http://sqlalche.me/e/e3q8)
I had this same question, and found that this worked. The expression here is specific to PostgreSQL. I originally was calculating this value on my application server, which led to unexpected results. See this post for more information. Note that in PostgreSQL, at least, when setting defaults using expressions you don't include SELECT at the start.
import sqlalchemy as sa
logged_ts_ms = sa.Column(sa.BigInteger, server_default=sa.text("""(extract(epoch from now()) * 1000)::bigint;"""))
I'm getting an error when I try to pass a non-executed query string into a raw query, using the params argument that Django recommends, to use as a subquery:
from apps.bikeshare.models import Station
qs = Station.objects.filter(...)
subquery_string = qs.values('id').order_by().query
raw = Station.objects.raw('SELECT * FROM bikeshare_station WHERE id IN (%s)', [subquery_string])
The query prints the correct SQL (WHERE clause omitted):
<RawQuerySet: SELECT * FROM bikeshare_station WHERE id IN (SELECT "bikeshare_station"."id" FROM "bikeshare_station" WHERE ...)>
However, executing the raw query (raw[0]) gives error:
Traceback (most recent call last):
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/IPython/core/interactiveshell.py", line 2847, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "<ipython-input-4-6eada3e31b4e>", line 1, in <module>
raw[0]
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/models/query.py", line 1329, in __getitem__
return list(self)[k]
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/models/query.py", line 1299, in __iter__
query = iter(self.query)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/models/sql/query.py", line 93, in __iter__
self._execute_query()
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/models/sql/query.py", line 127, in _execute_query
self.cursor.execute(self.sql, params)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 100, in execute
return super().execute(sql, params)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 68, in execute
return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 77, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/utils.py", line 89, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/home/gbrown/Envs/bikeshare-dev/lib/python3.5/site-packages/django/db/backends/utils.py", line 85, in _execute
return self.cursor.execute(sql, params)
django.db.utils.ProgrammingError: can't adapt type 'Query'
I'm aware that this is dangerous as it is SQL injection. However, I would like to support this, to create a QuerySet method (which happens to require raw query) that can be used on the end of a queryset chain where the filtered queryset is passed into the raw query as a subquery.
Would the params parameter mitigate any risk even if the error didn't happen? Or shall I just allow the risk and use basic python string formatting?
I've just released my method doesn't work properly anyway, since the query method doesn't return valid SQL, i.e.
qs = Station.objects.filter(name='Albert Gate, Hyde Park').values('id').order_by()
print(qs.query)
[out] SELECT "bikeshare_station"."id" FROM "bikeshare_station" WHERE "bikeshare_station"."name" = Albert Gate, Hyde Park
blows up because the name string is not quoted.
It seems the only way then is, as Jon Clements points out in his comment, to execute the query and pass in a list of ids which can be achieved:
qs = Station.objects.filter(name='Albert Gate, Hyde Park')
ids = list(qs.values_list('id', flat=True))
You could then pass the list of ids to the raw query rather than a string.
I have Postgres DB with a table of pending operations. One column in the operation in an enum with the status of the enum. I used the standard python (2.7) enum, with AutoNumber (myenum.py):
class AutoNumber(enum.Enum):
def __new__(cls):
value = len(cls.__members__) + 1
obj = object.__new__(cls)
obj._value_ = value
return obj
class MyStatus(AutoNumber):
INITIAL = ()
ACCEPTED = ()
DENIED = ()
ACK_PENDING = ()
AUTHORIZED = ()
ACTIVE = ()
END = ()
DELETED = ()
# end enum
The table looks like (also in myenum.py):
Base = declarative_base()
class MyOperation(Base):
__tablename__ = 'operations'
id = Column( Integer, primary_key=True )
status = Column( Enum(MyStatus) )
status_message = Column( String )
status_time = Column( DateTime )
def __repr__(self):
return "<MyOperation(%s, %s, %s, %s)>" % \
( self.id, self.status, self.status_time, self.status_message )
# end class
Generally this works fine. In the SAME FILE that defines MyStatus (myoper.py), I can change the status and save it back to the DB and it works fine:
def checkOper( oper ):
oper.status = MyStatus.DENIED
oper.status_message = "failed check (internal)"
oper.status_time = datetime.datetime.utcnow()
Here's how I call it (within myoper.py)
checkOper( oper )
session.add(oper)
session.commit()
This is all in the same file (myoper.py).
However, if I pass an oper object to an external function, and IT changes the status, then I get a sqlalchemy.exc.StatementError.
Here's the external function (myoper_test.py):
import datetime
from myoper import MyStatus
def extCheckOper( oper ):
oper.status = MyStatus.DENIED
oper.status_message = "failed check (external)"
oper.status_time = datetime.datetime.utcnow()
Here's how I call it (from myoper.py):
from myoper_test import extCheckOper
extCheckOper( oper )
session.add(oper)
session.commit()
Here's the stack trace:
Traceback (most recent call last):
File "./myoper.py", line 120, in <module>
session.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 906, in commit
self.transaction.commit()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 461, in commit
self._prepare_impl()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 441, in _prepare_impl
self.session.flush()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2177, in flush
self._flush(objects)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2297, in _flush
transaction.rollback(_capture_exception=True)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/langhelpers.py", line 66, in __exit__
compat.reraise(exc_type, exc_value, exc_tb)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/session.py", line 2261, in _flush
flush_context.execute()
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 389, in execute
rec.execute(self)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/unitofwork.py", line 548, in execute
uow
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 177, in save_obj
mapper, table, update)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/orm/persistence.py", line 737, in _emit_update_statements
execute(statement, multiparams)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 945, in execute
return meth(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
return connection._execute_clauseelement(self, multiparams, params)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
compiled_sql, distilled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
None, None)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1402, in _handle_dbapi_exception
exc_info
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
reraise(type(exception), exception, tb=exc_tb, cause=cause)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1116, in _execute_context
context = constructor(dialect, self, conn, *args)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 639, in _init_compiled
for key in compiled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 639, in <genexpr>
for key in compiled_params
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/sqltypes.py", line 1446, in process
value = self._db_value_for_elem(value)
File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/sqltypes.py", line 1354, in _db_value_for_elem
'"%s" is not among the defined enum values' % elem)
sqlalchemy.exc.StatementError: (exceptions.LookupError) "MyStatus.DENIED" is not among the defined enum values [SQL: u'UPDATE operations SET status=%(status)s, status_message=%(status_message)s, status_time=%(status_time)s WHERE operations.id = %(operations_id)s'] [parameters: [{'status': <MyStatus.DENIED: 6>, 'status_time': datetime.datetime(2017, 10, 18, 20, 22, 44, 350035), 'status_message': 'failed check (external)', 'operations_id': 3}]]
I've tried inspecting the type both in the internal file, and external file, but it's ways listed as <enum 'MyStatus'>.
I have found, that if I assign the oper.status to the enum .name, then that DOES work:
def extCheckOper( oper ):
oper.status = MyStatus.AUTHORIZED.name
oper.status_message = "authorized check (external)"
oper.status_time = datetime.datetime.utcnow()
But that's obviously pretty ugly.
So - what am I doing wrong? What is different about MyStatus in the file it's defined in, vs an external file that screws up SQL Alchemy?
I posted this question to the SQL Alchemy mailing list and got an answer. Link to thread
Turns out this one of those "gotcha's" about python and has nothing really to do with SQL Alchemy. Here's a reference: Executing Main Module Twice.
In this particular case, when I executed my script, the MyStatus was created with a particular id (python handle on the type). But when myoper_test imported MyStatus from myoper, it was created AGAIN with a different id.
So when the extCheckOper assigned a MyStatus value to the status field, it was a different MyStatus than SQL Alchemy created the DB mapping with, so when SQL Alchemy tried to save it to the DB, the " is " operator failed, since the (external) MyStatus was different than (original) MyStatus.
There are a couple of different workarounds. One way is to not run the code as main (after moving exiting main code into main() function):
$ python -c "from myoper import main; import sys; main(*sys.argv[1:])" ext_check 1
The better solution is to avoid this problem entirely - move the code that calls out externally to an internal test script. The code in main stays within mainly within the main script (sorry, couldn't resist... :-) ).
I have two tables, testInstance and bugzilla that are associated by a third one, bzCheck, like this:
class Instance(Base):
__tablename__ = "testInstance"
id = Column(Integer, primary_key=True)
bz_checks = relation(BZCheck, backref="instance")
class BZCheck(Base):
__tablename__ = "bzCheck"
instance_id = Column(Integer, ForeignKey("testInstance.id"), primary_key=True)
bz_id = Column(Integer, ForeignKey("bugzilla.id"), primary_key=True)
status = Column(String, nullable=False)
bug = relation(Bugzilla, backref="checks")
class Bugzilla(Base):
__tablename__ = "bugzilla"
id = Column(Integer, primary_key=True)
The backend is a postgresql server ; I'm using SQLalchemy 0.5
If I create the Instance, Bugzilla and BZCheck ojects, then do
bzcheck.bug = bugzilla
instance.bz_checks.append(bzcheck)
and then add and commit them ; everything is fine.
But now, let's assume I have an existing instance and an existing bugzilla and want to associate them:
instance = session.query(Instance).filter(Instance.id == 31).one()
bugzilla = session.query(Bugzilla).filter(Bugzilla.id == 19876).one()
check = BZCheck(status="OK")
check.bug = bugzilla
instance.bz_checks.append(check)
It fails:
In [6]: instance.bz_checks.append(check)
2012-01-09 18:43:50,713 INFO sqlalchemy.engine.base.Engine.0x...3bd0 select nextval('"bzCheck_instance_id_seq"')
2012-01-09 18:43:50,713 INFO sqlalchemy.engine.base.Engine.0x...3bd0 None
2012-01-09 18:43:50,713 INFO sqlalchemy.engine.base.Engine.0x...3bd0 ROLLBACK
It tries to get a new ID from an unexisting sequence instead of using the foreign key "testInstance.id"... I don't understand why.
I have had similar problems when trying to modify objects after commiting them ; I should have missed something fundamental but what ?
the part you're missing here is the stack trace. Always look at the stack trace - what is critical here is that it's autoflush, produced by the access of instance.bz_checks:
Traceback (most recent call last):
File "test.py", line 44, in <module>
instance.bz_checks.append(check)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/attributes.py", line 168, in __get__
return self.impl.get(instance_state(instance),dict_)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/attributes.py", line 453, in get
value = self.callable_(state, passive)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/strategies.py", line 563, in _load_for_state
result = q.all()
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/query.py", line 1983, in all
return list(self)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/query.py", line 2092, in __iter__
self.session._autoflush()
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/session.py", line 973, in _autoflush
self.flush()
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/session.py", line 1547, in flush
self._flush(objects)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/session.py", line 1616, in _flush
flush_context.execute()
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/unitofwork.py", line 328, in execute
rec.execute(self)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/unitofwork.py", line 472, in execute
uow
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/mapper.py", line 2291, in _save_obj
execute(statement, params)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 1405, in execute
params)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 1538, in _execute_clauseelement
compiled_sql, distilled_params
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 1646, in _execute_context
context)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/base.py", line 1639, in _execute_context
context)
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/default.py", line 330, in do_execute
cursor.execute(statement, parameters)
sqlalchemy.exc.IntegrityError: (IntegrityError) null value in column "instance_id" violates not-null constraint
'INSERT INTO "bzCheck" (bz_id, status) VALUES (%(bz_id)s, %(status)s) RETURNING "bzCheck".instance_id' {'status': 'OK', 'bz_id': 19876}
you can see this because the line of code is:
instance.bz_checks.append(check)
then autoflush:
self.session._autoflush()
File "/Users/classic/dev/sqlalchemy/lib/sqlalchemy/orm/session.py", line 973, in _autoflush
Three solutions:
a. temporarily disable autoflush (see http://www.sqlalchemy.org/trac/wiki/UsageRecipes/DisableAutoflush)
b. ensure that the BZCheck association object is always created with it's full state needed before accessing any collections:
BZState(bug=bugzilla, instance=instance)
(this is usually a good idea for association objects - they represent the association between two points so it's most appropriate that they be instantiated with this state)
c. change the cascade rules so that the operation of check.bug = somebug doesn't actually place check into the session just yet. You can do this with cascade_backrefs, described at http://www.sqlalchemy.org/docs/orm/session.html#controlling-cascade-on-backrefs. (but you'd need to be on 0.6 or 0.7)