We Keep Coding

How to override `Model.__init__` and respect `.using(db)` in Django?

I have the following code:
print(f"current database: {self.db}\ninfusate from database: {infusate._state.db}\ntracer from database: {tracer._state.db}")
FCirc.objects.using(self.db).get_or_create(
serum_sample=sample,
tracer=tracer,
element=label.element,
)
That is producing the following output and exception:
current database: validation
infusate from database: validation
tracer from database: validation
Validating FCirc updater: {'update_function': 'is_last_serum_peak_group', 'update_field': 'is_last', 'parent_field': 'serum_sample', 'child_fields': [], 'update_label': 'fcirc_calcs', 'generation': 2}
Traceback (most recent call last):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 581, in get_or_create
return self.get(**kwargs), False
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 435, in get
raise self.model.DoesNotExist(
DataRepo.models.fcirc.FCirc.DoesNotExist: FCirc matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/views/loading/validation.py", line 91, in validate_load_files
call_command(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 181, in call_command
return command.execute(*args, **defaults)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/management/commands/load_animals_and_samples.py", line 134, in handle
loader.load_sample_table(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/utils/sample_table_loader.py", line 426, in load_sample_table
FCirc.objects.using(self.db).get_or_create(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 588, in get_or_create
return self.create(**params), True
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 451, in create
obj = self.model(**kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/models/maintained_model.py", line 430, in __init__
super().__init__(*args, **kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 485, in __init__
_setattr(self, field.name, rel_obj)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 229, in __set__
raise ValueError('Cannot assign "%r": the current database router prevents relation between database "%s" and "%s".' % (value, instance._state.db, value._state.db))
ValueError: Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents this relation.
Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents this relation.
Knowing that this error relates to foreign relations between records in different databases, as a sanity check, I modified the source of related_descriptors.py to include more info:
raise ValueError('Cannot assign "%r": the current database router prevents relations between database "%s" and "%s".' % (value, instance._state.db, value._state.db))
And that prints:
Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents relations between database "default" and "validation".
So I was going nuts. Why is it ignoring my .using(self.db) call?!
Then I realized, "Oh yeah - I over-rode __init__ in the superclass to FCirc! I'm probably circumventing using(db).":
class FCirc(MaintainedModel, HierCachedModel):
...
class MaintainedModel(Model):
...
Out of the 2 superclass mixes, MaintainedModel seems to be the culprit in this case. It's the only one that overrides __init__. That override looks like this:
def __init__(self, *args, **kwargs):
"""
This over-ride of the constructor is to prevent developers from explicitly setting values for automatically
maintained fields. It also performs a one-time validation check of the updater_dicts.
"""
# ... about 80 lines of code that I'm very confident are unrelated to the problem. See the docstring above. Will paste upon request ...
# vvv THIS LINE IS LINE 430 FROM maintained_model.py IN THE TRACE ABOVE
super().__init__(*args, **kwargs)
How do I pass along self.db in the super constructor?

I figured it out! Another developer had added calls to full_clean (which I have a separate question about) in derived classes of the QuerySet class for a few of the models. E.g.:
class TracerQuerySet(models.QuerySet):
...
class Tracer(MaintainedModel):
objects = TracerQuerySet().as_manager()
...
Those calls to full_clean appear to only work on the default database. At least, I haven't been able to figure out how to tell full_clean to operate on our validation database.
I changed the 3 or 4 calls I found QuerySet derived classes to only call full_clean if the current database is the default database:
if self._db == settings.DEFAULT_DB:
tracer.full_clean()
After that, I no longer get the exception on the FCirc get_or_create call and the database operated on is the validation database.
I was curious though if I had any other database operations that were being assigned to the default database, so I added a debug print to another piece of code in django.db.utils.py: _router_func:
def _router_func(action):
def _route_db(self, model, **hints):
chosen_db = None
for router in self.routers:
try:
method = getattr(router, action)
except AttributeError:
# If the router doesn't have a method, skip to the next one.
pass
else:
chosen_db = method(model, **hints)
if chosen_db:
return chosen_db
instance = hints.get('instance')
if instance is not None and instance._state.db:
return instance._state.db
###
### See what code is getting the default database
###
print(f"Returning default database. Trace:")
traceback.print_stack()
return DEFAULT_DB_ALIAS
return _route_db
That revealed 2 other places that were querying the default database when I knew it should only be querying the validation database.
This one made sense, because it's a fresh database query:
def tracer_labeled_elements(self):
"""
This method returns a unique list of the labeled elements that exist among the tracers.
"""
from DataRepo.models.tracer_label import TracerLabel
return list(
TracerLabel.objects.filter(tracer__infusates__id=self.id)
.order_by("element")
.distinct("element")
.values_list("element", flat=True)
)
So I modified it to use the current database (based on the instance) instead:
def tracer_labeled_elements(self):
"""
This method returns a unique list of the labeled elements that exist among the tracers.
"""
from DataRepo.models.tracer_label import TracerLabel
db = self.get_using_db()
return list(
TracerLabel.objects.using(db).filter(tracer__infusates__id=self.id)
.order_by("element")
.distinct("element")
.values_list("element", flat=True)
)
And I added this to my MaintainedModel class:
def get_using_db(self):
"""
If an instance method makes an unrelated database query and a specific database is currently in use, this
method will return that database to be used in the fresh query's `.using()` call. Otherwise, django's code
base will set the ModelState to the default database, which may differ from where the current model object came
from.
"""
db = settings.DEFAULT_DB
if hasattr(self, "_state") and hasattr(self._state, "db"):
db = self._state.db
return db
That works, however I don't like that it requires the developer to include code in the derived class specific to the super class. Plus, settings.DEFAULT_DB is project-specific, so it creates an inter-dependency.
What I really need to do is change the default database that the django base code sets when no database is explicitly specified...
My bet is that that's easy to do. I've just never done it before. I'll start looking into that and I'll probably edit this answer again soon.
Previous "answer"
Alright, this is not really an answer, because I don't want developers to have to jump through these ridiculous hoops to use get_or_create on a model that inherits from MaintainedModel, but it solves the problem. It prevents the exception and everything is applied to the correct database.
Maybe this will give someone else a hint as to how to correctly solve the problem inside the override of the __init__ constructor in MaintainedModel:
from django.db.models.base import ModelState
ms = ModelState
setattr(ms, "db", self.db)
print(f"current database: {self.db}\ninfusate from database: {infusate._state.db}\ntracer from database: {tracer._state.db}\nsample from database: {sample._state.db}\n_state type: {type(tracer._state)} db type: {type(tracer._state.db)}")
using_obj = FCirc.objects.using(self.db)
setattr(using_obj, "_state", ms)
print(f"using_obj database: {using_obj._state.db}")
using_obj.get_or_create(
serum_sample=sample,
tracer=tracer,
element=label.element,
)
The output is:
current database: validation
infusate from database: validation
tracer from database: validation
sample from database: validation
_state type: <class 'django.db.models.base.ModelState'> db type: <class 'str'>
using_obj database: validation

Class name not definied, but it is

I just wanna create a class definition with a static field with a name. A file called exercises.py contains:
First error:
FAIL: test_00_packages (__main__.Ex00)
Traceback (most recent call last):
File "ex00.py", line 55, in test_00_packages
self.assertTrue("Exercise00" in globals())
AssertionError: False is not true
Later:
class Exercise00:
def __init__(self, STUDENT_NAME):
self.STUDENT_NAME = 'Name Name'
But if I try to print Exercise00.STUDENT_NAME I just get
NameError: name 'Exercise00' is not defined
But I guess I defined it?!
Here the complete error:
ERROR: test_01_static_field (__main__.Ex00)
----------------------------------------------------------------------
Traceback (most recent call last):
File "ex00.py", line 60, in test_01_static_field
print("[I] Name: " + Exercise00.STUDENT_NAME)
NameError: name 'Exercise00' is not defined
----------------------------------------------------------------------
My task is to create a class called Exercise00 with a static field STUDENT_NAME.
The line in ex00.py is:
def test_00_packages(self):
self.assertTrue("Exercise00" in globals())

Two problems:
The test class is in a separate file exercises.py; you need to import the relevant functionality from that file (from exercises import Exercise00) before the module contents are visible from ex00.py.
Once you fix that, you will get a different error. Like the name of the test says, you are supposed to be looking for a static field, i.e. something that belongs to the class itself. This code attaches STUDENT_NAME to instances of Exercise00.

I suppose you need to define STUDENT_NAME as a class-level field, not as an instance level attribute:
class Exercise00:
STUDENT_NAME = 'Name Name'
You can notice in the error message that the test calls class level field Exercise00.STUDENT_NAME:
print("[I] Name: " + Exercise00.STUDENT_NAME)
And you also need to import class Exercise00 in the test module:
from exercises import Exercise00
As soon as you add the import statement to the file with the test ex00.py, the class name string appears in globals() and the test passes.

Error while testing the raise of self-defined exceptions (using assertRaises())

I am creating tests for a python project. The normal tests work just fine, however I want to test if in a certain condition my function raises a self-defined exception. Therefor I want to use assertRaises(Exception, Function). Any ideas?
The function that raises the exception is:
def connect(comp1, comp2):
if comp1 == comp2:
raise e.InvalidConnectionError(comp1, comp2)
...
The exception is:
class InvalidConnectionError(Exception):
def __init__(self, connection1, connection2):
self._connection1 = connection1
self._connection2 = connection2
def __str__(self):
string = '...'
return string
The test method is the following:
class TestConnections(u.TestCase):
def test_connect_error(self):
comp = c.PowerConsumer('Bus', True, 1000)
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
However I get the following error:
Error
Traceback (most recent call last):
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\test_component.py", line 190, in test_connect_error
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
File "C:\Users\t5ycxK\PycharmProjects\ElectricPowerDesign\component.py", line 428, in connect
raise e.InvalidConnectionError(comp1, comp2)
InvalidConnectionError: <unprintable InvalidConnectionError object>

assertRaises expects to actually perform the call. Yet, you already perform it by yourself, thereby throwing the error before assertRaises actually executes.
self.assertRaises(e.InvalidConnectionError, c.connect(comp, comp))
# run this ^ with first static argument ^ and second argument ^ from `c.connect(comp, comp)`
Use either of those instead:
self.assertRaises(e.InvalidConnectionError, c.connect, comp, comp)
with self.assertRaises(e.InvalidConnectionError):
c.connect(comp, comp)

implementing a deferred exception in Python

I would like to implement a deferred exception in Python that is OK to store somewhere but as soon as it is used in any way, it raises the exception that was deferred. Something like this:
# this doesn't work but it's a start
class DeferredException(object):
def __init__(self, exc):
self.exc = exc
def __getattr__(self, key):
raise self.exc
# example:
mydict = {'foo': 3}
try:
myval = obtain_some_number()
except Exception as e:
myval = DeferredException(e)
mydict['myval'] = myval
def plus_two(x):
print x+2
# later on...
plus_two(mydict['foo']) # prints 5
we_dont_use_this_val = mydict['myval'] # Always ok to store this value if not used
plus_two(mydict['myval']) # If obtain_some_number() failed earlier,
# re-raises the exception, otherwise prints the value + 2.
The use case is that I want to write code to analyze some values from incoming data; if this code fails but the results are never used, I want it to fail quietly; if it fails but the results are used later, then I'd like the failure to propagate.
Any suggestions on how to do this? If I use my DeferredException class I get this result:
>>> ke = KeyError('something')
>>> de = DeferredException(ke)
>>> de.bang # yay, this works
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "<stdin>", line 6, in __getattr__
KeyError: 'something'
>>> de+2 # boo, this doesn't
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: unsupported operand type(s) for +: 'DeferredException' and 'int'

Read section 3.4.12 of the docs, "Special method lookup for new-style classes." It explains exactly the problem you have encountered. The normal attribute lookup is bypassed by the interpreter for certain operators, such as addition (as you found out the hard way). Thus the statement de+2 in your code never calls your getattr function.
The only solution, according to that section, is to insure that "the special method must be set on the class object itself in order to be consistently invoked by the interpreter."
Perhaps you'd be better off storing all your deferred exceptions in a global list, wrapping your entire program in a try:finally: statement, and printing out the whole list in the finally block.

AssertRaises fails even though exception is raised

I am running into the following rather strange problem:
I am developing a django app and in my models class I am defining an exception that should be raised when a validation fails:
class MissingValueException(Exception):
"""Raise when a required attribute is missing."""
def __init__(self, message):
super(MissingValueException, self).__init__()
self.message = message
def __str__(self):
return repr(self.message)
This code is called from a publication class in a validation method:
def validate_required_fields(self):
# Here is the validation code.
if all_fields_present:
return True
else:
raise MissingValueException(errors)
In my unit test I create a case where the exception should be raised:
def test_raise_exception_incomplete_publication(self):
publication = Publication(publication_type="book")
self.assertRaises(MissingValueException, publication.validate_required_fields)
This produces the following output:
======================================================================
ERROR: test_raise_exception_incomplete_publication (core_knowledge_platform.core_web_service.tests.logic_tests.BusinessLogicTests)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/tests/logic_tests.py", line 45, in test_raise_exception_incomplete_publication
self.assertRaises(MissingValueException, method, )
File "/usr/lib/python2.7/unittest/case.py", line 465, in assertRaises
callableObj(*args, **kwargs)
File "/media/data/Dokumente/Code/master_project/core_knowledge_platform/../core_knowledge_platform/core_web_service/models.py", line 150, in validate_required_fields
raise MissingValueException(errors)
MissingValueException: 'Publication of type book is missing field publisherPublication of type book is missing field titlePublication of type book is missing field year'
So it looks like the exception is raised (which is the case - I even checked it in an interactive IPython session), but it seems that assertRaises is not catching it.
Anyone has any idea why this might happen?
Thanks

This could happen if your tests and your product code are importing your exception class through two different paths, so asserRaises doesn't realize that the exception you got was the one you were looking for.
Look at your imports, make sure that they are the same in both places. Having the same directories available in two different ways in your PYTHONPATH can make this happen. Symbolic links in those entries can also confuse things.