We Keep Coding

How to override `Model.__init__` and respect `.using(db)` in Django?

I have the following code:
print(f"current database: {self.db}\ninfusate from database: {infusate._state.db}\ntracer from database: {tracer._state.db}")
FCirc.objects.using(self.db).get_or_create(
serum_sample=sample,
tracer=tracer,
element=label.element,
)
That is producing the following output and exception:
current database: validation
infusate from database: validation
tracer from database: validation
Validating FCirc updater: {'update_function': 'is_last_serum_peak_group', 'update_field': 'is_last', 'parent_field': 'serum_sample', 'child_fields': [], 'update_label': 'fcirc_calcs', 'generation': 2}
Traceback (most recent call last):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 581, in get_or_create
return self.get(**kwargs), False
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 435, in get
raise self.model.DoesNotExist(
DataRepo.models.fcirc.FCirc.DoesNotExist: FCirc matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/views/loading/validation.py", line 91, in validate_load_files
call_command(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/__init__.py", line 181, in call_command
return command.execute(*args, **defaults)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/core/management/base.py", line 398, in execute
output = self.handle(*args, **options)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/management/commands/load_animals_and_samples.py", line 134, in handle
loader.load_sample_table(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/utils/sample_table_loader.py", line 426, in load_sample_table
FCirc.objects.using(self.db).get_or_create(
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 588, in get_or_create
return self.create(**params), True
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 451, in create
obj = self.model(**kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/DataRepo/models/maintained_model.py", line 430, in __init__
super().__init__(*args, **kwargs)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 485, in __init__
_setattr(self, field.name, rel_obj)
File "/Users/rleach/PROJECT-local/TRACEBASE/tracebase/.venv/lib/python3.9/site-packages/django/db/models/fields/related_descriptors.py", line 229, in __set__
raise ValueError('Cannot assign "%r": the current database router prevents relation between database "%s" and "%s".' % (value, instance._state.db, value._state.db))
ValueError: Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents this relation.
Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents this relation.
Knowing that this error relates to foreign relations between records in different databases, as a sanity check, I modified the source of related_descriptors.py to include more info:
raise ValueError('Cannot assign "%r": the current database router prevents relations between database "%s" and "%s".' % (value, instance._state.db, value._state.db))
And that prints:
Cannot assign "<Tracer: lysine-[13C6]>": the current database router prevents relations between database "default" and "validation".
So I was going nuts. Why is it ignoring my .using(self.db) call?!
Then I realized, "Oh yeah - I over-rode __init__ in the superclass to FCirc! I'm probably circumventing using(db).":
class FCirc(MaintainedModel, HierCachedModel):
...
class MaintainedModel(Model):
...
Out of the 2 superclass mixes, MaintainedModel seems to be the culprit in this case. It's the only one that overrides __init__. That override looks like this:
def __init__(self, *args, **kwargs):
"""
This over-ride of the constructor is to prevent developers from explicitly setting values for automatically
maintained fields. It also performs a one-time validation check of the updater_dicts.
"""
# ... about 80 lines of code that I'm very confident are unrelated to the problem. See the docstring above. Will paste upon request ...
# vvv THIS LINE IS LINE 430 FROM maintained_model.py IN THE TRACE ABOVE
super().__init__(*args, **kwargs)
How do I pass along self.db in the super constructor?

I figured it out! Another developer had added calls to full_clean (which I have a separate question about) in derived classes of the QuerySet class for a few of the models. E.g.:
class TracerQuerySet(models.QuerySet):
...
class Tracer(MaintainedModel):
objects = TracerQuerySet().as_manager()
...
Those calls to full_clean appear to only work on the default database. At least, I haven't been able to figure out how to tell full_clean to operate on our validation database.
I changed the 3 or 4 calls I found QuerySet derived classes to only call full_clean if the current database is the default database:
if self._db == settings.DEFAULT_DB:
tracer.full_clean()
After that, I no longer get the exception on the FCirc get_or_create call and the database operated on is the validation database.
I was curious though if I had any other database operations that were being assigned to the default database, so I added a debug print to another piece of code in django.db.utils.py: _router_func:
def _router_func(action):
def _route_db(self, model, **hints):
chosen_db = None
for router in self.routers:
try:
method = getattr(router, action)
except AttributeError:
# If the router doesn't have a method, skip to the next one.
pass
else:
chosen_db = method(model, **hints)
if chosen_db:
return chosen_db
instance = hints.get('instance')
if instance is not None and instance._state.db:
return instance._state.db
###
### See what code is getting the default database
###
print(f"Returning default database. Trace:")
traceback.print_stack()
return DEFAULT_DB_ALIAS
return _route_db
That revealed 2 other places that were querying the default database when I knew it should only be querying the validation database.
This one made sense, because it's a fresh database query:
def tracer_labeled_elements(self):
"""
This method returns a unique list of the labeled elements that exist among the tracers.
"""
from DataRepo.models.tracer_label import TracerLabel
return list(
TracerLabel.objects.filter(tracer__infusates__id=self.id)
.order_by("element")
.distinct("element")
.values_list("element", flat=True)
)
So I modified it to use the current database (based on the instance) instead:
def tracer_labeled_elements(self):
"""
This method returns a unique list of the labeled elements that exist among the tracers.
"""
from DataRepo.models.tracer_label import TracerLabel
db = self.get_using_db()
return list(
TracerLabel.objects.using(db).filter(tracer__infusates__id=self.id)
.order_by("element")
.distinct("element")
.values_list("element", flat=True)
)
And I added this to my MaintainedModel class:
def get_using_db(self):
"""
If an instance method makes an unrelated database query and a specific database is currently in use, this
method will return that database to be used in the fresh query's `.using()` call. Otherwise, django's code
base will set the ModelState to the default database, which may differ from where the current model object came
from.
"""
db = settings.DEFAULT_DB
if hasattr(self, "_state") and hasattr(self._state, "db"):
db = self._state.db
return db
That works, however I don't like that it requires the developer to include code in the derived class specific to the super class. Plus, settings.DEFAULT_DB is project-specific, so it creates an inter-dependency.
What I really need to do is change the default database that the django base code sets when no database is explicitly specified...
My bet is that that's easy to do. I've just never done it before. I'll start looking into that and I'll probably edit this answer again soon.
Previous "answer"
Alright, this is not really an answer, because I don't want developers to have to jump through these ridiculous hoops to use get_or_create on a model that inherits from MaintainedModel, but it solves the problem. It prevents the exception and everything is applied to the correct database.
Maybe this will give someone else a hint as to how to correctly solve the problem inside the override of the __init__ constructor in MaintainedModel:
from django.db.models.base import ModelState
ms = ModelState
setattr(ms, "db", self.db)
print(f"current database: {self.db}\ninfusate from database: {infusate._state.db}\ntracer from database: {tracer._state.db}\nsample from database: {sample._state.db}\n_state type: {type(tracer._state)} db type: {type(tracer._state.db)}")
using_obj = FCirc.objects.using(self.db)
setattr(using_obj, "_state", ms)
print(f"using_obj database: {using_obj._state.db}")
using_obj.get_or_create(
serum_sample=sample,
tracer=tracer,
element=label.element,
)
The output is:
current database: validation
infusate from database: validation
tracer from database: validation
sample from database: validation
_state type: <class 'django.db.models.base.ModelState'> db type: <class 'str'>
using_obj database: validation

Python telegram bot's `get_chat_members_count` & avoiding flood limits or how to use wrappers and decorators

I'm checking a list of around 3000 telegram chats to get and retrieve the number of chat members in each chat using the get_chat_members_count method.
At some point I'm hitting a flood limit and getting temporarily banned by Telegram BOT.
Traceback (most recent call last):
File "C:\Users\alexa\Desktop\ico_icobench_2.py", line 194, in <module>
ico_tel_memb = bot.get_chat_members_count('#' + ico_tel_trim, timeout=60)
File "C:\Python36\lib\site-packages\telegram\bot.py", line 60, in decorator
result = func(self, *args, **kwargs)
File "C:\Python36\lib\site-packages\telegram\bot.py", line 2006, in get_chat_members_count
result = self._request.post(url, data, timeout=timeout)
File "C:\Python36\lib\site-packages\telegram\utils\request.py", line 278, in post
**urlopen_kwargs)
File "C:\Python36\lib\site-packages\telegram\utils\request.py", line 208, in _request_wrapper
message = self._parse(resp.data)
File "C:\Python36\lib\site-packages\telegram\utils\request.py", line 168, in _parse
raise RetryAfter(retry_after)
telegram.error.RetryAfter: Flood control exceeded. Retry in 85988 seconds
The python-telegram-bot wiki gives a detailed explanation and example on how to avoid flood limits here.
However, I'm struggling to implement their solution and I hope someone here has more knowledge of this than myself.
I have literally made a copy and paste of their example and can't get it to work no doubt because i'm new to python. I'm guessing I'm missing some definitions but I'm not sure which. Here is the code below and after that the first error I'm receiving. Obviously the TOKEN needs to be replaced with your token.
import telegram.bot
from telegram.ext import messagequeue as mq
class MQBot(telegram.bot.Bot):
'''A subclass of Bot which delegates send method handling to MQ'''
def __init__(self, *args, is_queued_def=True, mqueue=None, **kwargs):
super(MQBot, self).__init__(*args, **kwargs)
# below 2 attributes should be provided for decorator usage
self._is_messages_queued_default = is_queued_def
self._msg_queue = mqueue or mq.MessageQueue()
def __del__(self):
try:
self._msg_queue.stop()
except:
pass
super(MQBot, self).__del__()
#mq.queuedmessage
def send_message(self, *args, **kwargs):
'''Wrapped method would accept new `queued` and `isgroup`
OPTIONAL arguments'''
return super(MQBot, self).send_message(*args, **kwargs)
if __name__ == '__main__':
from telegram.ext import MessageHandler, Filters
import os
token = os.environ.get('TOKEN')
# for test purposes limit global throughput to 3 messages per 3 seconds
q = mq.MessageQueue(all_burst_limit=3, all_time_limit_ms=3000)
testbot = MQBot(token, mqueue=q)
upd = telegram.ext.updater.Updater(bot=testbot)
def reply(bot, update):
# tries to echo 10 msgs at once
chatid = update.message.chat_id
msgt = update.message.text
print(msgt, chatid)
for ix in range(10):
bot.send_message(chat_id=chatid, text='%s) %s' % (ix + 1, msgt))
hdl = MessageHandler(Filters.text, reply)
upd.dispatcher.add_handler(hdl)
upd.start_polling()
The first error I get is:
Traceback (most recent call last):
File "C:\Users\alexa\Desktop\z test.py", line 34, in <module>
testbot = MQBot(token, mqueue=q)
File "C:\Users\alexa\Desktop\z test.py", line 9, in __init__
super(MQBot, self).__init__(*args, **kwargs)
File "C:\Python36\lib\site-packages\telegram\bot.py", line 108, in __init__
self.token = self._validate_token(token)
File "C:\Python36\lib\site-packages\telegram\bot.py", line 129, in _validate_token
if any(x.isspace() for x in token):
TypeError: 'NoneType' object is not iterable
The second issue I have is how to use wrappers and decorators with get_chat_members_count.
The code I have added to the example is:
#mq.queuedmessage
def get_chat_members_count(self, *args, **kwargs):
return super(MQBot, self).get_chat_members_count(*args, **kwargs)
But nothing happens and I don't get my count of chat members. I'm also not saying which chat I need to count so not surprising I'm getting nothing back but where am I supposed to put the telegram chat id?

You are getting this error because MQBot receives an empty token. For some reason, it does not raise a descriptive exception but instead crashes unexpectedly.
So why token is empty? It seems that you are using os.environ.get incorrectly. The os.environ part is a dictionary and its' method get allows one to access dict's contents safely. According to docs:
get(key[, default])
Return the value for key if key is in the dictionary, else default. If default is not given, it defaults to None, so that this method never raises a KeyError.
According to your question, in this part token = os.environ.get('TOKEN') you pass token itself as a key. Instead, you should've passed the name of the environmental variable which contains your token.
You can fix this either rewriting that part like this token = 'TOKEN' or by setting environmental variable correctly and accessing it from os.environ.get via correct name.

Pymongo failing but won't give exception

Here is the query in Pymongo
import mong #just my library for initializing
collection_1 = mong.init(collect="col_1")
collection_2 = mong.init(collect="col_2")
for name in collection_2.find({"field1":{"$exists":0}}):
try:
to_query = name['something']
actual_id = collection_1.find_one({"something":to_query})['_id']
crap_id = name['_id']
collection_2.update({"_id":id},{"$set":{"new_name":actual_id}},upset=True)
except:
open('couldn_find_id.txt','a').write(name)
All this is doing is taking a field from one collection, finding the id of that field and updating the id of another collection. It works for about 1000-5000 iterations, but periodically fails with this and then I have to restart the script.
> Traceback (most recent call last):
File "my_query.py", line 6, in <module>
for name in collection_2.find({"field1":{"$exists":0}}):
File "/home/user/python_mods/pymongo/pymongo/cursor.py", line 814, in next
if len(self.__data) or self._refresh():
File "/home/user/python_mods/pymongo/pymongo/cursor.py", line 776, in _refresh
limit, self.__id))
File "/home/user/python_mods/pymongo/pymongo/cursor.py", line 720, in __send_message
self.__uuid_subtype)
File "/home/user/python_mods/pymongo/pymongo/helpers.py", line 98, in _unpack_response
cursor_id)
pymongo.errors.OperationFailure: cursor id '7578200897189065658' not valid at server
^C
bye
Does anyone have any idea what this failure is, and how I can turn it into an exception to continue my script even at this failure?
Thanks

The reason of the problem is described in pymongo's FAQ:
Cursors in MongoDB can timeout on the server if they’ve been open for
a long time without any operations being performed on them. This can
lead to an OperationFailure exception being raised when attempting to
iterate the cursor.
This is because of the timeout argument of collection.find():
timeout (optional): if True (the default), any returned cursor is
closed by the server after 10 minutes of inactivity. If set to False,
the returned cursor will never time out on the server. Care should be
taken to ensure that cursors with timeout turned off are properly
closed.
Passing timeout=False to the find should fix the problem:
for name in collection_2.find({"field1":{"$exists":0}}, timeout=False):
But, be sure you are closing the cursor properly.
Also see:
mongodb cursor id not valid error

Using python to parse email for action server

I found this python code on the forums as an answer to something that relates to my problem. I don't really understand python, so can somebody tell me why this isn't working?
(Some background information: I have a web form that get automatically emailed to openERP, which then automatically creates a lead. However, when a lead is created, info like phone and name do not get read from the email and sorted into their corresponding fields in the lead's form.)
# You can use the following variables:
# - self: ORM model of the record on which the action is triggered
# - object: browse_record of the record on which the action is triggered if there is one, otherwise None
# - pool: ORM model pool (i.e. self.pool)
# - time: Python time module
# - cr: database cursor
# - uid: current user id
# - context: current context
# If you plan to return an action, assign: action = {...}
def parse_description(description):
'''
there is parse function
It is example for parsing messages like this:
Name: John
Phone: +100500
'''
fields=['Name','Phone']
_dict={}
description=description.lower()
for line in description.split('\n'):
for field in fields:
if field in line:
split_line=line.split(':')
if len(split_line)>1:
pre_dict[field]=line.split(':')[1]
return dict
lead=self.browse(cr,uid,context['active_id'],context=context)
description=lead['description']
_dict=parse_description(description)
self.write(cr,uid,context['active_id'],{
'partner_name':_dict.get('name'),
'contact_name':_dict.get('name'),
'phone':_dict.get(u'phone'),
'mobile':_dict.get(u'phone')})
Update:
I got these traceback while I am fetching mail
2014-07-01 13:39:40,188 4992 INFO v8_demo openerp.addons.mail.mail_thread: Routing
mail from Atul Jain <jain.atul43#gmail.com> to jain.atul10#hotmail.com with
Message-Id <CAG=2G76_SRthL3ybGGyx2Lai5H=RMNxUOjRRR=+5-ODrcgtEZw#mail.gmail.com>:
fallback to model:crm.lead, thread_id:False, custom_values:None, uid:1
2014-07-01 13:39:40,445 4992 ERROR v8_demo openerp.addons.fetchmail.fetchmail:
Failed to fetch mail from imap server Gmail.
Traceback (most recent call last):
File "/home/atul/openerp-8/openerp/addons/fetchmail/fetchmail.py", line 206, in
fetch_mail
action_pool.run(cr, uid, [server.action_id.id], {'active_id': res_id, 'active_ids'
:[res_id], 'active_model': context.get("thread_model", server.object_id.model)})
File "/home/atul/openerp-8/openerp/addons/base/ir/ir_actions.py", line 967, in run
res = func(cr, uid, action, eval_context=eval_context, context=run_context)
File "/home/atul/openerp-8/openerp/addons/base/ir/ir_actions.py", line 805,
in run_action_code_multi
eval(action.code.strip(), eval_context, mode="exec", nocopy=True) # nocopy allows
to return 'action'
File "/home/atul/openerp-8/openerp/tools/safe_eval.py", line 254, in safe_eval
return eval(c, globals_dict, locals_dict)
File "", line 14, in <module>
File "", line 4, in parse_description
ValueError: "'bool' object has no attribute 'lower'" while evaluating
u"def parse_description(description):
fields=['name','phone']
_dict={}
description=description.lower()
for line in description.split('\\n'):
for field in fields:
if field in line:
split_line=line.split(':')
if len(split_line)>1:
_dict[field]=split_line[1]
return _dict
lead=self.browse(cr,uid,context['active_id'],context=context)\ndescription=lead['description']
_dict=parse_description(description)
self.write(cr,uid,context['active_id'],{ 'partner_name':_dict.get('name'), 'contact_name':_dict.get('name'),
'phone':_dict.get(u'phone'),
'mobile':_dict.get(u'phone')})"
Please help me in understanding the problem.

I've fixed the parse_description function:
def parse_description(description):
'''
there is parse function
It is example for parsing messages like this:
Name: John
Phone: +100500
'''
fields=['name','phone']
_dict={}
description=description.lower()
for line in description.split('\n'):
for field in fields:
if field in line:
split_line=line.split(':')
if len(split_line)>1:
_dict[field]=split_line[1]
return _dict
I changed the fields values to lower case because all operations on the description are on description.lower().
On the line pre_dict[field]=line.split(':')[1], you are splitting line to get your result. This has already been done: split_line=line.split(':') so you can just replace the pre_dict line with pre_dict[field]=split_line[1]
On that same line you are using a variable, pre_dict, which hasn't been referenced before. I think you mean to use _dict so the line should be _dict[field]=split_line[1]
The function returns dict which is a type, not a variable. You probably want it to return the dictionary which contains the field data, so it should return _dict instead; otherwise you'll always get the result <type 'dict'>
As for the remaining code, there's not enough context for me to understand what's happening or what's wrong. At least the parse_description function should be working now.

Threading in a django app

I'm trying to create what I call a scrobbler. The task is to read a Delicious user from a queue, fetch all of their bookmarks and put them in the bookmarks queue. Then something should go through that queue, do some parsing and then store the data in a database.
This obviously calls for threading because most of the time is spent waiting for Delicious to respond and then for bookmarked websites to respond and be passed through some API's and it would be silly for everything to wait for that.
However I am having trouble with threading and keep getting strange errors like database tables not being defined. Any help is appreciated :)
Here's the relevant code:
# relevant model #
class Bookmark(models.Model):
account = models.ForeignKey( Delicious )
url = models.CharField( max_length=4096 )
tags = models.TextField()
hash = models.CharField( max_length=32 )
meta = models.CharField( max_length=32 )
# bookmark queue reading #
def scrobble_bookmark(account):
try:
bookmark = Bookmark.objects.all()[0]
except Bookmark.DoesNotExist:
return False
bookmark.delete()
tags = bookmark.tags.split(' ')
user = bookmark.account.user
for concept in Concepts.extract( bookmark.url ):
for tag in tags:
Concepts.relate( user, concept['name'], tag )
return True
def scrobble_bookmarks(account):
semaphore = Semaphore(10)
for i in xrange(Bookmark.objects.count()):
thread = Bookmark_scrobble(account, semaphore)
thread.start()
class Bookmark_scrobble(Thread):
def __init__(self, account, semaphore):
Thread.__init__(self)
self.account = account
self.semaphore = semaphore
def run(self):
self.semaphore.acquire()
try:
scrobble_bookmark(self.account)
finally:
self.semaphore.release()
This is the error I get:
Exception in thread Thread-65:
Traceback (most recent call last):
File "/usr/lib/python2.6/threading.py", line 525, in __bootstrap_inner
self.run()
File "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 60, in run
scrobble_bookmark(self.account)
File "/home/swizec/Documents/trees/bookmarklet_server/../bookmarklet_server/Scrobbler/Scrobbler.py", line 28, in scrobble_bookmark
bookmark = Bookmark.objects.all()[0]
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 152, in __getitem__
return list(qs)[0]
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 76, in __len__
self._result_cache.extend(list(self._iter))
File "/usr/local/lib/python2.6/dist-packages/django/db/models/query.py", line 231, in iterator
for row in self.query.results_iter():
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 281, in results_iter
for rows in self.execute_sql(MULTI):
File "/usr/local/lib/python2.6/dist-packages/django/db/models/sql/query.py", line 2373, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python2.6/dist-packages/django/db/backends/sqlite3/base.py", line 193, in execute
return Database.Cursor.execute(self, query, params)
OperationalError: no such table: Scrobbler_bookmark
PS: all other tests depending on the same table pass with flying colours.

You can not use threading with in memory databases (sqlite3 in this case) in Django see this bug. This might work with PostgreSQL or MySQL.
I'd recommend something like celeryd instead of threads, message queues are MUCH easier to work with than threading.

This calls for a task queue, though not necessarily threads. You'll have your server process, one or several scrobbler processes, and a queue that lets them communicate. The queue can be in the database, or something separate like a beanstalkd. All this has nothing to do with your error, which sounds like your database is just misconfigured.

1) Does the error persist if you use a real database, not SQLite?
2) If you're using threads, you might need to create separate SQL cursors for use in threads.

i think the table really doesn't exists yew have to create it first by SQL commad
or any other way. As i have a small database for testing different modules i just delete
the database and recreate it using syncdb command