I'm adding automatic validation to one of my models in a pyramid application with before_insert.
So far I've got this:
def Property_before_insert_listener(mapper, connection, target):
formvalidator = PropertySchema()
try:
return formvalidator.to_python(target.__table__.columns)
except formencode.Invalid as error:
print ("***************************************ERROR" + str(error))
event.listen(
Property, 'before_insert', Property_before_insert_listener)
Everything seems to be working fine, and I get the proper error printed out in the console. However, after handling the error, it continues with the insert. How do I stop the insert from happening?
In the sqlAlquemy documentation for Mapper Events you have something that might help you:
retval=False
when True, the user-defined event function must have a return value, the purpose of which is either to control subsequent event propagation, or to otherwise alter the operation in progress by the mapper. Possible return > values are:
sqlalchemy.orm.interfaces.EXT_CONTINUE - continue event processing normally.
sqlalchemy.orm.interfaces.EXT_STOP - cancel all subsequent event handlers in the chain.
other values - the return value specified by specific listeners.
Related
I have a function that may or may not do multiple Redis queries.
Some of these queries need values returned from previous queries, so they are made without a pipeline:
if action == "comments":
if redis_connection.exists(key) == 0:
increment = Comments.objects.filter(user__uuid=uuid).count()
redis_connection.incrby(f"{uuid}_{comment_uuid}, increment)
While others may be called multiple times inside the function in a for loop, requiring a pipeline to optimize it:
if redis_connection.exists(user_comment_key) == 0:
reactions = CommentReactions.objects.filter(comment__uuid=comment_uuid, user__uuid=user_uuid)
for reaction in reactions:
with redis_connection.pipeline() as pipe:
pipe.sadd(user_comment_key, reaction)
This pipeline call is where I got confused. Since I'm calling it inside a context manager (with), will all other connections also be closed, or should I close every other single connection separately?
Background
In my code below I have a function called process that does some stuff and when it is running i want to make sure it is not run concurrently. I have a table called creation_status where i set a time stamp anytime i start the process. The reason i use a time stamp is because it allows me to know what time i started this process in case i need to.
I always check if there is already a time stamp and if there is raise an exception to make sure i am not running this script concurrently.
Code
def is_in_process() -> bool:
status = db.run_query(sql="SELECT is_in_process FROM creation_status")
return False if status[0].is_in_process is None else True
def set_status() -> None:
db.execute(sql="UPDATE creation_status SET is_in_process = NOW()")
def delete_status() -> None:
db.execute(sql="UPDATE creation_status SET is_in_process = NULL")
def process():
if is_in_process():
raise Exception("Must not run concurrent process creations." )
set_status()
# stuff happens
delete_status()
Issue
I want to make sure my query is atomic to eliminate race conditions. It is possible that the by the time i check the function is_in_process and call the function set_status another script could get kicked off. How do i ensure both those things happen in one go so i avoid race conditions.
Please let me know if i can explain something more clear and i am open to all suggestions.
Don't use multiple steps when you don't need to.
UPDATE creation_status SET is_in_process = NOW() where is_in_process is null returning is_in_process
Then check to see if a row is returned.
Of course this should probably be done in the same transaction as the rest of the stuff, which we can't tell from your code, but then things will just block until the previous is done, rather than aborting.
I'm running a python script on a raspberry pi that constantly checks on a Yocto button and when it gets pressed it puts data from a different sensor in a database.
a code snippet of what constantly runs is:
#when all set and done run the program
Active = True
while Active:
if ResponseType == "b":
while Active:
try:
if GetButtonPressed(ResponseValue):
DoAllSensors()
time.sleep(5)
else:
time.sleep(0.5)
except KeyboardInterrupt:
Active = False
except Exception, e:
print str(e)
print "exeption raised continueing after 10seconds"
time.sleep(10)
the GetButtonPressed(ResponseValue) looks like the following:
def GetButtonPressed(number):
global buttons
if ModuleCheck():
if buttons[number - 1].get_calibratedValue() < 300:
return True
else:
print "module not online"
return False
def ModuleCheck():
global moduleb
return moduleb.isOnline()
I'm not quite sure about what might be going wrong. But it takes about an hour before the RPI runs out of memory.
The memory increases in size constantly and the button is only pressed once every 15 minutes or so.
That already tells me that the problem must be in the code displayed above.
The problem is that the yocto_api.YAPI object will continue to accumulate _Event objects in its _DataEvents dict (a class-wide attribute) until you call YAPI.YHandleEvents. If you're not using the API's callbacks, it's easy to think (I did, for hours) that you don't need to ever call this. The API docs aren't at all clear on the point:
If your program includes significant loops, you may want to include a call to this function to make sure that the library takes care of the information pushed by the modules on the communication channels. This is not strictly necessary, but it may improve the reactivity of the library for the following commands.
I did some playing around with API-level callbacks before I decided to periodically poll the sensors in my own code, and it's possible that some setting got left enabled in them that is causing these events to accumulate. If that's not the case, I can't imagine why they would say calling YHandleEvents is "not strictly necessary," unless they make ARM devices with unlimited RAM in Switzerland.
Here's the magic static method that thou shalt call periodically, no matter what. I'm doing so once every five seconds and that is taking care of the problem without loading down the system at all. API code that would accumulate unwanted events still smells to me, but it's time to move on.
#noinspection PyUnresolvedReferences
#staticmethod
def HandleEvents(errmsgRef=None):
"""
Maintains the device-to-library communication channel.
If your program includes significant loops, you may want to include
a call to this function to make sure that the library takes care of
the information pushed by the modules on the communication channels.
This is not strictly necessary, but it may improve the reactivity
of the library for the following commands.
This function may signal an error in case there is a communication problem
while contacting a module.
#param errmsg : a string passed by reference to receive any error message.
#return YAPI.SUCCESS when the call succeeds.
On failure, throws an exception or returns a negative error code.
"""
errBuffer = ctypes.create_string_buffer(YAPI.YOCTO_ERRMSG_LEN)
#noinspection PyUnresolvedReferences
res = YAPI._yapiHandleEvents(errBuffer)
if YAPI.YISERR(res):
if errmsgRef is not None:
#noinspection PyAttributeOutsideInit
errmsgRef.value = YByte2String(errBuffer.value)
return res
while len(YAPI._DataEvents) > 0:
YAPI.yapiLockFunctionCallBack(errmsgRef)
if not (len(YAPI._DataEvents)):
YAPI.yapiUnlockFunctionCallBack(errmsgRef)
break
ev = YAPI._DataEvents.pop(0)
YAPI.yapiUnlockFunctionCallBack(errmsgRef)
ev.invokeData()
return YAPI.SUCCESS
I'm using Django / Postgresql, and I want to build some kind of media player behavior : start playing when asked, play until further command, and stop playing when asked.
My Django model is designed as follows :
a boolean attribute state
a function that switches the state value, and if it's true, calls the looping function
a function that loops while the state attribute is True, and stops when it is False.
The code is as follows:
state = models.BooleanField(default=False)
def switchState(self):
print 'switching from %s to %s' % (self.state, not(self.state))
self.state = not(self.state)
self.save()
if self.state :
self.loop()
def loop(self):
while self.state:
print 'looping'
time.sleep( 1 )
print 'stopped looping'
Behavior :
when I call switchState() on a model instance, state is set to True (I can see it in the database) and the loop() function starts printing lines
when I call switchState() again, state is set to False (again, I can see it in the database) but then, the loop() function does not stop. When I print state, its value is still True...
I can't get an up to date value of that damned state attribute.
I must be missing something, but what ?
Thanks for your help !
OK I finally found an answer. As Daniel Roseman told me, the problem wad due to the loop function, which was looping in its own transaction, not being able to see what happened in another transaction.
I first tried to force a refresh on my instance, with the new refresh_from_db() function, but that didn't work at all (I still don't get why, please tell me if you know).
As Daniel suggested, I read again the Django transaction doc, and finally found out a solution that seems to work. The idea is to allow the loop function to update the instance at each loop, using the with transaction.atomic() command.
def loop(self):
vTest = Source.objects.get(id=self.id)
while vTest:
with transaction.atomic():
vTest = Source.objects.get(id=self.id).state
print 'looping'
time.sleep( 1 )
print 'stopped looping'
Nb : you can't use
vTest = self.state
instead of
vTest = Source.objects.get(id=self.id).state
Maybe it's not the right way to do it, feel free to correct me if I'm wrong.
Is it possible to find out whether a task with a certain task id exists? When I try to get the status, I will always get pending.
>>> AsyncResult('...').status
'PENDING'
I want to know whether a given task id is a real celery task id and not a random string. I want different results depending on whether there is a valid task for a certain id.
There may have been a valid task in the past with the same id but the results may have been deleted from the backend.
Celery does not write a state when the task is sent, this is partly an optimization (see the documentation).
If you really need it, it's simple to add:
from celery import current_app
# `after_task_publish` is available in celery 3.1+
# for older versions use the deprecated `task_sent` signal
from celery.signals import after_task_publish
# when using celery versions older than 4.0, use body instead of headers
#after_task_publish.connect
def update_sent_state(sender=None, headers=None, **kwargs):
# the task may not exist if sent using `send_task` which
# sends tasks by name, so fall back to the default result backend
# if that is the case.
task = current_app.tasks.get(sender)
backend = task.backend if task else current_app.backend
backend.store_result(headers['id'], None, "SENT")
Then you can test for the PENDING state to detect that a task has not (seemingly)
been sent:
>>> result.state != "PENDING"
AsyncResult.state returns PENDING in case of unknown task ids.
PENDING
Task is waiting for execution or unknown. Any task id that is not
known is implied to be in the pending state.
http://docs.celeryproject.org/en/latest/userguide/tasks.html#pending
You can provide custom task ids if you need to distinguish unknown ids from existing ones:
>>> from tasks import add
>>> from celery.utils import uuid
>>> r = add.apply_async(args=[1, 2], task_id="celery-task-id-"+uuid())
>>> id = r.task_id
>>> id
'celery-task-id-b774c3f9-5280-4ebe-a770-14a6977090cd'
>>> if not "blubb".startswith("celery-task-id-"): print "Unknown task id"
...
Unknown task id
>>> if not id.startswith("celery-task-id-"): print "Unknown task id"
...
Right now I'm using following scheme:
Get task id.
Set to memcache key like 'task_%s' % task.id message 'Started'.
Pass task id to client.
Now from client I can monitor task status(set from task messages to memcache).
From task on ready - set to memcache key message 'Ready'.
From client on task ready - start special task that will delete key from memcache and do necessary cleaning actions.
You need to call .get() on the AsyncTask object you create to actually fetch the result from the backend.
See the Celery FAQ.
To further clarify on my answer.
Any string is technically a valid ID, there is no way to validate the task ID. The only way to find out if a task exists is to ask the backend if it knows about it and to do that you must use .get().
This introduces the problem that .get() blocks when the backend doesn't have any information about the task ID you supplied, this is by design to allow you to start a task and then wait for its completion.
In the case of the original question I'm going to assume that the OP wants to get the state of a previously completed task. To do that you can pass a very small timeout and catch timeout errors:
from celery.exceptions import TimeoutError
try:
# fetch the result from the backend
# your backend must be fast enough to return
# results within 100ms (0.1 seconds)
result = AsyncResult('blubb').get(timeout=0.1)
except TimeoutError:
result = None
if result:
print "Result exists; state=%s" % (result.state,)
else:
print "Result does not exist"
It should go without saying that this only work if your backend is storing results, if it's not there's no way to know if a task ID is valid or not because nothing is keeping a record of them.
Even more clarification.
What you want to do cannot be accomplished using the AMQP backend because it does not store results, it forwards them.
My suggestion would be to switch to a database backend so that the results are in a database that you can query outside of the existing celery modules. If no tasks exist in the result database you can assume the ID is invalid.
So I have this idea:
import project.celery_tasks as tasks
def task_exist(task_id):
found = False
# tasks is my imported task module from celery
# it is located under /project/project, where the settings.py file is located
i = tasks.app.control.inspect()
s = i.scheduled()
for e in s:
if task_id in s[e]:
found = True
break
a = i.active()
if not found:
for e in a:
if task_id in a[e]:
found = True
break
r = i.reserved()
if not found:
for e in r:
if task_id in r[e]:
found = True
break
# if checking the status returns pending, yet we found it in any queues... it means it exists...
# if it returns pending, yet we didn't find it on any of the queues... it doesn't exist
return found
According to https://docs.celeryproject.org/en/stable/userguide/monitoring.html the different types of queue inspections are:
active,
scheduled,
reserved,
revoked,
registered,
stats,
query_task,
so pick and choose as you please.
And there might be a better way to go about checking the queues for their tasks, but this should work for me, for now.
Try
AsyncResult('blubb').state
that may work.
It should return something different.
Please correct me if i'm wrong.
if built_in_status_check(task_id) == 'pending'
if registry_exists(task_id) == true
print 'Pending'
else
print 'Task does not exist'