i have one httpTrigger where i have implemented cache we have a requirement where we have to update cache after 2 hr.
Solution 1:
we can expire the cache after 2 hour.. but we don't want to use this solution
Solution 2:
we want a function to get triggered (update_cache()) after every 2 hour.
I find out some library
But i am unable to get how i can implement this..
# i want to trigger this function every 2 hour
def trigger_scheduler_update():
logging.info("Hi i am scheduler and got triggered...")
schedule.every(2).hours.do(trigger_scheduler_update)
But the problem i am facing here is we have to write this kind of code.
# ref: https://www.geeksforgeeks.org/python-schedule-library/
while True:
# Checks whether a scheduled task
# is pending to run or not
schedule.run_pending()
time.sleep(1)
As its an infinite loop i can place it in http trigger is there a way i can implement a scheduler that run after 2 hr.
i don't know that can it be done using threading?
i found one more library but looks like it also won't work.
Your function is shut down after a period of time, unless you are on a premium plan. Even then you cannot guarantee your function keeps on running.
What cache are you referring to?
Note that you cannot do threading in azure functions and you shouldn't actually. Abandon the idea of refreshing the cache from your httpTrigger function and create a separate scheduleTriggered function to update the cache that your http function is using.
Related
I have a simple function app in Python v2. The plan is to process millions of images, but right I just want to make the scaffolding right, i.e. no image processing, just dummy data. So I have two functions:
process with an HTTP trigger #app.route, this inserts 3 random image URLs to the Azure Queue Storage,
process_image with a Queue trigger #app.queue_trigger, that processes one image URL from above (currently only logs the event).
I trigger the first one with curl request and as expected, I can see the invocation in the Azure portal in the function's invocation section and I can see the items in the Storage Explorer's queue.
But unexpectedly, I do not see any invocations for the second function, even though after a few seconds the items disappear from the images queue and end up in the images-poison queue. So this means that something did run with the queue items 5 times. I see the following warning in the application insights checking traces and exceptions:
Message has reached MaxDequeueCount of 5. Moving message to queue 'case-images-deduplication-poison'.
Can anyone help with what's going on? Here's the gist of the code.
If I was to guess, something else is hitting that storage queue, like your dev machine or another function, can you put logging into the second function? (sorry c# guy so I don't know the code for logging)
Have you checked the individual function metric, in the portal, Function App >> Functions >> Function name >> overview >> Total execution Count and expand to the relevant time period?
Do note that it take up to 5 minutes for executions to show but after that you'll see them in the metrics
So I am making a website, and something that required for part of the security is having a waiting period when trying to do something, for example trying to delete something, this would help incase someone's account was stolen and someone tried to ruin their account.
I'm already using SQLite so I'm going to create a table in there where scheduled events will be defined.
What I'm wondering is what is the best way to constantly check these scheduled events, it may also be important to note I want to check at least every hour. My immediate thought was creating a separate thread and running a function on there with a while loop in it which will constantly run a chunk of code with a time.sleep(3600) at the end of the function, like this:
def check_events(self):
while True:
# code
time.sleep(3600)
I'm not sure though if this is the most efficient way of doing it.
That function currently is inside my website code class hence the self, is that something I need to put on the outside or no?
I would either create a cron job on your server (which is the most straightforward)
or use a schedule module to schedule your task, see example:
import time
import schedule
from sharepoint_cleaner import main as cleaner
from sharepoint_uploader import main as uploader
from transfer_statistics import main as transfer_stats
schedule.every(1).hours.do(uploader)
schedule.every(1).hours.do(transfer_stats)
schedule.every().sunday.do(cleaner)
while True:
schedule.run_pending()
time.sleep(10)
https://github.com/ansys/automatic-installer/blob/4d59573f8623c838aadfd49c312eeaca964c6601/sharepoint/scheduler.py#L3
I am triggering a Step Function execution via a Python cell in a SageMaker Notebook, like this:
state_machine_arn = 'arn:aws:states:us-west-1:1234567891:stateMachine:alexanderMyPackageStateMachineE3411O13-A1vQWERTP9q9'
sfn = boto3.client('stepfunctions')
..
sfn.start_execution(**kwargs) # Non Blocking Call
run_arn = response['executionArn']
print(f"Started run {run_name}. ARN is {run_arn}.")
and then in order to check that the execution (which might take hours to complete depending on the input) has been completed, before I start doing some custom post-analysis on the results, I manually execute a cell with:
response = sfn.list_executions(
stateMachineArn=state_machine_arn,
maxResults=1
)
print(response)
where I can see from the output the status of the execution, e.g. 'status': 'RUNNING'.
How can I automate this, i.e. trigger the Step Function and continue the execution on my post-analysis custom logic only after the execution has finished? Is there for example a blocking call to start the execution, or a callback method I could use?
I can think of putting a sleep method, so that the Python Notebook cell would periodically call list_executions() and check the status, and only when the execution is completed, continue to rest of the code. I can statistically determine the sleep period, but I was wondering if there is a simpler/more accurate way.
PS: Related: How to avoid simultaneous execution in aws step function, however I would like to avoid creating any new AWS resource, just for this, I would like to do everything from within the Notebook.
PPS: I cannot make any change to MyPackage and the Step Function definition.
Based on the comments.
If no new resources are to be created (no CloudWatch Event rules, lambda functions) nor any changes to existing Step Function are allowed, then pooling iteratively list_executions would be the best solution.
AWS CLI and boto3 have implemented similar solutions (not for Step Functions), but for some other services. They are called waiters (e.g. ec2 waiters). So basically you would have to create your own waiter for Step Function, as AWS does not provide one for that. AWS uses 15 seconds sleep time from what I recall for its waiters.
I'm using AWS python API (boto3). My script starts a few instances and then waits for them to come up online, before proceeding doing stuff. I want the wait to timeout after a predefined period, but I can't find any API for that in Python. Any ideas? A snippet of my current code:
def waitForInstance(id):
runningWaiter = self.ec2c.get_waiter("instance_status_ok")
runningWaiter.wait(InstanceIds = [id])
instance = ec2resource.Instance(id)
return instance.state
I can certainly do something like running this piece of code in a separate thread and terminate it if needed, but I was wondering whether there is already a built in API in boto3 for that and I'm just missing it.
A waiter has a configuration associated with it which can be accessed (using your example above) as:
runningWaiter.config
One of the settings in this config is max_attempts which controls how many attempts will be tried before giving up. The default value is 40. You can change that value like this:
runningWaiter.config.max_attempts = 10
This isn't directly controlling a timeout as your question asked but will cause the waiter to give up earlier.
Why not check the instances status from time to time?
#code copy from boto3 doc
for status in ec2.meta.client.describe_instance_status()['InstanceStatuses']:
print(status)
refence : http://boto3.readthedocs.org/en/latest/guide/migrationec2.html
BTW, it is better to use tag naming for all the instances with a standard naming convention. Query any aws resources with its original ID is a maintenance nightmare.
You could put a sleep timer in your code. Sleep for x minutes, check it to see if it is finished and go back to sleep if not. After y number of attempts take some sort it action.
I want to check users' subscribed dates for certain period. And send mail to users whose subscription is finishing (ex. reminds two days).
I think the best way is using thread and timer to check dates. But I have no idea how to call this function. I don't want to make a separate program or shell. I want to combine this procedure to my django code. I tried to call this function in my settings.py file. But it seems it is not a good idea. It calls the function and creates thread every time I imported settings.
That's case for manage.py command called periodically from cron. Oficial doc about creating those commands. Here bit more helpful.
If you want something simpler then django-command-extensions has commands for managing django jobs.
if you need more then only this one asynchronous job have a look at celery.
using Django-cron is much easier and simple
EDIT: Added a tip
from django_cron import cronScheduler, Job
class sendMail(Job):
# period run every 300 seconds (5 minutes)
run_every = 300
def job(self):
# This will be executed every 5 minutes
datatuple = check_subscription_finishing()
send_mass_mail(datatuple)
//and just register it
cronScheduler.register(sendMail)