I need to schedule tasks via Pythonanywhere, to update some records in my database.
As an example, I do have Events as models:
models.py
class Events(models.Model):
event_title = models.CharField(max_length=300)
event_info = models.TextField(max_length=2000)
event_date = models.DateField(default=now)
event_time = models.TimeField(default='00:00')
event_location = models.CharField(max_length=150)
event_status = models.CharField(max_length=30, choices=EVENT_STATUS_CHOICES, default='New Event')
Also within that Events class in models.py I have the following:
def save(self, *args, **kwargs):
if date.today() > self.event_date:
self.event_status = 'Completed Event'
super(Events, self).save(*args, **kwargs)
Thanks to that, all events in the past, change their status to Completed. Problem is, you need to refresh the page to have the save function run on the production website.
It also drives some other functions, e.g. showing the user the count of active events - this one as well, in order to show the right active number of events, has to be refreshed manually.
By scheduling tasks in PA to run overnight, iterating through all Events, and if in the past, changing their status.
I created the below file and uploaded it to PA Tasks.
tasks.py
from .models import Events
from datetime import date
def verify_events():
all_events = Events.objects.all()
for event in all_events:
if event.event_date < date.today():
event.event_status = 'Completed Event'
event.save()
I got an error from PA when the task run:
Traceback (most recent call last):
File "/home/xxxxx/XXXXXXX/xxxxxx/tasks.py", line 1, in <module>
from .models import Events
ImportError: attempted relative import with no known parent package
2022-10-24 23:30:30 -- Completed task, took 14.35 seconds, return code was 1.
Looks like tasks.py could not read it from the database (->import from Models)
What's the best way to set up PA tasks to run and write to the database?
EDIT
This is indeed an issue with the working directory while on PythonAnywhere.
https://help.pythonanywhere.com/pages/ScheduledTasks/
I did add the below to the tasks.py but still the same error (ModuleNotFoundError: No module named XXX):
import os
path = "/home/myusername/myprojectname/mainfolder/"
os.chdir(path)
Folder structure - both tasks.py and models.py are in the same folder:
myprojectname/mainfolder/tasks.py
myprojectname/mainfolder/models.py
__init__.py is there.
Still not working.
So, while on PA, should I change some settings of the working directory? Or how to fix it, so I can import models (i.e. Events from db) and update using Tasks?
ImportError has nothing to do with the database, it's a Python error raised on your line 1. Its description is pretty accurate: "attempted relative import with no known parent package"
Related
I'm trying to make a celery task that would send a basic reminder to our users. So in our automated communication project, we have these tasks:
As you can see there are few actions that are different. So for now I have created a logic that fetches all the users from the DB and then continues by checking the time difference. But for now, I only have set-up for 2 hours or more. How should I use it correctly? I do not want to re-write each if statement because it's bad practice. How should I make it clear and reduce the system load?
#app.task
def check_registered_users():
from apps.users.models import User
from apps.notifications.models import AutomatedCommunicationNotifications
day_start = datetime.utcnow().date()
day_end = day_start + timedelta(days=1)
users = User.objects.filter(is_active=True, date_joined__range=(day_start, day_end))
users_that_received_notification = AutomatedCommunicationNotifications.objects.all().values('user__id')
excluded_users = users.exclude(id__in=users_that_received_notification)
for user in excluded_users:
if user.last_login < user.last_login + timedelta(hours=2):
# Sign-up uncompleted Push notification 2 hours after last login
template = SiteConfiguration.get_solo().automated_comms_signup_uncompleted
send_plain_email_task(
email=user.email,
subject=template.subject,
html_message=template.content,
from_email=f'{settings.EMAIL_FROM_PREFIX} <{settings.DEFAULT_FROM_EMAIL}>',
)
P.S AutomatedCommunicationNotifications table is for us to track which user has already received a notification.
class AutomatedCommunicationNotifications(BaseModel):
""" Model for automated comms notifications """
user = models.ForeignKey(User, on_delete=models.CASCADE)
type = models.CharField(
max_length=255,
choices=NotificationTypes.get_choices(),
default=NotificationTypes.EMAIL_NOTIFICATION
)
def __str__(self):
return str(self.user.phone)
You'll have to iterate over your queried users at least once but here are tips that may help:
models.py
class User(...):
# add a field to determine if the user has registered or not
# set this to `True` when a User successfully registers:
is_registered = models.BooleanField(default=False)
class AutomatedCommunicationNotifications(BaseModel):
# add a related name field for easier coding:
user = models.ForeignKey(..., related_name = 'notifications')
tasks.py
# load packages outside of your function so this only runs once on startup:
from django.models import F
from apps.users.models import User
from apps.notifications.models import AutomatedCommunicationNotifications
#app.task
def check_registered_users():
# timestamps:
two_hours_ago = datetime.now() - timedelta(hours=2)
# query for unregistered users who have not received a notification:
users = User.objects.filter(
is_registered = False,
last_login__lt = two_hours_ago # last logged in 2 or more hours ago
).exclude(
notifications__type = "the type"
).prefetch_related(
'notifications' # prejoins tables to improve performance
)
for user in users:
# send email
...
I would do this with a cron job. You can let it run whenever you want, depends on how fast after your give time frame you want to sent this.
You start with making a folder in your app:
/django/yourapp/management/commands
There you make a python file which contains your logic. Make sure to import the right modules from your views.
from django.core.management.base import BaseCommand, CommandError
from yourapp.models import every, module, you, need
from django.utils import timezone
from datetime import datetime, date, timedelta
from django.core.mail import send_mail, EmailMessage
class Command(BaseCommand):
help = 'YOUR HELP TEXT FOR INTERNAL USE'
def handle(self, *args, **options):
# Your logic
I added the crontab to the www-data users crontab like this:
# m h dom mon dow command
45 3 * * * /websites/vaccinatieplanner/venv/bin/python /websites/vaccinatieplanner/manage.py reminder
You can use that crontab file to tweak your optimal time between checks. If you remove the 3 and replace it by a * then you will have it check every 45 mins.
In my Django application, I want to delete a user's media file(their profile picture and other images) after 4-5 days when the user deletes its account.
def delete_files(sender, instance, **kwargs):
path = str(os.getcwd())
try:
pathdl = f"{path}\\data\\media\\{instance.username}"
shutil.rmtree(pathdl)
except Exception:
print(Exception)
post_delete.connect(delete_files, sender=User)
I used post_delete to delete the files of the user, but how can I delete the file after 4-5 days or a certain time period.
It would be good to use django-celery-beat for periodical tasks:
http://docs.celeryproject.org/en/latest/userguide/periodic-tasks.html#beat-custom-schedulers
Use this as an example
Consider this as your users models.py. What you would need here is an expiration field that will be checked by the cronjob before deleting it.
models.py
class Foo(models.model):
UserId= models.CharField(max_length=40, unique=True) #user pk here
expiration_date = models.DateTimeField() # you would set the time here
views.py
import datetime
from django.utils import timezone
def add_foo(instance):
# Create an instance of foo with expiration date now + one day
objects.create(expiration_date=timezone.now() + datetime.timedelta(days=1))
path = str(os.getcwd())
try:
pathdl = f"{path}\\data\\media\\{instance.username}"
shutil.rmtree(pathdl)
User.objects.create(expiration_date=timezone.now() + datetime.timedelta(days=1))
except Exception:
print(Exception)
post_delete.connect(delete_files, sender=User)
tasks.py
from celery.schedules import crontab
from celery.task import periodic_task
from django.utils import timezone
#periodic_task(run_every=crontab(minute='*/5'))
def delete_old_foos():
# Query all the expired date in our database
userMedia = Users.objects.all()
#Or get a specific user id to delete their file
# Iterate through them
for file in userMedia :
# If the expiration date is bigger than now delete it
if file.expiration_date < timezone.now():
file.delete()
# log deletion
return "completed deleting file at {}".format(timezone.now())
You may also of-course incorporate this idea into any way you want to solve this problem.
This is a curious one for Django+Algolia. I'm using the Algolia specific Django package:
$ pip install algoliasearch-django
I have the following model schema:
import os
import datetime
from channels import Group
from django.db import models
from django.conf import settings
from django.utils.six import python_2_unicode_compatible
from django.utils.translation import ugettext_lazy as _
from django.core.files.storage import FileSystemStorage
from django.contrib.humanize.templatetags.humanize import naturaltime
BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))
SITE_UPLOAD_LOC = FileSystemStorage(location=os.path.join(BASE_DIR, 'uploads/site'))
USER_UPLOAD_LOC = FileSystemStorage(location=os.path.join(BASE_DIR, 'uploads/user'))
#python_2_unicode_compatible
class Room(models.Model):
"""
This model class sets up the room that people can chat within - much like a forum topic.
"""
title = models.CharField(max_length=255)
staff = models.BooleanField(default=False)
slug = models.SlugField(max_length=250, default='')
banner = models.ImageField(storage=USER_UPLOAD_LOC, null=True, blank=True)
def last_activity(self):
"""
For date and time values show how many seconds, minutes, or hours ago a message
was sent (i.e., persised to the database) compared to current timestamp return representing string.
"""
last_persisted_message = Messages.objects.filter(where=self.slug).order_by('-sent_at').first()
if last_persisted_message is not None:
# First we can store "last persisted message" time in ISO format (could be useful for sitemap.xml generation; SEO tasks etc)
last_persisted_message_iso = last_persisted_message.sent_at.isoformat()
# Use the natural time package form contrib.humanize to convert our datetime to a string.
last_persisted_message = naturaltime(last_persisted_message.sent_at)
return last_persisted_message
else:
return "No activity to report"
Which is indexed as:
from algoliasearch_django import AlgoliaIndex
class RoomIndex(AlgoliaIndex):
fields = ('title', 'last_activity')
settings = {
'searchableAttributes': ['title'],
'attributesForFaceting': ['title', 'last_activity'],
'hitsPerPage': 15,
}
index_name = 'Room Index'
Essentially, to bring the 'last_activity' value to the front end it needs to pass through the index which is updated as far as I can tell with running:
$ python manage.py algolia_reindex
However, the last activity comes from the last time (converted to humanized django naturaltime, e.g. '3 days ago' etc etc) a Message was sent within a websocket connection - persisted to the database. All of this functionality works except that to update I need to run the algolia_reindex command.
Rather unsure as to how this could potentially be done a little more simultaneously...?
Ok, so this one was slightly more complex as I was using websockets. When a message is sent and persisted to the database we can also do the following within the relevant "consumer" method (really, consumers.py is the websocket equivalent of the views.py file so I should have known this!)
The following lines of code worked:
client = algoliasearch.Client(settings.ALGOLIA['APPLICATION_ID'], settings.ALGOLIA['API_KEY'])
index = client.init_index('Room Index')
res = index.partial_update_objects([{"last_activity": naturaltime(datetime.datetime.now()), "objectID": your_object_id]}])
The trick for anyone listening would be to designate the your_object_id from what value of the message is passed in from the client side to the consumer.
Don't forget to add:
import datetime
from django.conf import settings
from django.contrib.humanize.templatetags.humanize import naturaltime
At the top of the consumers.py file!
I also found the python specific incremental updates documentation from Algolia extremely useful:
I https://www.algolia.com/doc/tutorials/indexing/synchronization/incremental-updates/
To render the updated time in "real time" - use which ever front-end tool floats your boat, I used jQuery but Vue.js or React.js would work equally well.
I've had some trouble trying to reset my cache every hour for a particular django view.
Right now, I am using the cache_page decorator to cache my view using Memcached. But the cache expires after a while and the request is uncached from some users.
#cache_page(3600)
def my_view(request):
...
How do I write my own django manage.py command to refresh my cache for this view every hour from cron?
In other words, what do I put in my refresh_cache.py file mentioned in the answer here:
Django caching - can it be done pre-emptively?
Thanks!
In your app, you can create a folder called management which contains another folder commands and an empty __init__.py file. Inside commands you create another __init__.py and a file where you write your custom command. Let's called it refresh.py:
# refresh.py
from django.core.management.base import BaseCommand, CommandError
from main.models import * # You may want to import your models in order to use
# them in your cron job.
class Command(BaseCommand):
help = 'Posts popular threads'
def handle(self, *args, **options):
# Code to refresh cache
Now you can add this file to your cron jobs. You can take a look at this tutorial but basically you use crontab -e to edit your cron jobs and crontab -l to see which cron jobs are running.
You can find all of this and more in the Django documentation.
I want to expand on Roberts answer to fill out the # Code to refresh cache
Timezome+location make caches much hard to work with, in my case I just disabled them, also I am not sure about using the test methods in the application code, but it seems to work nicely.
from django.core.management.base import BaseCommand, CommandError
from django.test.client import RequestFactory
from django.conf import settings
from ladder.models import Season
from ladder.views import season as season_view
class Command(BaseCommand):
help = 'Refreshes cached pages'
def handle(self, *args, **options):
"""
Script that calls all season pages to refresh the cache on them if it has expired.
Any time/date settings create postfixes on the caching key, for now the solution is to disable them.
"""
if settings.USE_I18N:
raise CommandError('USE_I18N in settings must be disabled.')
if settings.USE_L10N:
raise CommandError('USE_L10N in settings must be disabled.')
if settings.USE_TZ:
raise CommandError('USE_TZ in settings must be disabled.')
self.factory = RequestFactory()
seasons = Season.objects.all()
for season in seasons:
path = '/' + season.end_date.year.__str__() + '/round/' + season.season_round.__str__() + '/'
# use the request factory to generate the correct url for the cache hash
request = self.factory.get(path)
season_view(request, season.end_date.year, season.season_round)
self.stdout.write('Successfully refreshed: %s' % path)
I would like to create a simple dynamic Sudoku game. Idea is to create new "puzzle" every hour then put it to database and let users solve it. Each solve attempt is compared with database for verification. For that purpose I would like to create python script that generates puzzle and puts it to the database. My database set in models looks like this:
from django.db import models
class user(models.Model):
name = models.CharField(max_length=30)
password = models.CharField(max_length=30)
time_registered=models.DateTimeField()
time_uploaded=models.DateTimeField()
points=models.IntegerField()
saved_sudoku=models.CommaSeparatedIntegerField(max_length=81)
solved=models.BooleanField()
def __str__(self):
return self.name
class server_sudoku(models.Model):
time_uploaded=models.DateTimeField()
generated_sudoku=models.CommaSeparatedIntegerField(max_length=81)
Now, when I use :
name1=request.POST["name"]
pass1=request.POST["password"]
newuser=user(name=name1,password=pass1,time_registered=datetime.datetime.now(),time_uploaded=datetime.datetime.now(),points=0,saved_sudoku="",solved=False)
newuser.save()
in views.py it creates new user. So to verify my idea I created application "generate_sudoku.py". To test its connection to database I just try to add user. Code looks as follows:
#!/usr/bin/env python
from db_interface.models import user
import random
import datetime
newuser=user(name="name", password="pass", time_registered=datetime.datetime.now() ,time_uploaded=datetime.datetime.now(), points=0, saved_sudoku="", solved=False)
newuser.save()
This simple app gives me this error:
raise ImportError("Settings cannot be imported, because environment variable %s is undefined." % ENVIRONMENT_VARIABLE)
ImportError: Settings cannot be imported, because environment variable DJANGO_SETTINGS_MODULE is undefined.
Hope I made it clear, I would like to run this application by windows scheduler so that it is automatically run every hour...
Use a custom manage.py command.
First link on google : http://eliasbland.wordpress.com/2010/01/25/importerror-settings-cannot-be-imported-because-environment-variable-django_settings_module-is-undefined/ ;)
This works for me (in a lambda script, not _ _init _ _.py file) :
import os
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
from django.contrib.auth.models import User #import django stuff after
print User.objects.all()