We Keep Coding

How can I deal with a massive delete from Django Admin?

I'm working with Django 2.2.10.
I have a model called Site, and a model called Record.
Each record is associated with a single site (Foreign Key).
After my app runs for a few days/weeks/months, each site can have thousands of records associated with it. I use the database efficiently, so this isn't normally a problem.
In Django Admin, when I try to delete a site however, Django Admin tries to figure out every single associated object that will also be deleted, and because my ForeignKey uses on_delete=models.CASCADE, which is what I want, it tries to generate a page that lists thousands, possibly millions of records that will be deleted. Sometimes this succeeds, but takes a few seconds. Sometimes the browser just gives up waiting.
How can I have Django Admin not list every single record it intends to delete? Maybe just say something like "x number of records will be deleted" instead.
Update: Should I be overriding Django admin's delete_confirmation.html? It looks like the culprit might be this line:
<ul>{{ deleted_objects|unordered_list }}</ul>
Or is there an option somewhere that can be enabled to automatically not list every single object to be deleted, perhaps if the object count is over X number of objects?
Update 2: Removing the above line from delete_confirmation.html didn't help. I think it's the view that generates the deleted_objects variable that is taking too long. Not quite sure how to override a Django Admin view

Add this to your admin class, and than you can delete with this action without warning
actions = ["silent_delete"]
def silent_delete(self, request, queryset):
queryset.delete()
If you want to hide default delete action, add this to your admin class
def get_actions(self, request):
actions = super().get_actions(request)
if 'delete_selected' in actions:
del actions['delete_selected']
return actions

Since django 2.1 you can override get_deleted_objects to limit the amount of deleted objects listed (it's either a list or a nested list). The timeout is probably due to the django app server timing out on the view's response.
You could limit the size of the returned list:
class YourModelAdmin(django.admin.ModelAdmin):
def get_deleted_objects(self, objs, request):
deleted = super().get_deleted_objects(objs, request)
deleted_objs = deleted[0]
return (self.__limit_nested(deleted_objs),) + deleted[1:]
def __limit_nested(self, objs):
limit = 10
if isinstance(objs, list):
return list(map(self.__limit_nested, objs))
if len(objs) > limit:
return objs[:limit] + ['...']
return objs
But chances are the call to super takes too long as well, so you probably want to return [], {}, set(), [] instead of calling super; though it doesn't tell you about missing permissions or protected relations then (but I saw no alternative other than copy pasting code from django github). You will want to override the delete_confirmation.html and the delete_selected_confirmation.html template as well. You'll also want to make sure the admin has permission to delete any related objects that might get deleted by the cascading deletes.
In fact, the deletion itself may take too long. It's probably best defer the deletion (and permission checks if those are slow too) to a celery task.

Solution needed to a scenario

I am trying to make use of a column's value as a radio button's choice using below code
Forms.py
#retreiving data from database and assigning it to diction list
diction = polls_datum.objects.values_list('poll_choices', flat=True)
#initializing list and dictionary
OPTIONS1 = {}
OPTIONS = []
#creating the dictionary with 0 to no of options given in list
for i in range(len(diction)):
OPTIONS1[i] = diction[i]
#creating tuples from the dictionary above
#OPTIONS = zip(OPTIONS1.keys(), OPTIONS1.values())
for i in OPTIONS1:
k = (i,OPTIONS1[i])
OPTIONS.append(k)
class polls_form(forms.ModelForm):
#retreiving data from database and assigning it to diction list
options = forms.ChoiceField(choices=OPTIONS, widget = forms.RadioSelect())
class Meta:
model = polls_model
fields = ['options']
Using a form I am saving the data or choices in a field (poll_choices), when trying to display it on the index page, it is not reflecting until a server restart.
Can someone help on this please

of course "it is not reflecting until a server restart" - that's obvious when you remember that django server processes are long-running processes (it's not like PHP where each script is executed afresh on each request), and that top-level code (code that's at the module's top-level, not in a function) is only executed once per process when the module is first imported. As a general rule: don't do ANY db query at a module's top-level or at the top-level of a class statement - at best you'll get stale data, at worse it will crash your server process (if you're doing query before everything has been properly setup by django, or if you're doing query based on a schema update before the migration has been applied).
The possible solutions are either to wait until the form's initialisation to setup your field's choices, or to pass a callable as the formfield's choices options, cf https://docs.djangoproject.com/en/2.1/ref/forms/fields/#django.forms.ChoiceField.choices
Also, the way you're building your choices list is uselessly complicated - you could do it as a one-liner:
OPTIONS = list(enumerate(polls_datum.objects.values_list('poll_choices', flat=True))
but it's also very brittle - you're relying on the current db content and ordering for the choice value when you should use the polls_datum's pk instead (which is garanteed to be stable).
And finally: since you're working with what seems to be a related model, you may want to use a ModelChoiceField instead.

For future reference:
What version of Django are you using?
Have you read up on the documentation of ModelForms? https://docs.djangoproject.com/en/2.1/topics/forms/modelforms/
I'm not sure what you're trying to do with diction to dictionary to tuple. I think you could skip a step there and your future self will thank you for that.
Try to follow some tutorials and understand why certain steps are being taken. I can see from your code that you're rather new to coding or Python and there's room for improvement. Not trying to talk you down, but I'm trying to push you into the direction of becoming a better developer ;-)
REAL ANSWER:
That being said, I think the solution is to write the loading of the data somewhere in your form model, rather than 'loose' in forms.py. See bruno's answer for more information on this.
If you want to reload the data on each request that loads the form, you should create a function that gets called every time the form is loaded (for example in the form's __init__ function).

Exposing global data and functions in Pyramid and Jinja 2 templating

I have a base template for when a user is logged in, and on that base template, I need to add user specific options in a drop down menu. This drop down menu with options must be constant across all handlers, i.e., any time the base template is invoked (extended) with a child template.
Other than performing the necessary DB query, assigning the query results to a variable, and passing that variable to every handler (there are many), how can I consolidate this into one query and one variable, which gets passed directly to the base template? I am using jinja2 templates as well.
I would hate to do something so cumbersome in exchange for something far more simple and maintainable.
Any ideas? Thanks.
EDIT
So I still haven't found anything that's exactly what I'm looking for; however, I decided to at least make some headway in the interim. So, I made a custom decorator that takes a view's returned dict() and appends the appropriate data to it. For example:
def get_base_data(func):
def wrapper(request):
d = func(request)
user_id = request.user.id # used in query
contact_group_data = ContactGroups.query.filter(...criteria...).all()
d['contact_group_data'] = contact_group_data
return d
return wrapper
Now, I can at least decorate each method very concisely and simply by putting:
#view_config(...)
#get_base_data
def my_handler(request):
pass # rest of code...

This is one of most inobvious things in Pyramid and took a while to find for me, too.
You can modify the global template context in BeforeRender event.
http://docs.pylonsproject.org/projects/pyramid/en/latest/narr/hooks.html#using-the-before-render-event
Alternatively, you could use class based views, inherit all your views from one base view class which has get_base_data(), then the class instance is passed to the template context to all your views and then you could extract the data with {{ view.get_base_data }}.
http://ruslanspivak.com/2012/03/02/class-based-views-in-pyramid/
I vouch for the latter approach as it is more beautiful, predictable and easier to maintain engineering wise.

Clearing specific cache in Django

I am using view caching for a django project.
It says the cache uses the URL as the key, so I'm wondering how to clear the cache of one of the keys if a user updates/deletes the object.
An example: A user posts a blog post to domain.com/post/1234/ .. If the user edits that, i'd like to delete the cached version of that URL by adding some kind of delete cache command at the end of the view that saves the edited post.
I'm using:
#cache_page(60 * 60)
def post_page(....):
If the post.id is 1234, it seems like this might work, but it's not:
def edit_post(....):
# stuff that saves the edits
cache.delete('/post/%s/' % post.id)
return Http.....

From django cache docs, it says that cache.delete('key') should be enough. So, it comes to my mind two problems you might have:
Your imports are not correct, remember that you have to import cache from the django.core.cache module:
from django.core.cache import cache
# ...
cache.delete('my_url')
The key you're using is not correct (maybe it uses the full url, including "domain.com"). To check which is the exact url you can go into your shell:
$ ./manage.py shell
>>> from django.core.cache import cache
>>> cache.has_key('/post/1234/')
# this will return True or False, whether the key was found or not
# if False, keep trying until you find the correct key ...
>>> cache.has_key('domain.com/post/1234/') # including domain.com ?
>>> cache.has_key('www.domain.com/post/1234/') # including www.domain.com ?
>>> cache.has_key('/post/1234') # without the trailing / ?

I make a function to delete key starting with some text. This help me to delete dynamic keys.
list posts cached
def get_posts(tag, page=1):
cached_data = cache.get('list_posts_home_tag%s_page%s' % (tag, page))
if not cached_data:
cached_data = mycontroller.get_posts(tag, page)
cache.set('list_posts_home_tag%s_page%s' % (tag, page), cached_data, 60)
return cached_data
when update any post, call flush_cache
def update(data):
response = mycontroller.update(data)
flush_cache('list_posts_home')
return response
flush_cache to delete any dynamic cache
def flush_cache(text):
for key in list(cache._cache.keys()):
if text in key:
cache.delete(key.replace(':1:', ''))
Do not forget to import cache from django
from django.core.cache import cache

I had same problem and I found this solution.
if you are using this decorator:
django.views.decorators.cache.cache_page
you can use this function:
import hashlib
from typing import List
from django.core.cache import cache
def get_cache_keys_from_url(absolute_uri) -> List[str]:
url = hashlib.md5(f"absolute_uri".encode('ascii'))
return cache.keys(f'*{url.hexdigest()}*')
which you need to get cache keys for absolute_uri
and than use cache.delete_many(get_cache_keys_from_url(http://exemple.com/post/1234/)) for clear page with url - http://exemple.com/post/1234/

There's a trick that might work for this. The cache_page decorator takes an optional argument for key_prefix. It's supposed to be used when you have, say, multiple sites on a single cache. Here's what the docs say:
CACHE_MIDDLEWARE_KEY_PREFIX – If the cache is shared across multiple sites using the same Django installation, set this to the name of the site, or some other string that is unique to this Django instance, to prevent key collisions. Use an empty string if you don’t care.
But if you don't have multiple sites, you can abuse it thus:
cache_page(cache_length, key_prefx="20201215")
I used the date as the prefix so it's easy to rotate through new invalidation keys if you want. This should work nicely.
However! Please note that if you are using this trick, some caches (e.g., the DB cache, filesystem cache, and probably others) do not clean up expired entries except when they're accessed. If you use the trick above, you won't ever access the cache entry again and it'll linger until you clear it out. Probably doesn't matter, but worth considering.

Jinja2: Looking for a View-Helper

I'am new to the Jinja2 template engine. Is there something like the view-helpers from Zend Framework? Can i create simple functions and reuse them all over all my template-files?
Something like this?
#somewhere in my python code:
def nice_demo_function(message):
""""return a simple message"""
return message
So i can to use that:
<!-- now in my template-file -->
{% nice_demo_function('yes, this works great!') %}

There are a number of ways you can expose helper functions to your templates. You could define them using macros, and then import them into templates that use them. You could add functions to the globals attribute of your Template objects, or pass them to the render() method. You could subclass Template to do the same without having to repeat yourself each time. If you want to get really fancy, you could look into writing extensions as well (but you probably don't need to go that deep).

At some point you will have created a Jinja2 environment. The environment has an attribute on it called filters which is a dict that maps names to functions. So what you want to do is:
def my_helper(value):
return "-~*#--- %s ---#*~-" % value
env = Jinja2.Environment(...)
env.filters['my_helper'] = my_helper
Now in your template you can do:
<p>The winner is {{ winner | my_helper }}</p>
And your function will be called with the value of the variable, in this case winner. If you are using Pylons, this all happens in config/environment.py.