Is there a way to log DB queries with long execution time for my django app?
Here's a middleware that logs queries with execution time longer than 1 second:
from django.db import connection
import logging
LONG_QUERY_TIME_SEC = 1
class LongQueryLogMiddleware:
def process_response ( self, request, response ):
for q in connection.queries:
if float(q['time']) >= LONG_QUERY_TIME_SEC:
logging.warning("Found long query (%s sec): %s", q['time'], q['sql'])
return response
Notice that because connection.queries is process-wide, queries can be logged twice. This can be solved by running the following after at the end of the function:
from django.db import reset_queries
reset_queries()
Related
I am using an application factory to add views to my flask application like so :
(this is not my actual application factory, and has been shortened for the sake of brevity)
def create_app(config_name='default'):
app = Flask(__name__, template_folder="templates", static_folder='static')
admin_instance = Admin(app, name='Admin')
admin_instance.add_view(EntityAdmin(Entity, db.session))
My EntityAdmin class looks like this :
class EntityAdmin(ModelView):
column_filters = [
MyCustomFilter(column=None, name='Custom')
]
My custom filter looks like this :
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
entities = Entity.query.filter(Entity.active == True).all()
return [(entity.id, entity.name) for entity in entities]
The problem is that it seems that the get_options function is called when the app is instantiated, running a select query every time the create_app function gets called.
So if I update my database schema and run the flask db migrate command, I get an error because the new column I added does not exist when the select query is run. The query raises an error because my database schema is not in sync with the actual database.
Can I register my views only when an actual HTTP request is made ? How can I differentiate between a request and a command ?
You have one more problem with this filter: its options are created on the application instantiation so if your list of entities was changed during the application running it would still return the same list of options.
To fix both problems you don't need to postpone views registrations. You need the filter to get the list of options every time it is used.
This SO answer to the question "Resetting generator object in Python" describes a way to reuse a generator (in your case — a database query):
from flask import has_app_context
def get_entities():
# has_app_context is used to prevent database access
# when application is not ready yet
if has_app_context():
for entity in Entity.query.filter(Entity.active.is_(True)):
yield entity.id, entity.name
class ReloadingIterator:
def __init__(self, iterator_factory):
self.iterator_factory = iterator_factory
def __iter__(self):
return self.iterator_factory()
class MyCustomFilter(BaseSQLAFilter):
def get_options(self, view):
# This will return a generator which is
# reloaded every time it is used
return ReloadingIterator(get_entities)
The problem is that the query to the Entity table can be called multiple times during request. So I usually cache the result for a single request using Flask globals:
def get_entities():
if has_app_context():
if not hasattr(g, 'entities'):
query = Entity.query.filter(Entity.active.is_(True))
g.entities = [(entity.id, entity.name) for entity in query]
for entity_id, entity_name in g.entities:
yield entity_id, entity_name
The following static method (taken from flasky) is a way to populate the database with fake data, and I want to import it to Django.
models.py
class User(UserMixin, db.Model):
# ......
#staticmethod
def generate_fake(count=100):
from sqlalchemy.exc import IntegrityError
from random import seed
import forgery_py
seed()
for i in range(count):
u = User(email=forgery_py.internet.email_address(),
username=forgery_py.internet.user_name(True),
password=forgery_py.lorem_ipsum.word(),
confirmed=True,
name=forgery_py.name.full_name(),
location=forgery_py.address.city(),
about_me=forgery_py.lorem_ipsum.sentence(),
member_since=forgery_py.date.date(True))
db.session.add(u)
try:
db.session.commit()
except IntegrityError:
db.session.rollback()
The problem is, while I can do something like User.objects.create(...), I don't know how to rollback the database in case an IntegrityError happens (presumable due to a duplicate primary key).
By default, Django makes SQL queries in auto-commit mode, where every query is wrapped in a transaction, so you should not worry about an integrity error messing up your database. If an integrity error happens, Django won't insert the data.
To be extra-safe, you can wrap the creation code in a transaction, that in Django, is implemented as a python context handler:
from django.db import transaction
with transaction.atomic():
User.objects.create(...)
But that would be unnecessary! You can read more about Transactions and how Django handles them in this documentation page.
I need to make sure that an object that is read from the database and written back, cannot be modified in the meantime by another request/process.
Does transaction.atomic() guarantee that?
My tests so far tell me no. If there's nothing wrong with them what would be the right way to achieve atomic READS and WRITES?
My example that I have tested.
Put the Test class somewhere in your model. atomic_test.py and atomic_test2.py should be saved as management commands. Run python manage.py atomic_test first, then python manage.py atomic_test2. The second script does not block and its changes are lost.
models.py
class Test(models.Model):
value = models.IntegerField()
atomic_test.py
from django.core.management.base import NoArgsCommand
from django.db import transaction
from time import sleep
from core.models import Test
class Command(NoArgsCommand):
option_list = NoArgsCommand.option_list
def handle(self, **options):
Test.objects.all().delete()
t = Test(value=50)
t.save()
print '1 started'
with transaction.atomic():
t = Test.objects.all()[0]
sleep(10)
t.value = t.value + 10
t.save()
print '1 finished: %s' %Test.objects.all()[0].value
atomic_test2.py
from django.core.management.base import NoArgsCommand
from django.db import transaction
from time import sleep
from core.models import Test
class Command(NoArgsCommand):
option_list = NoArgsCommand.option_list
def handle(self, **options):
print '2 started'
with transaction.atomic():
t = Test.objects.all()[0]
t.value = t.value - 20
t.save()
print '2 finished: %s' %Test.objects.all()[0].value
Django's transaction.atomic() is a thin abstraction over the transaction facilities of the database. So its behavior really depends on the database layer, and is specific to the type of database and its settings. So to really understand how this works you need to read and understand the transaction documentation for your database. (In PostgreSQL, for example, the relevant documentation is Transaction Isolation and Explicit Locking).
As for your specific test case, the behavior you want can be achieved by using the select_for_update() method on a Django queryset (if your database supports it). Something like:
in atomic_test.py
with transaction.atomic():
t = Test.objects.filter(id=1).select_for_update()[0]
sleep(10)
t.value = t.value + 10
t.save()
in atomic_test2.py
with transaction.atomic():
t = Test.objects.filter(id=1).select_for_update()[0]
t.value = t.value - 20
t.save()
The second one should block until the first one finishes, and see the new value of 60.
Other options include using the SERIALIZABLE transaction isolation level or using a row lock, though Django doesn't provide a convenient API for doing those things.
It would be much better to update atomically on the database using the F function:
from django.db.models import F
Test.objects.filter(id=1).update(value=F("value") + 10)
(This generates SQL something like "UPDATE test_test SET value = value + 10 WHERE id = 1")
Hello thanks for reading. I am doing a quick site in Django and I have a very simple update statement in raw SQL I am making to my Postgres database. Something in here is making trouble:
from django.http import HttpResponse
from django.db import connection, transaction
def rsvp_update(request, rsvp_id, status):
cursor = connection.cursor()
cursor.execute("UPDATE public.rsvp SET status=%s WHERE rsvp_id = %s", [status, rsvp_id])
transaction.commit()
return HttpResponse('okay')
I am getting an error that says "TransactionManagementError at [URL]
This code isn't under transaction management". Any ideas?
You need to use the commit_manually decorator for the code where you manage transactions manually.
I've had some trouble trying to reset my cache every hour for a particular django view.
Right now, I am using the cache_page decorator to cache my view using Memcached. But the cache expires after a while and the request is uncached from some users.
#cache_page(3600)
def my_view(request):
...
How do I write my own django manage.py command to refresh my cache for this view every hour from cron?
In other words, what do I put in my refresh_cache.py file mentioned in the answer here:
Django caching - can it be done pre-emptively?
Thanks!
In your app, you can create a folder called management which contains another folder commands and an empty __init__.py file. Inside commands you create another __init__.py and a file where you write your custom command. Let's called it refresh.py:
# refresh.py
from django.core.management.base import BaseCommand, CommandError
from main.models import * # You may want to import your models in order to use
# them in your cron job.
class Command(BaseCommand):
help = 'Posts popular threads'
def handle(self, *args, **options):
# Code to refresh cache
Now you can add this file to your cron jobs. You can take a look at this tutorial but basically you use crontab -e to edit your cron jobs and crontab -l to see which cron jobs are running.
You can find all of this and more in the Django documentation.
I want to expand on Roberts answer to fill out the # Code to refresh cache
Timezome+location make caches much hard to work with, in my case I just disabled them, also I am not sure about using the test methods in the application code, but it seems to work nicely.
from django.core.management.base import BaseCommand, CommandError
from django.test.client import RequestFactory
from django.conf import settings
from ladder.models import Season
from ladder.views import season as season_view
class Command(BaseCommand):
help = 'Refreshes cached pages'
def handle(self, *args, **options):
"""
Script that calls all season pages to refresh the cache on them if it has expired.
Any time/date settings create postfixes on the caching key, for now the solution is to disable them.
"""
if settings.USE_I18N:
raise CommandError('USE_I18N in settings must be disabled.')
if settings.USE_L10N:
raise CommandError('USE_L10N in settings must be disabled.')
if settings.USE_TZ:
raise CommandError('USE_TZ in settings must be disabled.')
self.factory = RequestFactory()
seasons = Season.objects.all()
for season in seasons:
path = '/' + season.end_date.year.__str__() + '/round/' + season.season_round.__str__() + '/'
# use the request factory to generate the correct url for the cache hash
request = self.factory.get(path)
season_view(request, season.end_date.year, season.season_round)
self.stdout.write('Successfully refreshed: %s' % path)