I thought there was an easy answer to this in recent versions of Django but I can't find it.
I have code that touches the database. I want it to run every time Django starts up. I seem to have two options:
Option 1. AppConfig.ready() - this works but also runs before database tables have been created (i.e. during tests or when reinitializing the app without data). If I use this I have to catch multiple types of Exceptions and guess that the cause is an empty db:
def is_db_init_error(e, table_name):
return ("{}' doesn't exist".format(table_name) in str(e) or
"no such table: {}".format(table_name) in str(e)
)
try:
# doing stuff
except Exception as e:
if not is_db_init_error(e, 'foo'):
raise
else:
logger.warn("Skipping updating Foo object as db table doesn't exist")
Option 2. Use post_migrate.connect(foo_init, sender=self) - but this only runs when I do a migration.
Option 3. The old way - call it from urls.py - I wanted to keep stuff like this out of urls.py and I thought AppConfig was the one true path
I've settled for option 2 so far - I don't like the smelly try/except stuff in Option 1 and Option 3 bugs me as urls.py becomes a dumping ground.
However Option 2 often trips me up when I'm developing locally - I need to remember to run migrations whenever I want my init code to run. Things like pulling down a production db or similar often cause problems because migrations aren't triggered.
I would suggest the connection_created signal, which is:
Sent when the database wrapper makes the initial connection to the
database. This is particularly useful if you’d like to send any post
connection commands to the SQL backend.
So it will execute the signal's code when the app connects to the database at the start of the application's cycle.
It will also work within a multiple database configuration and even separate the connections made by the app at initialization:
connection
The database connection that was opened. This can be used in a multiple-database configuration to differentiate connection signals
from different databases.
Note:
You may want to consider using a combination of post_migrate and connection_created signals while checking inside your AppConfig.ready() if a migration happened (ex. flag the activation of a post_migrate signal):
from django.apps import AppConfig
from django.db.models.signals import post_migrate, connection_created
# OR for Django 2.0+
# django.db.backends.signals import post_migrate, connection_created
migration_happened = false
def post_migration_callback(sender, **kwargs):
...
migration_happened = true
def init_my_app(sender, connection):
...
class MyAppConfig(AppConfig):
...
def ready(self):
post_migrate.connect(post_migration_callback, sender=self)
if !migration_happened:
connection_created.connect(init_my_app, sender=self)
In Django >= 3.2 the post_migrate signal is sent even if no migrations are run, so you can use it to run startup code that talks to the database.
https://docs.djangoproject.com/en/3.2/ref/signals/#post-migrate
Sent at the end of the migrate (even if no migrations are run) and flush commands. It’s not emitted for applications that lack a models module.
Handlers of this signal must not perform database schema alterations as doing so may cause the flush command to fail if it runs during the migrate command.
Related
I'm using a django third party app called django-solo to give me a SingletonModel that I can use for some global project settings, since I don't need multiple objects to represent these settings.
This works great, but on a fresh database, I need to go in and create an instance of this model, or it won't show up in Django Admin.
How can I make django automatically make sure that when django is starting up, when it connects to the database, it creates this?
I tried using the following code that I got here in my settings_app/apps.py, but it doesn't seem to fire at any point:
from django.db.backends.signals import connection_created
def init_my_app(sender, connection, **kwargs):
from .models import MyGlobalSettings
# This creates an instance of MyGlobalSettings if it doesn't exist
MyGlobalSettings.get_solo()
print("Instance of MyGlobalSettings created...")
class SettingsAppConfig(AppConfig):
...
def ready(self):
connection_created.connect(init_my_app, sender=self)
The instance isn't created, and I don't see my print statement in the logs. Am I doing something wrong?
The sample code has something about using post_migrate as well, but I don't need any special code to run after a migration, so I'm not sure that I need that.
Update:
My INSTALLED_APPS looks like this:
INSTALLED_APPS = [
...
'settings_app.apps.SettingsAppConfig',
'solo', # This is for django-solo
]
Also note that the ready() method does run. If I add a print statement to it, I do see it in the logs. It's just my init_my_app() function that doesn't run.
I think it's likely that the database connection is initialised before your app's ready() method runs - so attaching to the signal at that point is too late.
You don't need to attach to the signal there anyway - just do the check directly in the ready() method:
def ready(self):
from .models import MyGlobalSettings
MyGlobalSettings.get_solo()
Note that there is potential to run into other issues here - e.g., you will get an error if migrations for the MyGlobalSettings model haven't yet been applied on the database (when running manage.py migrate for the first time, for example) - you will probably need to catch specific database exceptions and skip the creation of this object in such cases.
I have a Flask web application that uses SQLAlchemy to access a PostgreSQL database.
When I start the application, there is instantly created an " in transaction" connection in PostgreSQL.
When the application has been used for some time, several of these connections appear in pg_stat_activity.
After some time, it appears that deadlocks occurs on some resources, and I have to restart the application to get it working again.
I have read that this can happen, if I return from a view function that uses database before closing the db session. So in order to avoid this problem, I have created the following decorator:
#app.teardown_appcontext
def shotdown_session(exception=None):
db.session.remove()
This should cause all sessions to be closed after each request and effectively avoid the problem of having " in transaction" connections.
Unfortunately, it does not seem to have any effect.
So, how do I really solve this problem?
UPDATE:
I should probably add, that I have verified that my decorator function is actually run.
I verified this by adding a print to it:
#app.teardown_appcontext
def shotdown_session(exception=None):
print "#app.teardown_appcontext: shotdown_session()"
db.session.remove()
I have also verified that it is indeed run AFTER return of the view function by adding a print to the view function as well:
[...]
products = db.session.query(...).all()
print "BEFORE RETURN"
return render_template("show_products.html", products=products)
This produces log lines like these:
* Running on http://0.0.0.0:5000/
* Restarting with reloader
BEFORE RETURN
#app.teardown_appcontext: shotdown_session()
10.0.0.100 - - [03/Dec/2014 13:41:30] "GET /product/51 HTTP/1.1" 200 -
I also went through the code and added a db.session.remove() call before each return in each function using db.session.
This does get rid of the in transaction, however, it also causes problems. I pass SQLAlchemy model objects from the database along to the templates. Some templates then do some operations on these model objects that causes the application to fail since the object is no longer attached to a session.
EDIT 2014-12-08:
Connection set up can be seen here:
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker, scoped_session
from flask_sqlalchemy import SQLAlchemy
from config import cfg
engine = create_engine(cfg["db_uri"], echo=False, pool_size=10)
db = SQLAlchemy()
Base = db.Model
Session = scoped_session(sessionmaker(bind=engine))
The structure of the entire application can be found here: http://paste.yt/p3219.html
I've seen this situation occur when you run Flask in Debug mode. If your code throws an exception and the debugger kicks in, the transaction will never get "rolled back" or "removed". As a result, the session that was used on the request that failed never gets returned to the pool.
The solution is to disable debug mode.
EDIT:
There's another circumstance where I've seen this happen. If you have code that runs autonomously (i.e. not part of an HTTP transaction – like an independent thread started and spawned off at launch of the Flask app), it will usually involve a sleep. If you access the session before the sleep, then you'll end up with a hung transaction like this during the sleep.
Another possibility is you are accessing a session from the create app function. If you do so, make sure to .remove() it. Otherwise, that session could remain hung on the main thread in a gevent app.
from sqlalchemy.pool import NullPool
use the NullPoll as poolclass solved the problem for me. Not sure why.
EDIT(Mar 23 2021):
Despite I got downvote, If using uWSGI, this probably the only choice. Check the sqlachemy official doc
https://docs.sqlalchemy.org/en/14/core/pooling.html#using-connection-pools-with-multiprocessing-or-os-fork
In my case I had async functions that started creating connections that would become idle or idle in transaction (based on whether I was using isolation_level=AUTOMATIC or not). They would continue to pile up until the # of db connections would reach the pool_size limit. When I removed the async nature from these handlers, the transactions pooled properly and the connection space no longer clogged itself. ¯\_(ツ)_/¯
#user.route("/<string:user_id>", methods=["GET"])
# async def user_get(user_id): // doesn't work
def user_get(user_id):
user = db.session.query(User).filter_by(id=user_id).first()
return jsonify({"data": prl.to_dict()}), 200
I am trying to create a web application as a front end to another Python app. I have the user enter data into a form, and upon submitting, the idea is for the data to be saved in a database, and for the data to be passed to a thread object class. The thread is something that is strictly kicked-off based on a user action. My problem is that I can import threading, but cannot access threading.Thread. When the thread ends, it will update the server, so when the user views the job information, they'll see the results.
View:
#login_required(login_url='/login')
def createNetworkView(request):
if request.method == "POST":
# grab my variables from POST
job = models.MyJob()
# load my variables into MyJob object
job.save()
t = ProcessJobThread(job.id, my, various, POST, inputs, here)
t.start()
return HttpResponseRedirect("/viewJob?jobID=" + str(job.id))
else:
return HttpResponseRedirect("/")
My thread class:
import threading # this works
print "About to make thread object" # This works, I see this in the log
class CreateNetworkThread(threading.Thread): # failure here
def __init__(self, jobid, blah1, blah2, blah3):
threading.Thread.__init__(self)
def run(self):
doCoolStuff()
updateDB()
I get:
Exception Type: ImportError
Exception Value: cannot import name Thread
However, if I run python on the command line, I can import threading and also do from threading import Thread. What's the deal?
I have seen other things, like How to use thread in Django and Celery but that seemed overkill, and I don't see how that example could import threading and use threading.Thread, when I can't.
Thank you.
Edit: I'm using Django 1.4.1, Python 2.7.3, Ubuntu 12.10, SQLite for the DB, and I'm running the web application with ./manage.py runserver.
This was a silly issue I had. First, I had made a file called "threading.py" and someone suggest I delete it, which I did (or thought I did). The problem was because of me using Eclipse, the PyDev (Python) plugin for Eclipse only deleted the threading.py file I created, and hides the *.pyc file. I had a lingering threading.pyc file lingering around, even though PyDev has an option that I had enabled to delete orphaned .pyc files.
I have a myapp/management/__init__.py that is registering a post_syncdb handler like so:
from django.db.models import signals
from features import models as features
def create_features(app, created_models, verbosity, **kwargs):
print "Creating features!"
# Do stuff...
signals.post_syncdb.connect(create_features, sender=features)
I've verified the following:
Both features and myapp are in settings.INSTALLED_APPS
myapp.management is getting loaded prior to the syncdb running (verified via a print statement at the module level)
The features app is getting processed by syncdb, and it is emitting a post_syncdb signal (verified by examining syncdb's output with --verbosity=2.
I'm using the exact same idiom for another pair of apps, and that handler is called correctly. I've compared the two modules and found no relevant differences between the invocations.
However, myapp.management.create_features is never called. What am I missing?
try putting it in your models.py
Just came across the same issue, and the way I solved it was to remove the sender from function arguments and check for it inside the callback function.
from django.db.models import signals
from features import models as features
def create_features(app, created_models, verbosity, **kwargs):
print "Creating features!"
if app != features #this will work as it compares models module instances
return
# Do stuff...
signals.post_syncdb.connect(create_features)
That way you can keep them in your management module like Django docs suggest. I agree that it should work like you suggested. You could probably dig into the implementation of the Signal class in django.dispatch.
The point is in the sender. Your custom callback is called only if sender worked out. In my case the sender was db.models and it is not worked out if syncdb called not the first time, i.o. the synced models are exist in database. In the documents it is written, but do not putted a proper emphasis.
sender
The models module that was just installed. That is, if syncdb just installed an app called "foo.bar.myapp", sender will be the foo.bar.myapp.models module.
So my solution was drop the database and install my app again.
I'm trying to provide integration to my django application from subversion through the post commit hook.
I have a django test case (a subclass of unittest.TestCase) that (a) inserts a couple of records into a table, (b) spawns an svn commit, (c) svn commit runs a hook that uses my django model to look up info.
I'm using an sqlite3 db. The test is not using the :memory: db, it is using a real file. I have modified the django test code (for debugging this issue) to avoid deleting the test db when it is finished so I can inspect it.
The test code dumps model.MyModel.objects.all() and the records are there between (a) and (b).
When the hook fires at (c) it also dumps the model and there are no records. When I inspect the db manually after the test runs, there are no records.
Is there something going on in the django test framework that isn't commiting the records to the db file?
To clarify: (d) end the test case. Thus the svn commit hook is run before the test case terminates, and before any django db cleanup code should be run.
Extra info: I added a 15 second delay between (b) and (b) so that I could examine the db file manually in the middle of the test. The records aren't in the file.
Are you using Django trunk? Recent changes (Changeset 9756) run tests in a transaction which is then rolled back. Here's the check-in comment:
Fixed #8138 -- Changed
django.test.TestCase to rollback tests
(when the database supports it)
instead of flushing and reloading the
database. This can substantially
reduce the time it takes to run large
test suites.
The test framework is not saving the data to the database, the data is cleaned once the tests have finished.
I'm very late to the party on this, but I saw similar behavior in 2022 using Django 3.2 and Python 3.8 and lost hours trying to debug.
If you're seeing it too: check to see if you've installed and configured django-moderation. If so, you may need to approve any records you add in your setUp functions:
from django.test import TestCase
from myapp.models import MyModel
class ArbitraryTest(TestCase):
#classmethod
def setUpTestData(cls):
new_record = MyModel.objects.create(my_field="New Record")
new_record.save()
MyModel.moderated_object.fget(new_record).approve()
def test_function(self):
self.assertTrue(MyModel.objects.filter(my_field="New Record").count() > 0)