We Keep Coding

How do I ensure that my Django singleton model exists on startup?

I'm using a django third party app called django-solo to give me a SingletonModel that I can use for some global project settings, since I don't need multiple objects to represent these settings.
This works great, but on a fresh database, I need to go in and create an instance of this model, or it won't show up in Django Admin.
How can I make django automatically make sure that when django is starting up, when it connects to the database, it creates this?
I tried using the following code that I got here in my settings_app/apps.py, but it doesn't seem to fire at any point:
from django.db.backends.signals import connection_created
def init_my_app(sender, connection, **kwargs):
from .models import MyGlobalSettings
# This creates an instance of MyGlobalSettings if it doesn't exist
MyGlobalSettings.get_solo()
print("Instance of MyGlobalSettings created...")
class SettingsAppConfig(AppConfig):
...
def ready(self):
connection_created.connect(init_my_app, sender=self)
The instance isn't created, and I don't see my print statement in the logs. Am I doing something wrong?
The sample code has something about using post_migrate as well, but I don't need any special code to run after a migration, so I'm not sure that I need that.
Update:
My INSTALLED_APPS looks like this:
INSTALLED_APPS = [
...
'settings_app.apps.SettingsAppConfig',
'solo', # This is for django-solo
]
Also note that the ready() method does run. If I add a print statement to it, I do see it in the logs. It's just my init_my_app() function that doesn't run.

I think it's likely that the database connection is initialised before your app's ready() method runs - so attaching to the signal at that point is too late.
You don't need to attach to the signal there anyway - just do the check directly in the ready() method:
def ready(self):
from .models import MyGlobalSettings
MyGlobalSettings.get_solo()
Note that there is potential to run into other issues here - e.g., you will get an error if migrations for the MyGlobalSettings model haven't yet been applied on the database (when running manage.py migrate for the first time, for example) - you will probably need to catch specific database exceptions and skip the creation of this object in such cases.

Python code causes SQL server transaction log to increase exponentially

I have a python script to execute a stored procedure to purge the tables in database. This SP further calls another SP which has delete statements for each table in database. Something like below -
Python calls - Stored procedure Purge_DB
Purge_DB calls - Stored procedure Purge_Table
Purge_Table has definition to delete data from each table.
When I run this python script, the transaction logs increase exponentially and on running this script 2-3 times, I get the transaction log full error.
Please note that the deletion happens in transaction.
BEGIN TRAN
EXEC (#DEL_SQL)
COMMIT TRAN
Earlier I was executing the same SP using VB script and never got any issue related to transaction log.
Is there a different way that Python uses to create transaction log?
Why is the log size much bigger with Python than VB script?

This is resolved now.
Python starts a transaction when execute method is called and that transaction remains open until we explicitly call commit() method. Since, this purge SP was called for more than 100 tables, the transaction log was populated until transaction was closed in the python code and hence, it was getting full because of this job.
I have set the autocommit property of pyodbc to true which will now automatically commit each SQL statement as and when it is executed as part of that connection. Please refer to the documentation here -
https://github.com/mkleehammer/pyodbc/wiki/Database-Transaction-Management

Django: running code on every startup but after database is migrated

I thought there was an easy answer to this in recent versions of Django but I can't find it.
I have code that touches the database. I want it to run every time Django starts up. I seem to have two options:
Option 1. AppConfig.ready() - this works but also runs before database tables have been created (i.e. during tests or when reinitializing the app without data). If I use this I have to catch multiple types of Exceptions and guess that the cause is an empty db:
def is_db_init_error(e, table_name):
return ("{}' doesn't exist".format(table_name) in str(e) or
"no such table: {}".format(table_name) in str(e)
)
try:
# doing stuff
except Exception as e:
if not is_db_init_error(e, 'foo'):
raise
else:
logger.warn("Skipping updating Foo object as db table doesn't exist")
Option 2. Use post_migrate.connect(foo_init, sender=self) - but this only runs when I do a migration.
Option 3. The old way - call it from urls.py - I wanted to keep stuff like this out of urls.py and I thought AppConfig was the one true path
I've settled for option 2 so far - I don't like the smelly try/except stuff in Option 1 and Option 3 bugs me as urls.py becomes a dumping ground.
However Option 2 often trips me up when I'm developing locally - I need to remember to run migrations whenever I want my init code to run. Things like pulling down a production db or similar often cause problems because migrations aren't triggered.

I would suggest the connection_created signal, which is:
Sent when the database wrapper makes the initial connection to the
database. This is particularly useful if you’d like to send any post
connection commands to the SQL backend.
So it will execute the signal's code when the app connects to the database at the start of the application's cycle.
It will also work within a multiple database configuration and even separate the connections made by the app at initialization:
connection
The database connection that was opened. This can be used in a multiple-database configuration to differentiate connection signals
from different databases.
Note:
You may want to consider using a combination of post_migrate and connection_created signals while checking inside your AppConfig.ready() if a migration happened (ex. flag the activation of a post_migrate signal):
from django.apps import AppConfig
from django.db.models.signals import post_migrate, connection_created
# OR for Django 2.0+
# django.db.backends.signals import post_migrate, connection_created
migration_happened = false
def post_migration_callback(sender, **kwargs):
...
migration_happened = true
def init_my_app(sender, connection):
...
class MyAppConfig(AppConfig):
...
def ready(self):
post_migrate.connect(post_migration_callback, sender=self)
if !migration_happened:
connection_created.connect(init_my_app, sender=self)

In Django >= 3.2 the post_migrate signal is sent even if no migrations are run, so you can use it to run startup code that talks to the database.
https://docs.djangoproject.com/en/3.2/ref/signals/#post-migrate
Sent at the end of the migrate (even if no migrations are run) and flush commands. It’s not emitted for applications that lack a models module.
Handlers of this signal must not perform database schema alterations as doing so may cause the flush command to fail if it runs during the migrate command.

Run Unittest On Main Django Database

I'm looking for a way to run a full celery setup during django tests, asked in this other SO question
After thinking about it, I think I could settle for running a unittest (it's more of an integration test) in which I run the test script against the main Django (development) database. Is there a way to write unittests, run them with Nose and do so against the main database? I imagine it would be a matter of telling Nose (or whatever other framework) about the django settings.
I've looked at django-nose but wasn't able to find a way to tell it to use the main DB and not a test one.

I don't know about nose, but here is how to run against and existing db with django (1.6) unit tests.
from django.test.runner import DiscoverRunner
from django.db import transaction
class ExistingDBTestRunner(DiscoverRunner):
def run_tests(self, test_labels, extra_tests=None, **kwargs):
self.setup_test_environment()
suite = self.build_suite(test_labels, extra_tests)
#old_config = self.setup_databases()
result = self.run_suite(suite)
#self.teardown_databases(old_config)
self.teardown_test_environment()
return self.suite_result(suite, result)
Then in settings.py
if 'test' in sys.argv:
TEST_RUNNER = '<?>.ExistingDBTestRunner'
# alternative db settings?
It will be a little different in older versions of django. Also, you may need to override _fixture_setup and _fixture_teardown in your test cases to pass.
The above code will connect to a preexisting database but since each test is wrapped in a transaction the changes won't be available to other connections (like the celery worker). The easiest way to disable transactions is to subclass from unittest.TestCase instead of django.test.TestCase.

Have you had a look at django-nose? It seems like it would be the right tool for the job.

Why running django's session cleanup command kill's my machine resources?

I have a one year production site configured with django.contrib.sessions.backends.cached_db backend with a MySQL database backend. The reason why I chose cached_db is a mix of security with read performance.
The problem is, the cleanup command, responsible to delete all expired sessions, was never executed, resulting in a 2.3GB session table data length, 6 million rows and 500Mb index length.
When I try to run the ./manage.py cleanup (in Django 1.3) command, or ./manage.py clearsessions (Django`s 1.5 correspondent), the process never ends (or my patience doesn't complete 3 hours).
The code that Django use's to do this is:
Session.objects.filter(expire_date__lt=timezone.now()).delete()
In a first impression, I think that's normal because the table has 6M rows, but, after I inspect System's monitor, I discover that all memory and cpu was used by the python process, not mysqld, fullfilling my machine's resources. I think that's something terrible wrong with this command code. It seems that python iterates over all founded expired session rows before deleting each of them, one by one. In this case, a code refactoring to just raw a DELETE FROM command can resolve my problem and helps Django community, right? But, if this is the case, a Queryset delete command is acting weird and none optimized in my opinion. Am I right?