We Keep Coding

Where should custom statup code live in django projects?

While, switching to gunicorn on my current project, I understood that the way I handled model caching on startup was not a good one.
Setting
app name: website
project name: personal_cms
There are several models I equipped with a load method.
class SomeModel(models.Model):
something = models.CharField(max_length=60)
something_else = models.URLField()
#classmethod
def load(cls):
cache.set('{}'.format(cls.__name__), cls.objects.all(), None)
Typically, signals call these methods every time something change in the model. To load those while starting the server, I just added the following in wsgi.py:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'personal_cms.settings')
from website.models import SomeModel, SomeOtherModel
SomeModel.load()
SomeOtherModel.load()
application = get_wsgi_application()
The problem
The code above works, but only using while using python manage.py runserver, not gunicorn personal_cms.wsgi:application. The reason, from what I understood, is that running manage.py comes with a specific context. Without it, apps aren't loaded when gunicorn hits wsgi.py:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
Now that I think about it, it seems wrong to load the app's models from the project's wsgi.py. So, where should I call the load methods so that models are cached on startup no mater the wsgi server chosen?

All startup code should go in the ready method of an AppConfig. Then reference the config class, rather than the app name, on IBSTALLED_APPS. This is guaranteed to be called at startup.
See the AppConfig docs

Unresolved reference 'models'

I am writing custom template tag, and the error occurs that "Unresolved reference 'models'"
the following is my blog_tags.py.
from django import template
from .models import Post
register = template.Library()
#register.simple_tag
def total_posts():
return Post.published.count()
And my directory tree is as followed
blog/
__init__.py
models.py
...
templatetags/
__init__.py
blog_tags.py
And i do have a Post class in my Models.
And when i click the prompt by pycharm "install package Post", after finishing installed it, the error disappear.
I wonder do i have to do the same, which is install the package by the IDE, every time when i want to write a custom tag evolved with class in my Models?

If I'm interpreting your project structure correctly, your models module is located in a parent package relative to blog_tags. Accessing .models would try to find the module inside your templatetags package.
Try to change your import to this instead:
from ..models import Post

As this is Django and as in Django circular imports can be an issue, consider dynamically loading the model:
for django 1.7+ use the application registry:
from django.apps import apps
Post = apps.get_model('blog', 'Post')
for earlier versions:
from django.db.models.loading import get_model
Post = get_model('blog', 'Post')
Note: This only works if 'blog' is an installed app.

import your models with app namespace instead of relative import, so that the standard structure is maintained.
from django import template
# blog is your app name
from blog.models import Post
register = template.Library()
#register.simple_tag
def total_posts():
return Post.published.count()
Please check here unresolved error issue related to pycharm in django projects

Django django.contrib.sites where to put migration?

the django doc says to change the sites name and domain in the django.contrib.sites framework one should use a migration [1].
But they forgot to mention where I should put this migration. I tried to create a directory named "sites" and a directory named "django.contrib.sites". But no matter in which directory I put my migration, manage.py migration always says there is nothing to update.
I also tried to run python manage.py makemigrations --empty sites, but then the migration is created in the lib directory: ve/lib/python3.5/site-packages/django/contrib/sites/migrations/0003_auto_20160904_2144.py. This may be correct behaviour, but then I cannot set my change under source control.
In case something is wrong with my migration, here it is:
from __future__ import unicode_literals
from django.db import migrations, models
def set_site_name(apps, schema_editor):
Sites = apps.get_model('django.contrib.sites', 'site')
site = Sites.objects.filter(id=1).first()
if site != None:
site.name = "name"
site.domain = "name.com"
class Migration(migrations.Migration):
initial = True
operations = [
migrations.RunPython(set_site_name),
]
So my question is: where does django expect to find those migrations?
Thank you very much in advance for your help.
[1] https://docs.djangoproject.com/en/1.10/ref/contrib/sites/#enabling-the-sites-framework

Each app in a Django project must have a unique label. Naming your app sites isn't a good idea - it will clash with the django.contrib.sites app unless you change the label in the app config class.
If you have an existing app specific to your project, you could use that app to store the data migration.
Alternatively choose a different name like mysites. Create the app with ./manage.py startapp mysite, add the app to your INSTALLED_APPS, then create a blank migration.

Testing Django allauth

I am trying to test my app but not sure how to configure the django-allauth in the test environment. I am getting:
ImproperlyConfigured: No Facebook app configured: please add a SocialApp using the Django admin
My approach so far is to instantiate app objects inside tests.py with actual Facebook app parameters, an app which functions correctly locally in the browser:
from allauth.socialaccount.models import SocialApp
apper = SocialApp.objects.create(provider=u'facebook',
name=u'fb1', client_id=u'7874132722290502',
secret=u'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
apper.sites.create(domain='localhost:8000', name='creyu.org')
How can I get these tests running? Thanks

Where inside tests.py do you instantiate this app object? If it's inside the setUpModule() method, there shouldn't be a problem.
Personally, I would create a fixture init_facebook_app.json with the relevant information and then inside tests.py (before the test cases) define:
from django.core.management import call_command
def setUpModule():
call_command('loaddata', 'init_facebook_app.json', verbosity=0)
This ensures that the data in the fixture are loaded before the tests are run, and that they are loaded only once, i.e. not before each test. See this for reference regarding call_command.
Lastly, posting your Facebook app secret key anywhere on the internet is not a good idea - I would reset it if I were you.

I would create a migration so all your environments have the data
eg
import os
from django.db import models, migrations
from django.core.management import call_command
from django.conf import settings
class Migration(migrations.Migration):
def add_initial_providers(apps, schema_editor):
import pdb;pdb.set_trace()
call_command(
'loaddata',
os.path.join(settings.BASE_DIR, 'fixtures/social_auth.json'),
verbosity=0)
dependencies = [
('my_app', '001_auto_20160128_1846'),
]
operations = [
migrations.RunPython(add_initial_providers),
]

dynamically loading django apps at runtime

Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.
In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.
This is easy enough. Following the example code in
Django: Dynamically add apps as plugin, building urls and other settings automatically
we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in INSTALLED_APPS:
INSTALLED_APPS += (app_name, )
After that, if I was in a Django shell, I could something like
from django.conf import settings
and the apps would be listed in settings.INSTALLED_APPS. And if I did
from django.core import management
management.call_command('syncdb', interactive=False)
that would create the necessary DB tables for the apps.
However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.
What I would like to know is if there is something I could do --- without restarting --- to reload the settings and load/install new apps.
I have tried to directly import my settings.py, i.e.
from myproject import settings
and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,
from django.db import models
models.get_apps()
shows only the original apps in apps and not the newly added ones and likewise
management.call_command('syncdb', interactive=False)
will not see the newly added apps.
As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.
Ps. I working with Django 1.6, but on the advice of #RickyA, I see that there are some substantial changes to Django's treatment of applications in 1.7
https://docs.djangoproject.com/en/1.7/ref/applications/
I'm still not sure what this might mean for the problem I am facing.

Update for Django 1.8 on how to load an app that is not loaded yet
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)
# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)

With Django 2.2 this work for me
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
management.call_command('makemigrations', new_app_name, interactive=False)
management.call_command('migrate', new_app_name, interactive=False)

To answer my own question...
While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.
The basic solution is simple, and I found it at a wee little bixly blog.
Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do
import os
from django.conf import settings
from django.db.models import loading
from django.core import management
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in settings.INSTALLED_APPS:
settings.INSTALLED_APPS += (app_name, )
and then
loading.cache.loaded = False
management.call_command('syncdb', interactive=False)

It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:
#override_settings(INSTALLED_APPS=[...]) # added or removed some apps
class MyTest(TestCase):
# some tests with these apps
def test_foo(self):
pass
It has been possible since September 2014, not before the question.
Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can't be recommended in production, even that it could work eventually.

Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.
def get_installed_apps():
from os import walk, chdir, getcwd
previous_path = getcwd()
master = []
APPS_ROOT_PATH = '/my/project/apps/folder'
chdir(APPS_ROOT_PATH)
for root, directories, files in walk(top=getcwd(), topdown=False):
for file in files:
if 'apps.py' in file:
app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
print(app_path)
master.append(app_path)
chdir(previous_path)
return master
print(get_installed_apps())