Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.
In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.
This is easy enough. Following the example code in
Django: Dynamically add apps as plugin, building urls and other settings automatically
we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in INSTALLED_APPS:
INSTALLED_APPS += (app_name, )
After that, if I was in a Django shell, I could something like
from django.conf import settings
and the apps would be listed in settings.INSTALLED_APPS. And if I did
from django.core import management
management.call_command('syncdb', interactive=False)
that would create the necessary DB tables for the apps.
However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.
What I would like to know is if there is something I could do --- without restarting --- to reload the settings and load/install new apps.
I have tried to directly import my settings.py, i.e.
from myproject import settings
and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,
from django.db import models
models.get_apps()
shows only the original apps in apps and not the newly added ones and likewise
management.call_command('syncdb', interactive=False)
will not see the newly added apps.
As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.
Ps. I working with Django 1.6, but on the advice of #RickyA, I see that there are some substantial changes to Django's treatment of applications in 1.7
https://docs.djangoproject.com/en/1.7/ref/applications/
I'm still not sure what this might mean for the problem I am facing.
Update for Django 1.8 on how to load an app that is not loaded yet
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)
# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)
With Django 2.2 this work for me
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
management.call_command('makemigrations', new_app_name, interactive=False)
management.call_command('migrate', new_app_name, interactive=False)
To answer my own question...
While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.
The basic solution is simple, and I found it at a wee little bixly blog.
Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do
import os
from django.conf import settings
from django.db.models import loading
from django.core import management
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in settings.INSTALLED_APPS:
settings.INSTALLED_APPS += (app_name, )
and then
loading.cache.loaded = False
management.call_command('syncdb', interactive=False)
It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:
#override_settings(INSTALLED_APPS=[...]) # added or removed some apps
class MyTest(TestCase):
# some tests with these apps
def test_foo(self):
pass
It has been possible since September 2014, not before the question.
Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can't be recommended in production, even that it could work eventually.
Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.
def get_installed_apps():
from os import walk, chdir, getcwd
previous_path = getcwd()
master = []
APPS_ROOT_PATH = '/my/project/apps/folder'
chdir(APPS_ROOT_PATH)
for root, directories, files in walk(top=getcwd(), topdown=False):
for file in files:
if 'apps.py' in file:
app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
print(app_path)
master.append(app_path)
chdir(previous_path)
return master
print(get_installed_apps())
Related
While, switching to gunicorn on my current project, I understood that the way I handled model caching on startup was not a good one.
Setting
app name: website
project name: personal_cms
There are several models I equipped with a load method.
class SomeModel(models.Model):
something = models.CharField(max_length=60)
something_else = models.URLField()
#classmethod
def load(cls):
cache.set('{}'.format(cls.__name__), cls.objects.all(), None)
Typically, signals call these methods every time something change in the model. To load those while starting the server, I just added the following in wsgi.py:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'personal_cms.settings')
from website.models import SomeModel, SomeOtherModel
SomeModel.load()
SomeOtherModel.load()
application = get_wsgi_application()
The problem
The code above works, but only using while using python manage.py runserver, not gunicorn personal_cms.wsgi:application. The reason, from what I understood, is that running manage.py comes with a specific context. Without it, apps aren't loaded when gunicorn hits wsgi.py:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
Now that I think about it, it seems wrong to load the app's models from the project's wsgi.py. So, where should I call the load methods so that models are cached on startup no mater the wsgi server chosen?
All startup code should go in the ready method of an AppConfig. Then reference the config class, rather than the app name, on IBSTALLED_APPS. This is guaranteed to be called at startup.
See the AppConfig docs
I have Django models Driver and Trip. Nothing in my views and no urls. I'm using Django as mere Database, using scripts to store stuff there in DB.
Here is my tree of how every thing looks:
loadmngr/
models.py
views.py
urls.py
management/
commands/
> it.py <
Assume I have init.py inside management and commands.
All I'm doing is a simple import of my Django models. Here is it.py:
import sys
parent = '/Users/work/TM/loadmngr'
sys.path.insert(0, parent)
from models import Trip
I run python manage.py it and I get a RunTimeError:
RunTimeError: Model class models.Driver doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded.
The latter part of the error ...or else was imported before its application was loaded is what I believe could be the problem.
Question is: How can I properly use my Django models externally with properly configured settings?
the django doc says to change the sites name and domain in the django.contrib.sites framework one should use a migration [1].
But they forgot to mention where I should put this migration. I tried to create a directory named "sites" and a directory named "django.contrib.sites". But no matter in which directory I put my migration, manage.py migration always says there is nothing to update.
I also tried to run python manage.py makemigrations --empty sites, but then the migration is created in the lib directory: ve/lib/python3.5/site-packages/django/contrib/sites/migrations/0003_auto_20160904_2144.py. This may be correct behaviour, but then I cannot set my change under source control.
In case something is wrong with my migration, here it is:
from __future__ import unicode_literals
from django.db import migrations, models
def set_site_name(apps, schema_editor):
Sites = apps.get_model('django.contrib.sites', 'site')
site = Sites.objects.filter(id=1).first()
if site != None:
site.name = "name"
site.domain = "name.com"
class Migration(migrations.Migration):
initial = True
operations = [
migrations.RunPython(set_site_name),
]
So my question is: where does django expect to find those migrations?
Thank you very much in advance for your help.
[1] https://docs.djangoproject.com/en/1.10/ref/contrib/sites/#enabling-the-sites-framework
Each app in a Django project must have a unique label. Naming your app sites isn't a good idea - it will clash with the django.contrib.sites app unless you change the label in the app config class.
If you have an existing app specific to your project, you could use that app to store the data migration.
Alternatively choose a different name like mysites. Create the app with ./manage.py startapp mysite, add the app to your INSTALLED_APPS, then create a blank migration.
I am trying to test my app but not sure how to configure the django-allauth in the test environment. I am getting:
ImproperlyConfigured: No Facebook app configured: please add a SocialApp using the Django admin
My approach so far is to instantiate app objects inside tests.py with actual Facebook app parameters, an app which functions correctly locally in the browser:
from allauth.socialaccount.models import SocialApp
apper = SocialApp.objects.create(provider=u'facebook',
name=u'fb1', client_id=u'7874132722290502',
secret=u'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
apper.sites.create(domain='localhost:8000', name='creyu.org')
How can I get these tests running? Thanks
Where inside tests.py do you instantiate this app object? If it's inside the setUpModule() method, there shouldn't be a problem.
Personally, I would create a fixture init_facebook_app.json with the relevant information and then inside tests.py (before the test cases) define:
from django.core.management import call_command
def setUpModule():
call_command('loaddata', 'init_facebook_app.json', verbosity=0)
This ensures that the data in the fixture are loaded before the tests are run, and that they are loaded only once, i.e. not before each test. See this for reference regarding call_command.
Lastly, posting your Facebook app secret key anywhere on the internet is not a good idea - I would reset it if I were you.
I would create a migration so all your environments have the data
eg
import os
from django.db import models, migrations
from django.core.management import call_command
from django.conf import settings
class Migration(migrations.Migration):
def add_initial_providers(apps, schema_editor):
import pdb;pdb.set_trace()
call_command(
'loaddata',
os.path.join(settings.BASE_DIR, 'fixtures/social_auth.json'),
verbosity=0)
dependencies = [
('my_app', '001_auto_20160128_1846'),
]
operations = [
migrations.RunPython(add_initial_providers),
]
Can anyone suggest a detailed resource for including django 1.2 templating in our GAE apps? So far I have found
docs describing how to zip up the django files and add them to our project
docs on running native django projects in GA
docs about including 1.0 and 1.1 libraries into our projects
but nothing yet describing how to use django 1.2 templates in our projects. Specifically, how to formulate the arcane wizardry at the top of my python script that will magically convince GAE to use my zipped up django library.
I have this in my python script:
import sys
sys.path.insert(0, 'django/django.zip')
And similar to the GAE tutorial, here's how I'm rendering the template:
template_values = {
'formerror': formerror,
'media': media,
'status': status
}
path = os.path.join(os.path.dirname(__file__), formtemplate)
self.response.out.write(template.render(path, template_values)
But there is some piece missing for GAE to use Django 1.2 to render the templates. What is it?
I used this:
from google.appengine.dist import use_library
use_library('django', '1.1')
from google.appengine.ext.webapp import template
In this case I used 1.1 version but I think it should work the same for 1.2.
I had the same problem a while ago - I wanted to use version 1.2 version for templates instead of 0.96 (that is provided by GAE). The following code seems to work for me.
# some standard Google App Engine imports (optional)
import wsgiref.handlers
from google.appengine.ext import webapp
from google.appengine.ext import db
# Remove Django modules (0.96) from namespace
for k in [k for k in sys.modules if k.startswith('django')]:
del sys.modules[k]
# Force sys.path to have our own directory first, in case we want to import
# from it. This way, when we import Django, the interpreter will first search
# for it in our directory.
sys.path.insert(0, os.path.abspath(os.path.dirname(__file__)))
# Must set this env var *before* importing any part of Django
# (that's required in Django documentation)
os.environ['DJANGO_SETTINGS_MODULE'] = 'settings'
# New Django imports
from django.template.loader import render_to_string
from django.conf import settings
# Configure dir with templates
# our template dir is: /templates
TEMPLATE_DIR = os.path.join(os.path.dirname(__file__),'templates')
settings.configure(TEMPLATE_DIRS = (TEMPLATE_DIR,'') )
However, if you need only templates from Django, and no other API, consider using Jinja instead. That's what I'm planning to do.