While, switching to gunicorn on my current project, I understood that the way I handled model caching on startup was not a good one.
Setting
app name: website
project name: personal_cms
There are several models I equipped with a load method.
class SomeModel(models.Model):
something = models.CharField(max_length=60)
something_else = models.URLField()
#classmethod
def load(cls):
cache.set('{}'.format(cls.__name__), cls.objects.all(), None)
Typically, signals call these methods every time something change in the model. To load those while starting the server, I just added the following in wsgi.py:
import os
from django.core.wsgi import get_wsgi_application
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'personal_cms.settings')
from website.models import SomeModel, SomeOtherModel
SomeModel.load()
SomeOtherModel.load()
application = get_wsgi_application()
The problem
The code above works, but only using while using python manage.py runserver, not gunicorn personal_cms.wsgi:application. The reason, from what I understood, is that running manage.py comes with a specific context. Without it, apps aren't loaded when gunicorn hits wsgi.py:
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.
Now that I think about it, it seems wrong to load the app's models from the project's wsgi.py. So, where should I call the load methods so that models are cached on startup no mater the wsgi server chosen?
All startup code should go in the ready method of an AppConfig. Then reference the config class, rather than the app name, on IBSTALLED_APPS. This is guaranteed to be called at startup.
See the AppConfig docs
Related
I have Django models Driver and Trip. Nothing in my views and no urls. I'm using Django as mere Database, using scripts to store stuff there in DB.
Here is my tree of how every thing looks:
loadmngr/
models.py
views.py
urls.py
management/
commands/
> it.py <
Assume I have init.py inside management and commands.
All I'm doing is a simple import of my Django models. Here is it.py:
import sys
parent = '/Users/work/TM/loadmngr'
sys.path.insert(0, parent)
from models import Trip
I run python manage.py it and I get a RunTimeError:
RunTimeError: Model class models.Driver doesn't declare an explicit app_label and either isn't in an application in INSTALLED_APPS or else was imported before its application was loaded.
The latter part of the error ...or else was imported before its application was loaded is what I believe could be the problem.
Question is: How can I properly use my Django models externally with properly configured settings?
the django doc says to change the sites name and domain in the django.contrib.sites framework one should use a migration [1].
But they forgot to mention where I should put this migration. I tried to create a directory named "sites" and a directory named "django.contrib.sites". But no matter in which directory I put my migration, manage.py migration always says there is nothing to update.
I also tried to run python manage.py makemigrations --empty sites, but then the migration is created in the lib directory: ve/lib/python3.5/site-packages/django/contrib/sites/migrations/0003_auto_20160904_2144.py. This may be correct behaviour, but then I cannot set my change under source control.
In case something is wrong with my migration, here it is:
from __future__ import unicode_literals
from django.db import migrations, models
def set_site_name(apps, schema_editor):
Sites = apps.get_model('django.contrib.sites', 'site')
site = Sites.objects.filter(id=1).first()
if site != None:
site.name = "name"
site.domain = "name.com"
class Migration(migrations.Migration):
initial = True
operations = [
migrations.RunPython(set_site_name),
]
So my question is: where does django expect to find those migrations?
Thank you very much in advance for your help.
[1] https://docs.djangoproject.com/en/1.10/ref/contrib/sites/#enabling-the-sites-framework
Each app in a Django project must have a unique label. Naming your app sites isn't a good idea - it will clash with the django.contrib.sites app unless you change the label in the app config class.
If you have an existing app specific to your project, you could use that app to store the data migration.
Alternatively choose a different name like mysites. Create the app with ./manage.py startapp mysite, add the app to your INSTALLED_APPS, then create a blank migration.
I have made a scraper to scrape some links from web page and want to run this scraper every 1 hours which resides in django app, but django it is impossible to run a scraper every 1 hours because the django views depends on the request response object. to solve this problem I have decided to use a python library named celery and according to the documentation I have write celery.py and tasks.py files
By django project structure is like this
newsportal
- newsportal
-settings.py
-celery.py
__init__.py
- news
-tasks.py
-views.py
-models.py
celery.py has the following code
from __future__ import absolute_import
import os
from celery import Celery
# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'newsportal.settings')
from django.conf import settings # noqa
app = Celery('newsportal')
# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)
#app.task(bind=True)
def debug_task(self):
print('Request: {0!r}'.format(self.request))
__init__.py file has the following lines of code
from __future__ import absolute_import
# This will make sure the app is always imported when
# Django starts so that shared_task will use this app.
from .celery import app as celery_app # noqa
while as tasks.py has the following lines of code
from __future__ import absolute_import
from celery import shared_task
from crawler import crawler
from .models import News
#shared_task
def news():
'''
scrape all links
'''
news = [] #store dict object
allnews.append(crawler())
for news_dict in allnews:
for news, url in news_dict.items():
#Save all the scrape news in database
News.objects.create(title=news, url=url, source=source)
what I want to do is to run the above news() function every 1 hours and save the result to the database.
I want to save the result of the tasks to the django database, how can I achive this.
according to the celery docs, to save the result given by the worker we need install django-celery==3.1.17, as I have already installed, and do migration.
For the database backend in celery according to celery docs, we should put
app.conf.update(
CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend',
)
line of code on settings.py file, on putting this of code in `settings.py` file I got the error of
settings.py", line 141, in <module>
app.conf.update(
NameError: name 'app' is not defined
as I have already Import and put the following line of code in settings.py file as below
from __future__ import absolute_import
BROKER_URL = 'redis://localhost'
The main thing I want to do is,
Running the above crawler every 1 hour and saving the result of
crawler in databse called news
How can I accomplish this using celery or am I missing something ?
Are there any other alternatives way to accomplish this task
I believe you would use app.conf.update(...) in your celery.py if you wanted to add that configuration there.
Your app.config_from_object('django.conf:settings') call in celery.py indicates that you're loading the configuration settings from your settings.py file though.
So you should just be able to put CELERY_RESULT_BACKEND='djcelery.backends.database:DatabaseBackend' at the end of your settings.py file instead.
This should prevent you from getting that error.
I know this is a little late however I can highly recommend the Django Celery Result package found here.
Installation is straight forward and the package is recommended by Celery itself. Simply return some output from your task and it will be stored in the database and accessible under the Django admin.
I am trying to test my app but not sure how to configure the django-allauth in the test environment. I am getting:
ImproperlyConfigured: No Facebook app configured: please add a SocialApp using the Django admin
My approach so far is to instantiate app objects inside tests.py with actual Facebook app parameters, an app which functions correctly locally in the browser:
from allauth.socialaccount.models import SocialApp
apper = SocialApp.objects.create(provider=u'facebook',
name=u'fb1', client_id=u'7874132722290502',
secret=u'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX')
apper.sites.create(domain='localhost:8000', name='creyu.org')
How can I get these tests running? Thanks
Where inside tests.py do you instantiate this app object? If it's inside the setUpModule() method, there shouldn't be a problem.
Personally, I would create a fixture init_facebook_app.json with the relevant information and then inside tests.py (before the test cases) define:
from django.core.management import call_command
def setUpModule():
call_command('loaddata', 'init_facebook_app.json', verbosity=0)
This ensures that the data in the fixture are loaded before the tests are run, and that they are loaded only once, i.e. not before each test. See this for reference regarding call_command.
Lastly, posting your Facebook app secret key anywhere on the internet is not a good idea - I would reset it if I were you.
I would create a migration so all your environments have the data
eg
import os
from django.db import models, migrations
from django.core.management import call_command
from django.conf import settings
class Migration(migrations.Migration):
def add_initial_providers(apps, schema_editor):
import pdb;pdb.set_trace()
call_command(
'loaddata',
os.path.join(settings.BASE_DIR, 'fixtures/social_auth.json'),
verbosity=0)
dependencies = [
('my_app', '001_auto_20160128_1846'),
]
operations = [
migrations.RunPython(add_initial_providers),
]
Is it possible to dynamically load Django apps at runtime? Usually, apps are loaded at initialization, using the INSTALLED_APPS tuple in settings.py. However, is it possible to load additional apps at runtime? I am encountering this issue in different situations. One situation, for example, arises during testing, when I would like to dynamically load or unload apps.
In order to make the problem more concrete, imagine I have a directory called apps where I put my apps and I would like to automatically install any new app that goes in there without manually editing the settings.py.
This is easy enough. Following the example code in
Django: Dynamically add apps as plugin, building urls and other settings automatically
we put the following code in settings.py to could loop over the names of all sub-directories in the app directory and increment the INSTALLED_APPS tuple in settings.py like this:
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in INSTALLED_APPS:
INSTALLED_APPS += (app_name, )
After that, if I was in a Django shell, I could something like
from django.conf import settings
and the apps would be listed in settings.INSTALLED_APPS. And if I did
from django.core import management
management.call_command('syncdb', interactive=False)
that would create the necessary DB tables for the apps.
However, if I were to now add some more apps to the apps/ directory, without re-starting, these would not be listed in settings.INSTALLED_APPS, and so a subsequent call to the syncdb would have no effect.
What I would like to know is if there is something I could do --- without restarting --- to reload the settings and load/install new apps.
I have tried to directly import my settings.py, i.e.
from myproject import settings
and then reload that settings using the python builtin after any app directory changes. Although settings.INSTALLED_APPS is now changed to include the newly added apps, this ultimately makes no difference. For example,
from django.db import models
models.get_apps()
shows only the original apps in apps and not the newly added ones and likewise
management.call_command('syncdb', interactive=False)
will not see the newly added apps.
As I stated above, I am thinking about this situation particularly in the context of testings where I dynamically would add or remove apps.
Ps. I working with Django 1.6, but on the advice of #RickyA, I see that there are some substantial changes to Django's treatment of applications in 1.7
https://docs.djangoproject.com/en/1.7/ref/applications/
I'm still not sure what this might mean for the problem I am facing.
Update for Django 1.8 on how to load an app that is not loaded yet
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
# To load the new app let's reset app_configs, the dictionary
# with the configuration of loaded apps
apps.app_configs = OrderedDict()
# set ready to false so that populate will work
apps.ready = False
# re-initialize them all; is there a way to add just one without reloading them all?
apps.populate(settings.INSTALLED_APPS)
# now I can generate the migrations for the new app
management.call_command('makemigrations', new_app_name, interactive=False)
# and migrate it
management.call_command('migrate', new_app_name, interactive=False)
With Django 2.2 this work for me
from collections import OrderedDict
from django.apps import apps
from django.conf import settings
from django.core import management
new_app_name = "my_new_app"
settings.INSTALLED_APPS += (new_app_name, )
apps.app_configs = OrderedDict()
apps.apps_ready = apps.models_ready = apps.loading = apps.ready = False
apps.clear_cache()
apps.populate(settings.INSTALLED_APPS)
management.call_command('makemigrations', new_app_name, interactive=False)
management.call_command('migrate', new_app_name, interactive=False)
To answer my own question...
While I do not have a completely general solution to this problem, I do have one that is sufficient for the purposes of dynamically loading apps during testing.
The basic solution is simple, and I found it at a wee little bixly blog.
Continuing with my example above, if I was in a django shell and wanted to add and load some new apps that were added to my apps directory, I could do
import os
from django.conf import settings
from django.db.models import loading
from django.core import management
APPS_DIR = '/path_to/apps/'
for item in os.listdir(APPS_DIR):
if os.path.isdir(os.path.join(APPS_DIR, item)):
app_name = 'apps.%s' % item
if app_name not in settings.INSTALLED_APPS:
settings.INSTALLED_APPS += (app_name, )
and then
loading.cache.loaded = False
management.call_command('syncdb', interactive=False)
It is possible to dynamically load and unload applications in tests in Django >= 1.7 (and also in the current 4.1) by a override_settings() decorator:
#override_settings(INSTALLED_APPS=[...]) # added or removed some apps
class MyTest(TestCase):
# some tests with these apps
def test_foo(self):
pass
It has been possible since September 2014, not before the question.
Many other topics from the question are solved by django.apps also in Django >= 1.7. Generally: Dynamic configuration at startup is easy in the current Django. Dynamic loading and unloading after startup can't be recommended in production, even that it could work eventually.
Yes! Anything (or almost everything) in Python is possible. You should use os.walk() in order to get all folders, subfolders and files within your apps path in order to get all of your apps including the nested ones.
def get_installed_apps():
from os import walk, chdir, getcwd
previous_path = getcwd()
master = []
APPS_ROOT_PATH = '/my/project/apps/folder'
chdir(APPS_ROOT_PATH)
for root, directories, files in walk(top=getcwd(), topdown=False):
for file in files:
if 'apps.py' in file:
app_path = f"{root.replace(BASE_DIR + '/', '').replace('/', '.')}.apps"
print(app_path)
master.append(app_path)
chdir(previous_path)
return master
print(get_installed_apps())