We Keep Coding

Django: Execute code only for `manage.py runserver`, not for `migrate`, `help` etc

We are using Django as backend for a website that provides various things, among others using a Neural Network using Tensorflow to answer to certain requests.
For that, we created an AppConfig and added loading of this app config to the INSTALLED_APPS in Django's settings.py. This AppConfig then loads the Neural Network as soon as it is initialized:
settings.py:
INSTALLED_APPS = [
...
'bert_app.apps.BertAppConfig',
]
.../bert_apps/app.py:
class BertAppConfig(AppConfig):
name = 'bert_app'
if 'bert_app.apps.BertAppConfig' in settings.INSTALLED_APPS:
predictor = BertPredictor() #loads the ANN.
Now while that works and does what it should, the ANN is now loaded for every single command run through manage.py. While we of course want it to be executed if you call manage.py runserver, we don't want it to be run for manage.py migrate, or manage.py help and all other commands.
I am generally not sure if this is the proper way how to load an ANN for a Django-Backend in general, so does anybody have any tips how to do this properly? I can imagine that loading the model on startup is not quite best practice, and I am very open to suggestions on how to do that properly instead.
However, there is also some other code besides the actual model-loading that also takes a few seconds and that is definitely supposed to be executed as soon as the server starts up (so on manage.py runserver), but also not on manage.py help (as it takes a few seconds as well), so is there some quick fix for how to tell Django to execute it only on runserver and not for its other commands?

I had a similar problem, solved it with checking argv.
class SomeAppConfig(AppConfig):
def ready(self, *args, **kwargs):
is_manage_py = any(arg.casefold().endswith("manage.py") for arg in sys.argv)
is_runserver = any(arg.casefold() == "runserver" for arg in sys.argv)
if (is_manage_py and is_runserver) or (not is_manage_py):
init_your_thing_here()
Now a bit closer to the if not is_manage_py part: in production you run your web server with uwsgi/uvicorn/..., which is still a web server, except it's not run with manage.py. Most likely, it's the only thing that you will ever run without manage.py
Use AppConfig.ready() - it's intended for it:
Subclasses can override this method to perform initialization tasks such as registering signals. It is called as soon as the registry is fully populated. - [django documentation]
To get your AppConfig back, use:
from django.apps import apps
apps.get_app_config(app_name)
# apps.get_app_configs() # all

This is another way, in your manage.py will have something probably look like this
def main():
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'slambook.settings')
try:
from django.core.management import execute_from_command_line
except ImportError as exc:
raise ImportError(
"Couldn't import Django. Are you sure it's installed and "
"available on your PYTHONPATH environment variable? Did you "
"forget to activate a virtual environment?"
) from exc
execute_from_command_line(sys.argv)
# check if has runserver
if `runserver` in sys.argv:
#execute your custom function
if __name__ == '__main__':
main()
you can check sys.argv if it have runserver, if so then execute your script or function

Working with flask-rq2 extension in heroku

I followed this tutorial to run tasks with Rides Queue:
https://flask-rq2.readthedocs.io/en/latest/
First
app = Flask(__name__,template_folder='templates')
app.config['RQ_REDIS_URL'] = os.environ['REDIS_URL']
Then
rq = RQ(app)
default_worker.work(burst=True)
And After execute this line
job = task.queue(arg1)
i have faced this error:
i tried to set env vairiable FLASK_APP="app.py" i got this error again but with message
AttributeError: module 'app' has no attribute 'task' 18:43:49 Moving
job to 'failed' queue
i think there is misconfigured options related to worker but where is this in official docs?

Changed FLASK_APP="app.py" to FLASK_APP="app:app" and worked fine but also you need to change this line default_worker.work(burst=True) to inside the method and i did not know way it's not working if i put it in main suite

pytest can't access django classes

I have a Django application, working fine. When I run the tests with pytest, it only works with utility classes (so not Django related).
For example, a test from package A calling an utility class from this package, or another, works fine.
However, I'm facing an error as soon as I import a django class.
example 1 :
I import my model (in the test), starting the test class with :
from app.common.models import Country
--> ImportError: No module named django.db
[django.db is called in models.py]
example 2 : I import an url resolver (in the test), starting the test class with :
from django.core.urlresolvers import reverse
--> ImportError: No module named django.core.urlresolvers
first try of fix
Following another topic, I set the content of PYTHONPATH :
/home/user/pyenv/lib/python3.5/site-packages
This folder contains installed packages in the virtualenv : django, pytest, psycopg2, and more.
1) If I set DJANGO_SETTINGS_MODULE to the same file as the application, "py.test" gives this error ending with :
File "/home/user/pyenv/lib/python3.5/site-packages/django/db/backends/postgresql/base.py", line 24, in
raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
2) If I set DJANGO_SETTINGS_MODULE to a smaller test setting file, containing only the database infos, I face a different error (self.client failing in a test) :
class Test(unittest.TestCase):
def testUrls(self):
response = self.client.get('/countries/all/get')
self.assertEqual(response.status_code, 200)
--> AttributeError: 'Test' object has no attribute 'client'
more infos :
1) The python interpreter, for the application, is located in a virtualenv.
2) conftest.py is located in application root folder, so same place as manage.py, and has the following content:
import os
import sys
sys.path.append(os.path.dirname(__file__))
3) pytest.ini is located in same folder as manage.py too, with this content :
[pytest]
python_files = test_*.py test*.py
4) most important : the application works fine, so db settings are valid
If you have any idea of what's wrong, and how to test django classes, any idea will be welcomed. Thank you in advance.

I use pytest-django and I also explicitly set DJANGO_SETTINGS_MODULE=project.my_settings in pytest.ini. Works great every time.

I faced other issues trying the standard Django testing tool + coverage.
Using Pytest with "--cov" provided me what I needed, in one command only.
I finally found a fix for the issue. After trying again, and so restarting everything (pc, virtualenv), it worked fine.
So :
1) start virtualenv and move to the application folder
2) don't set PYTHONPATH
3) set DJANGO_SETTINGS_MODULE to application settings in pytest.ini
(so only one settings file in the project)
4) run the test with : py.test --cov=my_application_name
Now I can test models or forms without error.
For the previously mentioned test about urls, it works now with the following syntax:
import unittest
from django.test import Client
class Test(unittest.TestCase):
def setUp(self):
# global variable
self.client = Client()
def testUrls(self):
reponse = self.client.get('/countries/all/get')
self.assertEqual(reponse.status_code, 200)

web2py db is not defined

I'm trying to run a script at command line that uses the models with the following command:
c:\web2py>python web2py.py -M -N -S automate -R applications/automate/modules/eventserver.py
but I keep getting the error:
web2py Web Framework
Created by Massimo Di Pierro, Copyright 2007-2011
Version 1.99.7 (2012-03-04 22:12:08) stable
Database drivers available: SQLite3, pymysql, pg8000, IMAP
Traceback (most recent call last):
File "c:\web2py\gluon\shell.py", line 206, in run
execfile(startfile, _env)
File "applications/automate/modules/eventserver.py", line 6, in <module>
deviceHandler = devicehandler.DeviceHandler()
File "applications\automate\modules\devicehandler.py", line 10, in __init__
self.devices = self.getActiveDevices()
File "applications\automate\modules\devicehandler.py", line 18, in getActiveDe
vices
print db
NameError: global name 'db' is not defined
What am I doing wrong?
edit: From my research I have only found the solution "add -M to your command" but I've already done that and it still doesnt work.
edit2: I have db = DAL('sqlite://storage.sqlite') in my db.py so it should get loaded

edit2: I have db = DAL('sqlite://storage.sqlite') in my db.py so it should get loaded
Assuming db.py is in the /models folder, the db object created there will be available in later executed model files as well as in the controller and view, but it will not be available within modules that you import. Instead, you will have to pass the db object to a function or class in the module. Another option is to add the db object to the current thread local object, which can then be imported and accessed within the module:
In /models/db.py:
from gluon import current
db = DAL('sqlite://storage.sqlite')
current.db = db
In /modules/eventserver.py:
from gluon import current
def somefunction():
db = current.db
[do something with db]
Note, if you do define the db object in the module, don't define it at the top level -- define it in a function or class.
For more details, see the book section on modules and current.

Why does Celery work in Python shell, but not in my Django views? (import problem)

I installed Celery (latest stable version.)
I have a directory called /home/myuser/fable/jobs. Inside this directory, I have a file called tasks.py:
from celery.decorators import task
from celery.task import Task
class Submitter(Task):
def run(self, post, **kwargs):
return "Yes, it works!!!!!!"
Inside this directory, I also have a file called celeryconfig.py:
BROKER_HOST = "localhost"
BROKER_PORT = 5672
BROKER_USER = "abc"
BROKER_PASSWORD = "xyz"
BROKER_VHOST = "fablemq"
CELERY_RESULT_BACKEND = "amqp"
CELERY_IMPORTS = ("tasks", )
In my /etc/profile, I have these set as my PYTHONPATH:
PYTHONPATH=/home/myuser/fable:/home/myuser/fable/jobs
So I run my Celery worker using the console ($ celeryd --loglevel=INFO), and I try it out.
I open the Python console and import the tasks. Then, I run the Submitter.
>>> import fable.jobs.tasks as tasks
>>> s = tasks.Submitter()
>>> s.delay("abc")
<AsyncResult: d70d9732-fb07-4cca-82be-d7912124a987>
Everything works, as you can see in my console
[2011-01-09 17:30:05,766: INFO/MainProcess] Task tasks.Submitter[d70d9732-fb07-4cca-82be-d7912124a987] succeeded in 0.0398268699646s:
But when I go into my Django's views.py and run the exact 3 lines of code as above, I get this:
[2011-01-09 17:25:20,298: ERROR/MainProcess] Unknown task ignored: "Task of kind 'fable.jobs.tasks.Submitter' is not registered, please make sure it's imported.": {'retries': 0, 'task': 'fable.jobs.tasks.Submitter', 'args': ('abc',), 'expires': None, 'eta': None, 'kwargs': {}, 'id': 'eb5c65b4-f352-45c6-96f1-05d3a5329d53'}
Traceback (most recent call last):
File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/listener.py", line 321, in receive_message
eventer=self.event_dispatcher)
File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/job.py", line 299, in from_message
eta=eta, expires=expires)
File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/worker/job.py", line 243, in __init__
self.task = tasks[self.task_name]
File "/home/myuser/mysite-env/lib/python2.6/site-packages/celery/registry.py", line 63, in __getitem__
raise self.NotRegistered(str(exc))
NotRegistered: "Task of kind 'fable.jobs.tasks.Submitter' is not registered, please make sure it's imported."
It's weird, because the celeryd client does show that it's registered, when I launch it.
[2011-01-09 17:38:27,446: WARNING/MainProcess]
Configuration ->
. broker -> amqp://GOGOme#localhost:5672/fablemq
. queues ->
. celery -> exchange:celery (direct) binding:celery
. concurrency -> 1
. loader -> celery.loaders.default.Loader
. logfile -> [stderr]#INFO
. events -> OFF
. beat -> OFF
. tasks ->
. tasks.Decayer
. tasks.Submitter
Can someone help?

This is what I did which finally worked
in Settings.py I added
CELERY_IMPORTS = ("myapp.jobs", )
under myapp folder I created a file called jobs.py
from celery.decorators import task
#task(name="jobs.add")
def add(x, y):
return x * y
Then ran from commandline: python manage.py celeryd -l info
in another shell i ran python manage.py shell, then
>>> from myapp.jobs import add
>>> result = add.delay(4, 4)
>>> result.result
and the i get:
16
The important point is that you have to rerun both command shells when you add a new function. You have to register the name both on the client and and on the server.
:-)

I believe your tasks.py file needs to be in a django app (that's registered in settings.py) in order to be imported. Alternatively, you might try importing the tasks from an __init__.py file in your main project or one of the apps.
Also try starting celeryd from manage.py:
$ python manage.py celeryd -E -B -lDEBUG
(-E and -B may or may not be necessary, but that's what I use).

See Automatic Naming and Relative Imports, in the docs:
http://celeryq.org/docs/userguide/tasks.html#automatic-naming-and-relative-imports
The tasks name is "tasks.Submitter" (as listed in the celeryd output),
but you import the task as "fable.jobs.tasks.Submitter"
I guess the best solution here is if the worker also sees it as "fable.jobs.tasks.Submitter",
it makes more sense from an app perspective.
CELERY_IMPORTS = ("fable.jobs.tasks", )