We Keep Coding

Django settings: sharing ORM with a standalone module

Introduction:
I am learning Django as I develop my first project using PyCharm with Python 3.8 and Postgresql.
I have a problem with the settings.
Environment description:
My aim is to have a Django project for a web site connected to a database (let's call it "web_db"), which is fed at regular intervals with data from an external source (another database, let's call it "source_db").
For this ETL process from one database to the other one, I am developing a Python module called "source2web". I read the content of source_db using psycopg2.
I am developing everything inside one PyCharm project.
Now, since I will be using Django ORM to access the content of web_db from the Django application server, I concluded I might as well use Django ORM in source2web to load my content into web_db.
This means I am trying to share some code between the Django project and source2web: the ORM models of Django.
source2web is simply launched as a python module ( python source2web.py).
Inside my PyCharm project, I have a src folder inside which are all my python modules, with a tree like this (folders are noted with parenthesis) :
(src)
|--(project_name)
|--(module1)
| |--source2web.py
|
|--(django)
|--(django_project)
|--(django_project)
|--(another_app)
The folders django_project/django_project and django_project/another_app have been created classicaly by Django with the commands django-admin startproject django_project and python3 manage.py startapp another_app.
The ORM models I want to share are the model classes inside the file django_project/another_app/models.py
The problem:
I tinkered to find a way to use the Django ORM models from source2web. I thought I had it : inside the code of source2web, I added these lines :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django
django.setup()
But then, I realize that the Django server and the standalone module source2web can both work, but not with the exact same content inside project_name.django.django_project.django_project.settings.py : the name of another_app in the list of INSTALLED_APPS must be different.
Django requires the app to be listed as just another_app
source2web requires the app to be listed as project_name.django.django_project.another_app
I am sure I could find a 'dirty way' to solve this, but I also think I am not the first one to face this problem (sharing Django ORM code between a Django project and a standalone program) and I hope there is cleaner way to manage this.
What I can think of is duplicating the module project_name.django.django_project.django_project.settings and then just change the settings call inside source2web ( os.environ['DJANGO_SETTINGS_MODULE'] = ... ) but I never like duplicating code and maybe there is a cleaner than that ?

I have found at least one decently clean workaround :
In the project_name.django.django_project.django_project.settings file, I have left the value of INSTALLED_APPS that work with Django, i.e., the list of default apps created with the Django Project plus another_app.
Then in the source2web module, I have changed my code for initializing Django ORM :
In module source2web, I added 2 lines :
import project_name.django.django_project.django_project.settings as djangosettings
djangosettings.INSTALLED_APPS = ['project_name.django.django_project.another_app',]
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django`
django.setup()
This way, I leave untouched the settings for the Django server, and only override what's necessary in the source2web module (i.e. the value of INSTALLED_APPS, getting rid at the same time of all the default apps I don't need for my ETL process).
It works and seems robust enough.
Still, it brought me to question the whole project structure :
Because obviously, the whole problem is the result of the PyCharm project considering for source root src/, while the Django project considers for root project_name/django/django_project/ .
There should be a way to reconcile them, but how ?

Conditional Django Model Creation

I am writing a app for django which i am planning to publish. This app requires a Bolean Setting variable CONSUMER_REGISTRATION.
Aim of getting this variable is to decide whether to define ConsumerRegistrationModel or not.
This is what I did.
from django.db import models
from django.conf import settings
if getattr(settings, 'CONSUMER_REGISTRATION', False):
class ConsumerRegistration(models.Model):
...
Its working fine. The only Issue i am facing that developers will need to run makemigrations and migrate commands each time they change the variable in settings.
1- Can this work be automated ?. So if they change the variable then some code in django auto run the makemigrations and migrate commands.
2- Or is it perfectly fine to leave this work on developers ??
3- Also I want to ask that is it a good aproach to do this in django ?

The accepted answer doesn't really provide a way to do what the OP is asking, which is to conditionally declare a model.
People may have various reasons for wanting to do this, from not declaring a model at all, to declaring models differently based on settings (it is implied that if you are doing this: you are intend to run the same code base in different places using different settings).
One solution is to put the model in its own app, and conditionally include the app based on a setting:
# Set this your per-project settings:
CONSUMER_REGISTRATION = True
# Set this in the generic settings
INSTALLED_APPS = [...]
if CONSUMER_REGISTRATION:
INSTALLED_APPS.append('consumer_registration') # Models will be loaded.
There's nothing wrong with creating an app which just contains a model.
With regards to "automating" it, the table will be created if migrations are run when the setting is true. It will not delete the table if it is changed to false.

You could simply define the model without any conditionals and tweak your app logic so that instances of ConsumerRegistration model are only interacted with (i.e. created, updated etc.) when the 'CONSUMER_REGISTRATION' flag is set to True.
Running migrations every single time the value of 'CONSUMER_REGISTRATION' is changed would make much more mess than leaving ConsumerRegistration table empty.
As indicated by #dahrens, you could isolate the ConsumerRegistration model along with relevant logic in a separate app, which would only be installed as needed by developers.

How to Running external command (Django) for overriding files in server?

After modified inspectdb, I want to run it when index page on web is loaded. So, in my view.py - def index, I´m trying to do the next:
def index(request):
subprocess.Popen("rm /path/app/models.py", shell=True)
subprocess.Popen("python2.7 /path/manage.py inspectdb_New > /path/app/models.py", shell=True)
return render_to_response('index/index.html', context_instance = RequestContext(request))
That is not working. I tried with os.system, subprocess.call as well, but it´s still not working.
For me, at least, looks like I can´t modify models.py in execution time but I don´t know what could be the problem...
Any idea guys?
Thanks.

You can't change your models.py while the webserver is running, because models (and other python code) are only loaded at server startup.

There are 2 parts in you question:
a) you want the shell to do something when view function is called. I have alarm beeping already.
b) you want to load new model. It won't gonna happen. You must reload to have new code loaded in (model introspected, etc.). Probably you use runserver command, take a look on Django autoreloader code: https://github.com/django/django/blob/master/django/utils/autoreload.py
It tracks files listed by gen_filenames() and reload when something was changed. You model was not listed so the change to the code will not be reflected.
I would touch something or remove .pyc files to force the reloader.

How to use custom AdminSite class?

Which is the best way to implement my own django.contrib.admin.sites.AdminSite?
Actually I get a problem with the registration of INSTALLED_APPS in django.contrib.admin.autodiscover. If I use my custom AdminSite class in urls.py, there were no apps displayed on the admin page.
I fixed this with a litte hack. I wrote this class:
from django.contrib.admin.sites import site as default_site
class AdminSiteRegistryFix( object ):
'''
This fix links the '_registry' property to the orginal AdminSites
'_registry' property. This is necessary, because of the character of
the admins 'autodiscover' function. Otherwise the admin site will say,
that you havn't permission to edit anything.
'''
def _registry_getter(self):
return default_site._registry
def _registry_setter(self,value):
default_site._registry = value
_registry = property(_registry_getter, _registry_setter)
And implement my custom AdminSite like this:
from wltrweb.hacks.django.admin import AdminSiteRegistryFix
from django.contrib.admin import AdminSite
class MyAdminSite( AdminSite, AdminSiteRegistryFix ):
# do some magic
pass
site = MyAdminSite()
So I can use this site for urls.py.
Anyone knows a better way? Since I access a var starting with a underscore it is no more than a hack. I don't like hacks.
Edit: Another way would be to rewrite the django.contrib.admin.autodiscover function, but in this case I would have redundant code.

The Problem
Using a custom class derived from django.contrib.admin.AdminSite for the admin site of a project, without having to write custom registration code to register models with the new class. When I use 3rd party apps with their own models, I'd rather not have to edit custom registration code only because models were added or removed from these apps.
The Solution
You have to switch the instance created with the default class used for the admin site to your own instance, created with your own class before django.contrib.admin's autodiscover function is called. I do this by:
Having an app that will perform the switch. (I use my project-specific app named core for my own purposes.)
Two choices:
Django 1.6 to 1.9: use __init__ of the app to perform the switch. In Django 1.8, you will get a deprecation warning due to the change in Django 1.9 quoted below. Note that this method will work with 1.9 too because the Django modules loaded by the code shown below have been changed in 1.9 so that they no longer load models. When I use this method my core/__init__.py file contains:
from django.contrib import admin
from django.contrib.admin import sites
class MyAdminSite(admin.AdminSite):
pass
mysite = MyAdminSite()
admin.site = mysite
sites.site = mysite
Django 1.9 and over: use the app configuration of the app to perform the switch. As of Django 1.9, as the release notes state:
All models need to be defined inside an installed application or declare an explicit app_label. Furthermore, it isn’t possible to import them before their application is loaded. In particular, it isn’t possible to import models inside the root package of an application.
I prefer to limit the imports I do at the root level to avoid the risk of loading models. While as of version 1.9 using the __init__ method above will work, there's no telling if 1.10 or a later version will introduce a change that will cause problems.
When I use this method the core/__init__.py sets default_app_config = "core.apps.DefaultAppConfig" and I have a core/apps.py like this:
from django.apps import AppConfig
class DefaultAppConfig(AppConfig):
name = 'core'
def ready(self):
from django.contrib import admin
from django.contrib.admin import sites
class MyAdminSite(admin.AdminSite):
pass
mysite = MyAdminSite()
admin.site = mysite
sites.site = mysite
While it is possible to use this method with versions 1.7 and 1.8, it is a bit risky to use it with those versions. See the notes below.
Placing this app earlier than django.contrib.admin in the INSTALLED_APPS list. (This is absolutely necessary for 1.7 and later. In earlier versions of Django, it might work okay even if the app is later than django.contrib.admin. However, see the notes below.)
Notes and Caveats
The app that performs the switch should really be the first app in the INSTALLED_APPS list so as to minimize the chance that something else will grab the value of site from django.contrib.admin before the switch is made. If another app manages to get that value before the switch is done, then that other app will have a reference to the old site. Hilarity will surely ensue.
The method above won't work nicely if two apps are trying to install their own new default admin site class. This would have to be handled on a case-by-case basis.
It is possible that a future release of Django could break this method.
For version prior to 1.9, I preferred using __init__ to do site switch over using the app configuration because the documentation on initialization indicates that the ready() method of the app configurations is called relatively late. Between the time an app's module is loaded, and the time ready() is called, models have been loaded, and in some case, it could mean that a module has grabbed the value of site from django.contrib.admin before ready is called. So as to minimize the risk, I have the app's __init__ code do the switch.
I believe the risk that existed in version 1.7 and 1.8 and that I avoided by using __init__ to perform the site switch as early as possible does not exist in 1.9. Everybody is prohibited from loading modules before all the applications are loaded. So doing the switch in the ready callback of the first application listed in INSTALLED_APPS should be safe. I've upgraded a large project to 1.9 and used the app configuration method, without any problem.

From Django 2.1, there is an 'out-of-the-box' solution: https://docs.djangoproject.com/en/2.1/ref/contrib/admin/#overriding-the-default-admin-site
from django.contrib import admin
class MyAdminSite(admin.AdminSite):
...
Swapping the custom admin site is now done by adding your own AdminConfig to installed apps.
from django.contrib.admin.apps import AdminConfig
class MyAdminConfig(AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
INSTALLED_APPS = [
...
'myproject.apps.MyAdminConfig', # replaces 'django.contrib.admin'
...
]
Note the difference between AdminConfig and SimpleAdminConfig, where the latter doesn't trigger admin.autodiscover(). I'm currently using this solution in a project.

Having encountered the same kind of issue while implemeting site-wide custom AdminSite in Django 3.2, I found a workaround to make it work.
It seems the documentation should be updated accordingly.
from django.contrib.admin.apps import AdminConfig
class MyAdminConfig(AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
This raises an exception:
RuntimeError: 'myproject.apps' declares more than one default AppConfig: 'AdminConfig', 'MyAdminConfig'.
It is solved by importing django.contrib.admin.apps instead of django.contrib.admin.apps.AdminConfig:
from django.contrib.admin import apps
class MyAdminConfig(apps.AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
Then the exception moves on:
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: admin
This is caused by the settings.py configuration:
INSTALLED_APPS = [
'myproject.apps.MyAdminConfig', #replaces django.contrib.admin
...
It is solved by removing the 'MyAdminConfig' from the settings:
INSTALLED_APPS = [
'myproject.apps', #replaces django.contrib.admin
...

Quoting from https://docs.djangoproject.com/en/1.10/ref/contrib/admin/#customizing-the-adminsite-class
If you'd like to set up your own administrative site with custom behavior, however, you're free to subclass AdminSite and override or add anything you like. Then, simply create an instance of your AdminSite subclass (the same way you'd instantiate any other Python class), and register your models and ModelAdmin subclasses with it instead of using the default.
I guess that is the most explicit approach, but it also means that you need to change the register code in your apps admin.py files.
There is really no need to use autodiscover when using your own AdminSite instance since you will likely be importing all the per-app admin.py modules in your myproject.admin module.
The assumption seems to be, that once you start writing your custom admin site, it becomes pretty much project specific and you know beforehand which apps you want to include.
So if you don't want to work with the hack above, I only really see these two options. Replace all register calls to your custom admin site or register the models explicitly in your adminsite module.

Django app initalization code (like connecting to signals)

I need a place to run an initialization code that is application specific (like connecting to signals).
When I put the code to __init__.py module of an application I've ended up with a circular import of the models.
Is there a way to fire a function when the framework is setup and before any request is executed?
I use quite old version of django 96.6, but I'm also interested in the solutions for the current version.
Regarding the duplication of other questions:
Here is how the question differ from the duplicates suggested by S.Lott in comments:
Correct place to put extra startup code in django?
Django need to be fully initialized when the function is ran. So code in manage.py won't work.
Where should I place the one-time operation operation in the Django framework?
The function initialize the connection between my applications. So the code must be ran in each thread that will actually handle the requests.
Comments to current solutions:
I can't use urls as most of my apps don't have any urls exposed. They just listen to signals and store additional information in the database.

Signals, specifically, are recommended to be put in the models.py of your app.
Try models.py or urls.py and let us know if you have any luck.

The best place for stuff like this... anywhere, just import it in your urls.py file (for obvious reasons urls are loading before any requests).

If you don't provide urls, then you really need to put it in models.py, that's just the way it is.
Now, on to your problems: You want to define it in its own module, great, do that. To avoid a circular import, use django.db.models.get_model to return the model dynamically for you. You can provide an initialisation function for your signals module to import the relevant model and connect the relevant signals. This function would then be called at the end of models.py, being run only ever once and after your model is initialised.
There's still a chance that this wont work (if the models aren't yet ready when you set it up), but give it a try and let us know.

For me, the following works:
In init.py:
from . import models
from . import signals
signals.py imports from models, but not vice versa. signals.py contains module code that is run immediately when it is imported and is thus run when the django server starts.