We Keep Coding

Django settings: sharing ORM with a standalone module

Introduction:
I am learning Django as I develop my first project using PyCharm with Python 3.8 and Postgresql.
I have a problem with the settings.
Environment description:
My aim is to have a Django project for a web site connected to a database (let's call it "web_db"), which is fed at regular intervals with data from an external source (another database, let's call it "source_db").
For this ETL process from one database to the other one, I am developing a Python module called "source2web". I read the content of source_db using psycopg2.
I am developing everything inside one PyCharm project.
Now, since I will be using Django ORM to access the content of web_db from the Django application server, I concluded I might as well use Django ORM in source2web to load my content into web_db.
This means I am trying to share some code between the Django project and source2web: the ORM models of Django.
source2web is simply launched as a python module ( python source2web.py).
Inside my PyCharm project, I have a src folder inside which are all my python modules, with a tree like this (folders are noted with parenthesis) :
(src)
|--(project_name)
|--(module1)
| |--source2web.py
|
|--(django)
|--(django_project)
|--(django_project)
|--(another_app)
The folders django_project/django_project and django_project/another_app have been created classicaly by Django with the commands django-admin startproject django_project and python3 manage.py startapp another_app.
The ORM models I want to share are the model classes inside the file django_project/another_app/models.py
The problem:
I tinkered to find a way to use the Django ORM models from source2web. I thought I had it : inside the code of source2web, I added these lines :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django
django.setup()
But then, I realize that the Django server and the standalone module source2web can both work, but not with the exact same content inside project_name.django.django_project.django_project.settings.py : the name of another_app in the list of INSTALLED_APPS must be different.
Django requires the app to be listed as just another_app
source2web requires the app to be listed as project_name.django.django_project.another_app
I am sure I could find a 'dirty way' to solve this, but I also think I am not the first one to face this problem (sharing Django ORM code between a Django project and a standalone program) and I hope there is cleaner way to manage this.
What I can think of is duplicating the module project_name.django.django_project.django_project.settings and then just change the settings call inside source2web ( os.environ['DJANGO_SETTINGS_MODULE'] = ... ) but I never like duplicating code and maybe there is a cleaner than that ?

I have found at least one decently clean workaround :
In the project_name.django.django_project.django_project.settings file, I have left the value of INSTALLED_APPS that work with Django, i.e., the list of default apps created with the Django Project plus another_app.
Then in the source2web module, I have changed my code for initializing Django ORM :
In module source2web, I added 2 lines :
import project_name.django.django_project.django_project.settings as djangosettings
djangosettings.INSTALLED_APPS = ['project_name.django.django_project.another_app',]
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django`
django.setup()
This way, I leave untouched the settings for the Django server, and only override what's necessary in the source2web module (i.e. the value of INSTALLED_APPS, getting rid at the same time of all the default apps I don't need for my ETL process).
It works and seems robust enough.
Still, it brought me to question the whole project structure :
Because obviously, the whole problem is the result of the PyCharm project considering for source root src/, while the Django project considers for root project_name/django/django_project/ .
There should be a way to reconcile them, but how ?

Does Django's singleton architecture make it unworkable as a standalone ORM in a library?

I like the Django ORM. It's simple, easy to use, and reasonably powerful.
I'm currently developing some internal sites for the VFX company I work for, for which I've used Django. In the meantime, we are developing other python applications and libraries to be used in various contexts in production. There's a number of places in which our core library needs to be interacting with some databases, and using an ORM like Django would really help things. I'm aware of other options like SqlAlchemy or PeeWee, but I'd like to see if Django will work since I use it on the websites and I like its API better.
Using Django as an ORM in a library is tricky (as I explored in a previous question), because Django expects to be used as a website with "apps". In a library, I might want to define any number of data models, which would exist in appropriate places in the library but not inside any Django app (as we're not using any other parts of the framework). So far so good.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is a glaring use case where this all falls apart. Say that my Django web application wants to use some functionality from the company core python library, which now uses the Django ORM for various things. Since I make a call to django.conf.settings.configure in the library, Django is going to scream about defining the settings more than once when it tries to run the main application.
So basically, a library using the Django ORM is incompatible with Django. Wonderful.
Is there any way around this? I mean, it's a lovely ORM - is it really this impossible to use in a standalone modular way? Is the Django architecture utterly singleton in nature, making this impossible?
*Not a duplicate
I'm trying to have a company python library that uses Django as an ORM. Some of the things that could depend on it might be Django websites themselves. How do I get around Django's singleton insistence on only setting the settings config once? Or is it possible? None of these answers address this!

You can check if django has already been configured.
from django.apps import apps
from django.conf import settings
if not apps.ready:
settings.configure()
django.setup()
When starting Django application - core python library can be configured as separate app an be loaded on startup.
Also, check this answer on dynamic app loading at runtime.

A simple answer is how to initialize Django in a standalone application and do it compatible with Django applications.
import os
import django
if not 'DJANGO_SETTINGS_MODULE' in os.environ:
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysettings'
# # or without DJANGO_SETTINGS_MODULE directly
# from django.conf import settings
# settings.configure(DATABASES=... other...)
django.setup()
# this shouldn't be before DJANGO_SETTINGS_MODULE or settings.configure(...)
from myapp.models import MyModel
# this shouldn't be called before django.setup()
queryset = MyModel.objects.filter(...)
This is more compatible with Django then the answer by Oleg Russkin (where a risk of cyclic dependency at django.setup() is possible if the code is called inside inside a setup() initiated by another similar code or a normal Django project started by manage. It is similar to manage.py where django.setup() is also called internally by execute_from_command_line(sys.argv). The setup initializes all modules related to INSTALLED_APPS all urls modules and consequently all views etc. Many of them are lazy, but still. If any code called by setup() depends on this then neither the condition not apps.ready doesn't help. The setup() is not reentrant and the startup fails.)
Much more general answer
An important concept of Django is to support writing reusable parts of code ("applications" in Django terminology, that can be developed and tested independently. There can be also a tree of dependencies, but uncontrolled mutual dependencies should be avoided if possible) Reusable applications are expected that they can be easier combined to whole project ("project" in Django terminology is with all settings necessary to run it by Python.)
The only unavoidable and useful "singleton" in Django ORM are database connections django.db.connections and django.conf.settings especially INSTALLED_APPS. Only one connection should be used to the same database from the same process or thread.
Django is very configurable. An extreme example: It is possible to write a single file project where all code like settings, models, URL configs and views is defined in one file. That extreme that is probably useful only for some special tests or very short demos or as an exercise. It is even possible to define a project by one file with two "reusable" applications :-)
Django supports also "legacy databases" where the database structure is shared with existing non Django applications and models can be created by inspectdb command. Table names in such models are explicit and don't contain the app name. On the other hand the app name prefix is useful to prevent a possible conflict of the same table names from independent "applications". An important decision is if you can use it as a "legacy" database or a normal Django database.
You can decide between following two solutions or to combine them:
Use e.g. foo_models or bar.models and import all models from them e.g. to app.models and add only that "app" to INSTALLED_APPLICATIONS. This can be viable if it is only for one company and never otherwise and central name assigment is possible. (easiest but little naive)
Use some coarse separation of namespaces to several apps. You should probably use not more than one app with simple names without app name prefix.
Think ahead about migrations**. They will be probably very complicated and very soon impossible if you will create later more projects for the same database and different subsets of database tables without separating them to more apps and without app namespace.
There is really no "singleton" in Django ORM except of django.db.connections itself. If you use more databases you can direct some tables to a specific database by DATABASE_ROUTERS, even with two different models that use the same table name without a prefix.

Will using the Django ORM in a python library cause trouble?

I've enjoyed using Django quite a bit over the years. Right now I'm working for a company that is building some shared internal libraries for accessing information from our databases. Right now things are terribly sloppy - lots of inline SQL code. Some colleagues were working on doing some accessing code, and it occurred to me that this is the sort of thing that I'm used to Django doing out of the box. I had also heard that Django is fundamentally modular, so that I could just use the ORM system in isolation. I know there are other packages for this like SqlAlchemy, but I also hear that Django does things easier for the average case, which is likely all we'll need. I also know Django, and don't know SQLAlchemy.
So, I can do a proof of concept, like the following. Here is the directory structure:
+-- example.py
+-- transfer
| +-- __init__.py
| +-- models.py
Here is models.py
import django.conf
import django.db.models
django.conf.settings.configure(
DATABASES = ..., # database info
INSTALLED_APPS = ("transfer",),
SECRET_KEY = 'not telling',
)
django.setup()
class TransferItem(django.db.models.Model)
# model info here
example.py
from transfer.models import TransferItem
items = TransferItem.objects.all()
print items
This seems to work fine, as far as it goes. But I'm worried about the bootstrap code in a library context. Specifically:
Is there danger in django thinking of this as an app? In this case, "transfer" is a root module, but in a library context this could be buried deep. For example, "mycompany.data.protocols.transfer". Theoretically, we could have these data models defined throughout the codebase. How can this "app list" scale?
The call to setup really worries me. The django docs specifically say only to call setup once. And yet the nature of a library is that any python application could import whatever type of data model they want. I can't make any assumptions about which django "apps" they might want, or what order they want it in. Would if one type of model is used, data is returned, and only then does the python application decide it needs to import another type of model (quite possibly in a different "app")?
So, the fundamental question is this:
Is there a good way to use django ORM in a python library?
EDIT: This is not a duplicate of the CLI tool question. I know how to run django outside the web server. I even gave code samples showing this. I want to know if there's a way to run django where I don't have "apps" per se - where model files could be imported by client code and used in any order.

OK, here's my own attempt to answer the question after some research.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is still a big concern I have which might be lethal. Django's initialization itself is still singleton in nature. If my library were itself imported by a Django application, it would all crash and burn. I've asked for solutions to this in this follow up question.

use Django database models externally

My site is named ficosa. I have a Django server set up with a sqlite database that contains a table named core_Data. Now, I am developing a python file named serverMQTT.py that should insert data into that sqlite database. This file is outside Django so in order to import the Django models from ficosa site I call django.setup()
import django
from django.conf import settings
from ficosa import settings as fsettings
settings.configure(default_settings=fsettings, DEBUG=True)
django.setup()
Now this script or any imported module can use any part of Django it needs.
from core.models import Data
However, I am having an error:
AttributeError: 'module' object has no attribute 'LOGGING_CONFIG'
I would be gratefull if sombody could help me

That's not how you configure settings in a standalone app. Note what the documentation says:
Be aware that if you do pass in a new default module, it entirely replaces the Django defaults, so you must specify a value for every possible setting that might be used in that code you are importing.
Presumably, your fsettings module only contains database settings. In which case, just override that one thing:
settings.configure(DATABASES=fsettings.DATABASES, DEBUG=True)

How to use custom AdminSite class?

Which is the best way to implement my own django.contrib.admin.sites.AdminSite?
Actually I get a problem with the registration of INSTALLED_APPS in django.contrib.admin.autodiscover. If I use my custom AdminSite class in urls.py, there were no apps displayed on the admin page.
I fixed this with a litte hack. I wrote this class:
from django.contrib.admin.sites import site as default_site
class AdminSiteRegistryFix( object ):
'''
This fix links the '_registry' property to the orginal AdminSites
'_registry' property. This is necessary, because of the character of
the admins 'autodiscover' function. Otherwise the admin site will say,
that you havn't permission to edit anything.
'''
def _registry_getter(self):
return default_site._registry
def _registry_setter(self,value):
default_site._registry = value
_registry = property(_registry_getter, _registry_setter)
And implement my custom AdminSite like this:
from wltrweb.hacks.django.admin import AdminSiteRegistryFix
from django.contrib.admin import AdminSite
class MyAdminSite( AdminSite, AdminSiteRegistryFix ):
# do some magic
pass
site = MyAdminSite()
So I can use this site for urls.py.
Anyone knows a better way? Since I access a var starting with a underscore it is no more than a hack. I don't like hacks.
Edit: Another way would be to rewrite the django.contrib.admin.autodiscover function, but in this case I would have redundant code.

The Problem
Using a custom class derived from django.contrib.admin.AdminSite for the admin site of a project, without having to write custom registration code to register models with the new class. When I use 3rd party apps with their own models, I'd rather not have to edit custom registration code only because models were added or removed from these apps.
The Solution
You have to switch the instance created with the default class used for the admin site to your own instance, created with your own class before django.contrib.admin's autodiscover function is called. I do this by:
Having an app that will perform the switch. (I use my project-specific app named core for my own purposes.)
Two choices:
Django 1.6 to 1.9: use __init__ of the app to perform the switch. In Django 1.8, you will get a deprecation warning due to the change in Django 1.9 quoted below. Note that this method will work with 1.9 too because the Django modules loaded by the code shown below have been changed in 1.9 so that they no longer load models. When I use this method my core/__init__.py file contains:
from django.contrib import admin
from django.contrib.admin import sites
class MyAdminSite(admin.AdminSite):
pass
mysite = MyAdminSite()
admin.site = mysite
sites.site = mysite
Django 1.9 and over: use the app configuration of the app to perform the switch. As of Django 1.9, as the release notes state:
All models need to be defined inside an installed application or declare an explicit app_label. Furthermore, it isn’t possible to import them before their application is loaded. In particular, it isn’t possible to import models inside the root package of an application.
I prefer to limit the imports I do at the root level to avoid the risk of loading models. While as of version 1.9 using the __init__ method above will work, there's no telling if 1.10 or a later version will introduce a change that will cause problems.
When I use this method the core/__init__.py sets default_app_config = "core.apps.DefaultAppConfig" and I have a core/apps.py like this:
from django.apps import AppConfig
class DefaultAppConfig(AppConfig):
name = 'core'
def ready(self):
from django.contrib import admin
from django.contrib.admin import sites
class MyAdminSite(admin.AdminSite):
pass
mysite = MyAdminSite()
admin.site = mysite
sites.site = mysite
While it is possible to use this method with versions 1.7 and 1.8, it is a bit risky to use it with those versions. See the notes below.
Placing this app earlier than django.contrib.admin in the INSTALLED_APPS list. (This is absolutely necessary for 1.7 and later. In earlier versions of Django, it might work okay even if the app is later than django.contrib.admin. However, see the notes below.)
Notes and Caveats
The app that performs the switch should really be the first app in the INSTALLED_APPS list so as to minimize the chance that something else will grab the value of site from django.contrib.admin before the switch is made. If another app manages to get that value before the switch is done, then that other app will have a reference to the old site. Hilarity will surely ensue.
The method above won't work nicely if two apps are trying to install their own new default admin site class. This would have to be handled on a case-by-case basis.
It is possible that a future release of Django could break this method.
For version prior to 1.9, I preferred using __init__ to do site switch over using the app configuration because the documentation on initialization indicates that the ready() method of the app configurations is called relatively late. Between the time an app's module is loaded, and the time ready() is called, models have been loaded, and in some case, it could mean that a module has grabbed the value of site from django.contrib.admin before ready is called. So as to minimize the risk, I have the app's __init__ code do the switch.
I believe the risk that existed in version 1.7 and 1.8 and that I avoided by using __init__ to perform the site switch as early as possible does not exist in 1.9. Everybody is prohibited from loading modules before all the applications are loaded. So doing the switch in the ready callback of the first application listed in INSTALLED_APPS should be safe. I've upgraded a large project to 1.9 and used the app configuration method, without any problem.

From Django 2.1, there is an 'out-of-the-box' solution: https://docs.djangoproject.com/en/2.1/ref/contrib/admin/#overriding-the-default-admin-site
from django.contrib import admin
class MyAdminSite(admin.AdminSite):
...
Swapping the custom admin site is now done by adding your own AdminConfig to installed apps.
from django.contrib.admin.apps import AdminConfig
class MyAdminConfig(AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
INSTALLED_APPS = [
...
'myproject.apps.MyAdminConfig', # replaces 'django.contrib.admin'
...
]
Note the difference between AdminConfig and SimpleAdminConfig, where the latter doesn't trigger admin.autodiscover(). I'm currently using this solution in a project.

Having encountered the same kind of issue while implemeting site-wide custom AdminSite in Django 3.2, I found a workaround to make it work.
It seems the documentation should be updated accordingly.
from django.contrib.admin.apps import AdminConfig
class MyAdminConfig(AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
This raises an exception:
RuntimeError: 'myproject.apps' declares more than one default AppConfig: 'AdminConfig', 'MyAdminConfig'.
It is solved by importing django.contrib.admin.apps instead of django.contrib.admin.apps.AdminConfig:
from django.contrib.admin import apps
class MyAdminConfig(apps.AdminConfig):
default_site = 'myproject.admin.MyAdminSite'
Then the exception moves on:
django.core.exceptions.ImproperlyConfigured: Application labels aren't unique, duplicates: admin
This is caused by the settings.py configuration:
INSTALLED_APPS = [
'myproject.apps.MyAdminConfig', #replaces django.contrib.admin
...
It is solved by removing the 'MyAdminConfig' from the settings:
INSTALLED_APPS = [
'myproject.apps', #replaces django.contrib.admin
...

Quoting from https://docs.djangoproject.com/en/1.10/ref/contrib/admin/#customizing-the-adminsite-class
If you'd like to set up your own administrative site with custom behavior, however, you're free to subclass AdminSite and override or add anything you like. Then, simply create an instance of your AdminSite subclass (the same way you'd instantiate any other Python class), and register your models and ModelAdmin subclasses with it instead of using the default.
I guess that is the most explicit approach, but it also means that you need to change the register code in your apps admin.py files.
There is really no need to use autodiscover when using your own AdminSite instance since you will likely be importing all the per-app admin.py modules in your myproject.admin module.
The assumption seems to be, that once you start writing your custom admin site, it becomes pretty much project specific and you know beforehand which apps you want to include.
So if you don't want to work with the hack above, I only really see these two options. Replace all register calls to your custom admin site or register the models explicitly in your adminsite module.