We Keep Coding

Django settings: sharing ORM with a standalone module

Introduction:
I am learning Django as I develop my first project using PyCharm with Python 3.8 and Postgresql.
I have a problem with the settings.
Environment description:
My aim is to have a Django project for a web site connected to a database (let's call it "web_db"), which is fed at regular intervals with data from an external source (another database, let's call it "source_db").
For this ETL process from one database to the other one, I am developing a Python module called "source2web". I read the content of source_db using psycopg2.
I am developing everything inside one PyCharm project.
Now, since I will be using Django ORM to access the content of web_db from the Django application server, I concluded I might as well use Django ORM in source2web to load my content into web_db.
This means I am trying to share some code between the Django project and source2web: the ORM models of Django.
source2web is simply launched as a python module ( python source2web.py).
Inside my PyCharm project, I have a src folder inside which are all my python modules, with a tree like this (folders are noted with parenthesis) :
(src)
|--(project_name)
|--(module1)
| |--source2web.py
|
|--(django)
|--(django_project)
|--(django_project)
|--(another_app)
The folders django_project/django_project and django_project/another_app have been created classicaly by Django with the commands django-admin startproject django_project and python3 manage.py startapp another_app.
The ORM models I want to share are the model classes inside the file django_project/another_app/models.py
The problem:
I tinkered to find a way to use the Django ORM models from source2web. I thought I had it : inside the code of source2web, I added these lines :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django
django.setup()
But then, I realize that the Django server and the standalone module source2web can both work, but not with the exact same content inside project_name.django.django_project.django_project.settings.py : the name of another_app in the list of INSTALLED_APPS must be different.
Django requires the app to be listed as just another_app
source2web requires the app to be listed as project_name.django.django_project.another_app
I am sure I could find a 'dirty way' to solve this, but I also think I am not the first one to face this problem (sharing Django ORM code between a Django project and a standalone program) and I hope there is cleaner way to manage this.
What I can think of is duplicating the module project_name.django.django_project.django_project.settings and then just change the settings call inside source2web ( os.environ['DJANGO_SETTINGS_MODULE'] = ... ) but I never like duplicating code and maybe there is a cleaner than that ?

I have found at least one decently clean workaround :
In the project_name.django.django_project.django_project.settings file, I have left the value of INSTALLED_APPS that work with Django, i.e., the list of default apps created with the Django Project plus another_app.
Then in the source2web module, I have changed my code for initializing Django ORM :
In module source2web, I added 2 lines :
import project_name.django.django_project.django_project.settings as djangosettings
djangosettings.INSTALLED_APPS = ['project_name.django.django_project.another_app',]
os.environ['DJANGO_SETTINGS_MODULE'] = 'project_name.django.django_project.django_project.settings'
import django`
django.setup()
This way, I leave untouched the settings for the Django server, and only override what's necessary in the source2web module (i.e. the value of INSTALLED_APPS, getting rid at the same time of all the default apps I don't need for my ETL process).
It works and seems robust enough.
Still, it brought me to question the whole project structure :
Because obviously, the whole problem is the result of the PyCharm project considering for source root src/, while the Django project considers for root project_name/django/django_project/ .
There should be a way to reconcile them, but how ?

Can I have a main models file in a django project?

The Problem
I'm working on a django project in which I need to share the same database models over multiple apps, for example when a user opens a page the page is displayed depending on user settings, which can be changed by the user in a different app than the one which displayes the page
At the moment I've made the browser app which contains most of my models (because it's the one I started with) and I've got other apps like watch_file and user_settings which then import
This is working but I'm noticing that I'm running into organization problems where I'm hopping from file to file to check models and I'm importing all over the place...
My potential Solution
I was thinking about making one big model file somewhere and just importing the models I need for every app in it's own model file, however, I read this question in which someone in the comments stated that generally this is not done
Also I red this question which said that sharing models between apps isn't a good idea at all in regards to organization
The question
In regards to it not being a good idea to make a general model file for all models my project is using, How would I go about organizing it so I don't have to import so many models from different files for an app to work?
Could I create a something like a helper app which isn't displayed but is only used to help other apps by having all models in it's models.py file and importing it frome here in other apps?

NB : when talking about "apps" in the following, I mean "project-specific apps", not "reusable apps".
It's perfectly legal (technically) and hardly avoidable in any non-trivial project to have inter-packages dependencies (Django "apps" are first and foremost python packages), so having one app importing models from another app is definitly not a problem in and by itself. Where it becomes a problem is when you start having circular dependencies (A depends on B which depends on C which depends on A...)
A pattern that usually works fine is to have a "core" app on which all other apps can depend but that must not depends on any other app (except contribs app or third-part apps of course), and a "main" app that can depend on any app but that no other app is allowed to depend on. The core app provides the "foundations" for your project - the basic building blocks that all the project will need - and the main app (which usually has no models at all) is responsible for "top-level integration" of everthing (IOW it's where you put everything that doesn't clearly belongs to another app and "bridges" the other apps features together).
Then you try to keep other apps mostly self-contained except for dependencies on contribs apps, third-part apps and the core app.
Also note that an app does not necessarily have models AND views AND templates AND templatetags etc etc - you can have simple apps defining a couple models or utils, a "frontend" app (somehow similar to the "main" app in concept actually) that's responsible for all your project's views, an "api" app providing a REST api over your other apps models, etc etc.

Does Django's singleton architecture make it unworkable as a standalone ORM in a library?

I like the Django ORM. It's simple, easy to use, and reasonably powerful.
I'm currently developing some internal sites for the VFX company I work for, for which I've used Django. In the meantime, we are developing other python applications and libraries to be used in various contexts in production. There's a number of places in which our core library needs to be interacting with some databases, and using an ORM like Django would really help things. I'm aware of other options like SqlAlchemy or PeeWee, but I'd like to see if Django will work since I use it on the websites and I like its API better.
Using Django as an ORM in a library is tricky (as I explored in a previous question), because Django expects to be used as a website with "apps". In a library, I might want to define any number of data models, which would exist in appropriate places in the library but not inside any Django app (as we're not using any other parts of the framework). So far so good.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is a glaring use case where this all falls apart. Say that my Django web application wants to use some functionality from the company core python library, which now uses the Django ORM for various things. Since I make a call to django.conf.settings.configure in the library, Django is going to scream about defining the settings more than once when it tries to run the main application.
So basically, a library using the Django ORM is incompatible with Django. Wonderful.
Is there any way around this? I mean, it's a lovely ORM - is it really this impossible to use in a standalone modular way? Is the Django architecture utterly singleton in nature, making this impossible?
*Not a duplicate
I'm trying to have a company python library that uses Django as an ORM. Some of the things that could depend on it might be Django websites themselves. How do I get around Django's singleton insistence on only setting the settings config once? Or is it possible? None of these answers address this!

You can check if django has already been configured.
from django.apps import apps
from django.conf import settings
if not apps.ready:
settings.configure()
django.setup()
When starting Django application - core python library can be configured as separate app an be loaded on startup.
Also, check this answer on dynamic app loading at runtime.

A simple answer is how to initialize Django in a standalone application and do it compatible with Django applications.
import os
import django
if not 'DJANGO_SETTINGS_MODULE' in os.environ:
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysettings'
# # or without DJANGO_SETTINGS_MODULE directly
# from django.conf import settings
# settings.configure(DATABASES=... other...)
django.setup()
# this shouldn't be before DJANGO_SETTINGS_MODULE or settings.configure(...)
from myapp.models import MyModel
# this shouldn't be called before django.setup()
queryset = MyModel.objects.filter(...)
This is more compatible with Django then the answer by Oleg Russkin (where a risk of cyclic dependency at django.setup() is possible if the code is called inside inside a setup() initiated by another similar code or a normal Django project started by manage. It is similar to manage.py where django.setup() is also called internally by execute_from_command_line(sys.argv). The setup initializes all modules related to INSTALLED_APPS all urls modules and consequently all views etc. Many of them are lazy, but still. If any code called by setup() depends on this then neither the condition not apps.ready doesn't help. The setup() is not reentrant and the startup fails.)
Much more general answer
An important concept of Django is to support writing reusable parts of code ("applications" in Django terminology, that can be developed and tested independently. There can be also a tree of dependencies, but uncontrolled mutual dependencies should be avoided if possible) Reusable applications are expected that they can be easier combined to whole project ("project" in Django terminology is with all settings necessary to run it by Python.)
The only unavoidable and useful "singleton" in Django ORM are database connections django.db.connections and django.conf.settings especially INSTALLED_APPS. Only one connection should be used to the same database from the same process or thread.
Django is very configurable. An extreme example: It is possible to write a single file project where all code like settings, models, URL configs and views is defined in one file. That extreme that is probably useful only for some special tests or very short demos or as an exercise. It is even possible to define a project by one file with two "reusable" applications :-)
Django supports also "legacy databases" where the database structure is shared with existing non Django applications and models can be created by inspectdb command. Table names in such models are explicit and don't contain the app name. On the other hand the app name prefix is useful to prevent a possible conflict of the same table names from independent "applications". An important decision is if you can use it as a "legacy" database or a normal Django database.
You can decide between following two solutions or to combine them:
Use e.g. foo_models or bar.models and import all models from them e.g. to app.models and add only that "app" to INSTALLED_APPLICATIONS. This can be viable if it is only for one company and never otherwise and central name assigment is possible. (easiest but little naive)
Use some coarse separation of namespaces to several apps. You should probably use not more than one app with simple names without app name prefix.
Think ahead about migrations**. They will be probably very complicated and very soon impossible if you will create later more projects for the same database and different subsets of database tables without separating them to more apps and without app namespace.
There is really no "singleton" in Django ORM except of django.db.connections itself. If you use more databases you can direct some tables to a specific database by DATABASE_ROUTERS, even with two different models that use the same table name without a prefix.

django package organization

I plan to build my project in Django framework. However, I noticed that all Django packages have models.py file. Now, let say I have a set of general purpose functions that I share between several apps in the project and I plan to put these functions definitions in a separate package (or app for that matter?). So, should I create an app "general" and copy-paste these functions into the models.py file? Or can I just create a general.py file in the "general" app directory and leave models.py empty? What is the "Django" way to do that?
Thanks.

models.py file is used to define the structure of database. So you should leave it for defining your database entries. You can make an app named generals and put general.py in that app, and from there you can use it by calling it in any app.

I usually create a utils.py file under my main app that is created from the django-admin.py when starting the project.

I plan to put these functions definitions in a separate package (or app for that matter?)
Before you decide to make this an app (and if you do decide to make it an app), I recommend you take a look at James Bennet keynote on Developing reusable apps and hist post on laying out an application. From one of his slides:
Should this be its own application?
Is it orthogonal to whatever else I’m doing?
Will I need similar functionality on other sites?
Yes? Then I should break it out into a separate application.
If you're cramming too much functionality in one single general purpose app, it might be better to split your general purpose app into multiple reusable apps.
Going back to your original question, Django is expecting a models.py file in every app. So you must have the file even if it's empty.
Inside your models.py, you should only have the application’s model classes. So, you wouldn't be following a best practice if you put inside models.py some miscellaneous code you want to reuse.
From the laying out an application post I mentioned before:
At the application level, I usually drop in a few more files depending on exactly what the application is going to be using:
If the application defines any custom manipulators, I put them in a file called forms.py instead of in the views file.
If there are multiple custom managers in the app, I put them in a file called managers.py instead of the models file.
If I’m defining any custom context processors, I put them in a file called context_processors.py.
If I’m setting up any custom dispatcher signals, they go in a file called signals.py.
If the application is setting up any syndication feeds, the feed classes go in a file called feeds.py. Similarly, sitemap classes go in sitemaps.py.
Middleware classes go in a file called middleware.py.
Any miscellaneous code which doesn’t clearly go anywhere else goes in a file or module called utils.
All of this does not answer directly your original question:
can I just create a general.py file in the "general" app directory and leave models.py empty?
But I hope this gives you additional information to make a decision that better fits your project and requirements.

link django with modules of other applications

I would like to link django with an existing python application, reusing the modules, the classes and some configurationfiles that were defined in this application. All modules reside in the ./bin directory of this application. In fact the application is a forecasting system, and django will be used as the frontend to visualise the results.
The current directory structure looks like this.
./application/bin/module1.py
/module2.py
/config.txt
/datadir/...
/webresults/run1/myfig.png
/run2/myfig.png
/...
./django/manage.py
/project1/urls.py
/settings.py
/wsgi.py
/app1/views.py
/models.py
/forms.py
/...
/templates/base.html
/showResults.html
/...
An apacheserver is set up to serve static pages and images from the directory ./application/webresults/ and django pages from /django/project1/wsgi.py.
For the moment I have copied all relevant modules from /application/bin/ to django/app1/ so I can reuse them eg. in views.py and models.py. With respect to maintenance of the system not an optimal solution.
So I am looking for a more elegant solution to solve this. Some of the things we would like to achieve:
moving around the backendapplication is harder than moving around django instance, so preferable django is relocated (it was not coded in the most efficient way :-))
reuse classes of application in django
reuse models of django in the application
reuse SQL_objectmapping in the application
use one configfile for settings relevant to both django and the application
The solution we are thinking of, would be to merge all djangocode into /application/bin/ and remap the djangopath in apacheconfiguration
./application/bin/manage.py
/module1.py
/module2.py
/project1/urls.py
/settings.py
/wsgi.py
/app1/views.py
/models.py
/forms.py
/...
/templates/base.html
/showResults.html
/...
Are there any recipes on how this could be handled? Any advice appreciated.

There's nothing wrong with tossing your Django project within another module. So long as your PYTHONPATH's are setup correctly, and your WSGI is configured right, everything should work normally.
My only comment would be that, tautologically, "separate things should be separate."
When I have similar problems, I usually create a simple UNIX socket server (using Tornado), that Django can communicate with for API calls, etc.
If I need to expose the functionality of the socket server, I'll throw in a cheap WSGI server (again, with Tornado) that serves up requests, and stick it under a different subdomain within my hosting configuration.

I think my original question was not very clear. Since then I stumbled upon some other answers and pages that helped me , mainly on how to reuse django functionality in external modules and applications.
http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
why 'list index out of range' in my django code;
https://docs.djangoproject.com/en/dev/ref/models/options/#app-label
With this information I managed to access my models (in apps-subfolder) from the pythonscripts on the root of the application (eg. module1.py).
In short the code to access app1.models.Mymodel from module1.py looks now like this :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project1.settings'
from django.core.management import setup_environ
from project1 import settings
setup_environ(settings)
import app1.models
class MyChildmodel(app1.models.Mymodel):
class Meta:
app_label = 'app1'
def myfunction():
....