We Keep Coding

Does Django's singleton architecture make it unworkable as a standalone ORM in a library?

I like the Django ORM. It's simple, easy to use, and reasonably powerful.
I'm currently developing some internal sites for the VFX company I work for, for which I've used Django. In the meantime, we are developing other python applications and libraries to be used in various contexts in production. There's a number of places in which our core library needs to be interacting with some databases, and using an ORM like Django would really help things. I'm aware of other options like SqlAlchemy or PeeWee, but I'd like to see if Django will work since I use it on the websites and I like its API better.
Using Django as an ORM in a library is tricky (as I explored in a previous question), because Django expects to be used as a website with "apps". In a library, I might want to define any number of data models, which would exist in appropriate places in the library but not inside any Django app (as we're not using any other parts of the framework). So far so good.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is a glaring use case where this all falls apart. Say that my Django web application wants to use some functionality from the company core python library, which now uses the Django ORM for various things. Since I make a call to django.conf.settings.configure in the library, Django is going to scream about defining the settings more than once when it tries to run the main application.
So basically, a library using the Django ORM is incompatible with Django. Wonderful.
Is there any way around this? I mean, it's a lovely ORM - is it really this impossible to use in a standalone modular way? Is the Django architecture utterly singleton in nature, making this impossible?
*Not a duplicate
I'm trying to have a company python library that uses Django as an ORM. Some of the things that could depend on it might be Django websites themselves. How do I get around Django's singleton insistence on only setting the settings config once? Or is it possible? None of these answers address this!

You can check if django has already been configured.
from django.apps import apps
from django.conf import settings
if not apps.ready:
settings.configure()
django.setup()
When starting Django application - core python library can be configured as separate app an be loaded on startup.
Also, check this answer on dynamic app loading at runtime.

A simple answer is how to initialize Django in a standalone application and do it compatible with Django applications.
import os
import django
if not 'DJANGO_SETTINGS_MODULE' in os.environ:
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysettings'
# # or without DJANGO_SETTINGS_MODULE directly
# from django.conf import settings
# settings.configure(DATABASES=... other...)
django.setup()
# this shouldn't be before DJANGO_SETTINGS_MODULE or settings.configure(...)
from myapp.models import MyModel
# this shouldn't be called before django.setup()
queryset = MyModel.objects.filter(...)
This is more compatible with Django then the answer by Oleg Russkin (where a risk of cyclic dependency at django.setup() is possible if the code is called inside inside a setup() initiated by another similar code or a normal Django project started by manage. It is similar to manage.py where django.setup() is also called internally by execute_from_command_line(sys.argv). The setup initializes all modules related to INSTALLED_APPS all urls modules and consequently all views etc. Many of them are lazy, but still. If any code called by setup() depends on this then neither the condition not apps.ready doesn't help. The setup() is not reentrant and the startup fails.)
Much more general answer
An important concept of Django is to support writing reusable parts of code ("applications" in Django terminology, that can be developed and tested independently. There can be also a tree of dependencies, but uncontrolled mutual dependencies should be avoided if possible) Reusable applications are expected that they can be easier combined to whole project ("project" in Django terminology is with all settings necessary to run it by Python.)
The only unavoidable and useful "singleton" in Django ORM are database connections django.db.connections and django.conf.settings especially INSTALLED_APPS. Only one connection should be used to the same database from the same process or thread.
Django is very configurable. An extreme example: It is possible to write a single file project where all code like settings, models, URL configs and views is defined in one file. That extreme that is probably useful only for some special tests or very short demos or as an exercise. It is even possible to define a project by one file with two "reusable" applications :-)
Django supports also "legacy databases" where the database structure is shared with existing non Django applications and models can be created by inspectdb command. Table names in such models are explicit and don't contain the app name. On the other hand the app name prefix is useful to prevent a possible conflict of the same table names from independent "applications". An important decision is if you can use it as a "legacy" database or a normal Django database.
You can decide between following two solutions or to combine them:
Use e.g. foo_models or bar.models and import all models from them e.g. to app.models and add only that "app" to INSTALLED_APPLICATIONS. This can be viable if it is only for one company and never otherwise and central name assigment is possible. (easiest but little naive)
Use some coarse separation of namespaces to several apps. You should probably use not more than one app with simple names without app name prefix.
Think ahead about migrations**. They will be probably very complicated and very soon impossible if you will create later more projects for the same database and different subsets of database tables without separating them to more apps and without app namespace.
There is really no "singleton" in Django ORM except of django.db.connections itself. If you use more databases you can direct some tables to a specific database by DATABASE_ROUTERS, even with two different models that use the same table name without a prefix.

How do you actually use a reusable django app in a project?

This question has been troubling me for some days now and I've tried asking in many places for advice, but it seems that nobody can answer it clearly or even provide a reference to an answer.
I've also tried searching for tutorials, but I just cannot find any type of tutorial that explains how you would use a reusable third-party django app (most tutorials explain how to write them, none explain how to use them).
Also, I've taken a look here:
How to re-use a reusable app in Django - it doesn't explain how to actually use it IN a project itself
and here:
How to bind multiple reusable Django apps together? - the answer by aquaplanet kind of makes sense, but I thought I would ask this question to solve the mental block I am facing in trying to understand this.
In order to best explain this, let me do so by example (note, it is not something I am actually building).
I am creating a project that acts like Reddit. I will have users, links and voting/points. Based on this crude example, I will want to reuse 3 (arbitrary) third-party apps: user, voting/points and links.
I decide to use each of them as any other python package (meaning that they will be treated as a package and none of their code should be touched) [would this method actually work? Or do you have to be able to edit third-party apps to build a project??)
With these apps now within my project, I will use a main app to handle all the template stuff (meaning everything I see on the frontend will be in a single app).
I will then either use that same main app for custom logic (in views.py) or I will break up that logic among different apps (but will still use a single frontend-only app).
From the 3 paragraphs above, is this structure applicable (or can it work) ?
Now lets say that this structure is applicable and I am using a single main app for the frontend and custom logic.
What would I write in models.py? How would I integrate things from the 3 reusable apps into the main models.py file?
How would I reference the reusable apps in views.py? Lets take the example of contrib.auth
With this built-in app, for logging out I would write:
from django.contrib.auth import logout
from django.contrib.auth.decorators import login_required
from django.shortcuts import redirect
#login_required
def user_logout(request):
logout(request)
return redirect('/home/')
Although the above code is simple, is that basically how things would be done with any reusable app?
My question is very long, but I think that this reusable app issue is something a lot of developers aren't quite clear about themselves and maybe this answer will help a lot of others who have heard about the promises of reusable apps, but fail to understand how to actually use them.

TL;DR:
Nope & it depends...
Some (Very) Common Reusable Apps
django.contrib.admin
django.contrib.auth
django.contrib.staticfiles
... those are all reusable Django apps, that happen to be shipped with Django (most of them were not, at some point in time)
Ok, some other reusable apps that don't ship with Django:
django-rest-framework
django-registration
South
Those are all truly reusable apps, and nothing less. There are very many more apps like that.
How do they do it?
To me your question looks more like "how do I build reusable apps", then "how to use them". Actually using them is very different from app to app, because they do very different things. There is only one rule: RTFM No way around that either.
Often, they rely on one or more of the following:
additional value(s) in settings.py
addition (usually one include statement) to urls.py
subclassing and/or mixins for Models, Forms, Fields, Views etc.
template tags and/or filters
management commands
...
Those are all powerful ways though which your app can provide functionality to other apps. There is no recipe (AFAIK) to make a reusable app, because there are so many different scenarios to consider. It all depends on what exactly your app should do.
Reusable apps provide functionalities
I'd argue that it's important to not think of reusable apps as "working together" with other app, but instead recognize that that they "provide functionality." The details of the functionality provided should dictate the way the target developer is supposed to use your library.
Not everything should be reusable
Obviously enough, even though many apps can "in principle" be reusable, it often makes little sense to do so, because it is way faster to clump things together (and make them just "work together").

I'm not sure why you think you need a main app for the "frontend" stuff. The point of a reusable app is that it takes care of everything, you just add (usually) a single URL to include the urls.py of the app, plus your own templates and styling as required.
And you certainly don't need to wrap the app's views in your own views, unless you specifically want to override some functionality.
I don't understand at all your question about models. There's no such thing as a "main" models file, and using a reusable app's models is just the same as using models from any of your own apps.
Normally you would not edit a third-party app, that would make it very hard to integrate updates. Just install the app in your virtualenv (you are using virtualenv, of course!) with pip, which will put it in the lib directory, and you can reference it just like any other app. Make sure you add it to INSTALLED_APPS.

business logic in Django

I'd like to know where to put code that doesn't belong to a view, I mean, the logic.
I've been reading a few similar posts, but couldn't arrive to a conclusion.
What I could understand is:
A View is like a controller, and lot of logic should not put in the controller.
Models should not have a lot of logic either.
So where is all the logic based stuff supposed to be?
I'm coming from Groovy/Grails and for example if we need to access the DB or if we have a complex logic, we use services, and then those services are injected into the controllers.
Is it a good practice to have .py files containing things other than Views and Models in Django?
PS: I've read that some people use a services.py, but then other people say this is a bad practice, so I'm a little confused...

I don't know why you say
we can't put a lot of logic in the controller, and we cannot have the models with a lot of logic either
You can certainly put logic in either of those places. It depends to a great extent what that logic is: if it's specifically related to a single model class, it should go in the model. If however it's more related to a specific page, it can go in a view.
Alternatively, if it's more general logic that's used in multiple views, you could put it in a separate utility module. Or, you could use class-based views with a superclass that defines the logic, and subclasses which inherit from it.

Having a java background I can relate with this question.
I have been working on python for quite some time. Even though I do my best to treat Java as Java and Python as Python, some times I mix them both so that I can get a good deal out of both.
In short
Put all model related stuff in models app, it could be from simply models definition to custom save , pre save hooks .....
Put any request/ response related stuff in views, and some logic like verifying Jon schema, validation request body ... handling exceptions and so on ....
Put your business logic in separate folder/ app or module per views directory/ app. Meaning have separate middle module between your models and views.
There isn't strict rule to organise your code as long as you are consistent.
Project : Ci
Models: ci/model/device.py
Views: ci/views/list_device.py
Business logic:
(1) ci/business_logic/discover_device.py
Or
(2) ci/views/discover_device.py

Short answer: Django is more of a MTV or MVT (Model / Template / View), as described in the official FAQ : https://docs.djangoproject.com/en/dev/faq/general/#django-appears-to-be-a-mvc-framework-but-you-call-the-controller-the-view-and-the-view-the-template-how-come-you-don-t-use-the-standard-names
The business logic has its place in your views, but nothing prevents you from putting it inside a "utils.py", "services.py" or anything to your liking.

If the functionality fits well as a method of some model instance, put it there. After all, models are just classes.
Otherwise, just write a Python module (some .py file) and put the code there, just like in any other Python library.
Don't put it in the views. Views should be the only part of your code that is aware of HTTP, and they should stay as small as possible.

django package organization

I plan to build my project in Django framework. However, I noticed that all Django packages have models.py file. Now, let say I have a set of general purpose functions that I share between several apps in the project and I plan to put these functions definitions in a separate package (or app for that matter?). So, should I create an app "general" and copy-paste these functions into the models.py file? Or can I just create a general.py file in the "general" app directory and leave models.py empty? What is the "Django" way to do that?
Thanks.

models.py file is used to define the structure of database. So you should leave it for defining your database entries. You can make an app named generals and put general.py in that app, and from there you can use it by calling it in any app.

I usually create a utils.py file under my main app that is created from the django-admin.py when starting the project.

I plan to put these functions definitions in a separate package (or app for that matter?)
Before you decide to make this an app (and if you do decide to make it an app), I recommend you take a look at James Bennet keynote on Developing reusable apps and hist post on laying out an application. From one of his slides:
Should this be its own application?
Is it orthogonal to whatever else I’m doing?
Will I need similar functionality on other sites?
Yes? Then I should break it out into a separate application.
If you're cramming too much functionality in one single general purpose app, it might be better to split your general purpose app into multiple reusable apps.
Going back to your original question, Django is expecting a models.py file in every app. So you must have the file even if it's empty.
Inside your models.py, you should only have the application’s model classes. So, you wouldn't be following a best practice if you put inside models.py some miscellaneous code you want to reuse.
From the laying out an application post I mentioned before:
At the application level, I usually drop in a few more files depending on exactly what the application is going to be using:
If the application defines any custom manipulators, I put them in a file called forms.py instead of in the views file.
If there are multiple custom managers in the app, I put them in a file called managers.py instead of the models file.
If I’m defining any custom context processors, I put them in a file called context_processors.py.
If I’m setting up any custom dispatcher signals, they go in a file called signals.py.
If the application is setting up any syndication feeds, the feed classes go in a file called feeds.py. Similarly, sitemap classes go in sitemaps.py.
Middleware classes go in a file called middleware.py.
Any miscellaneous code which doesn’t clearly go anywhere else goes in a file or module called utils.
All of this does not answer directly your original question:
can I just create a general.py file in the "general" app directory and leave models.py empty?
But I hope this gives you additional information to make a decision that better fits your project and requirements.

How do you manage your Django applications?

I just wanted to try to build a project with django. Therefore I have a (basic) question on how to manage such a project. Since I cannot find any guidelines or so on how to split a project into applications.
Let's take a kind of SO as an example. Which applications would you use?
I'd say there should be the applications "users" and "questions". But what if there was a topic system with static articles, too. Maybe they also could receive votes.
How to build the apps structure then? One app for "questions", "votes" and "topics" or just one app "content"?
I have no idea what to do. Maybe it's because I know not very much about Django yet, but I'm interested either...

There aren't hard-and-fast rules, but I would say it's better to err on the side of more specialized applications. Ideally an application should handle just one functional concern: i.e. "tagging" or "commenting" or "auth/auth" or "posts." This type of design will also help you reuse available open source applications instead of reinventing the wheel (i.e. Django comes with auth and comments apps, django-tagging or django-taggable can almost certainly do what you need, etc).
Generic foreign keys can help you decouple applications such as tagging or commenting that might be applied to models from several other applications.

You should try and separate the project in as much applications as possible. For most projects an application will not contain more than 5 models. For example a project like SO would have separate applications for UsersProfiles, Questions, Tags (there's a ready one in django for this), etc. If there was a system with static pages that'd be a separate application too (there are ready ones for this purpose). You should also try and make your applications as generic as possible, so you may reuse them in other projects. There's a good presentation on reusable apps.

Just like any set of dependencies... try to find the most useful stand-alone aspects of the project and make those stand-alone apps. Other Django Apps will have higher level functionality, and reuse the parts of the lowest level apps that you have set up.
In my project, I have a calendar app with its own Event object in its models. I also have a carpool database set up, and for the departure time and the duration I use the calendar's Event object right in my RideShare tables. The carpooling database is calendar-aware, and gets all the nice .ics export and calendar views from the calendar app for 'free.'
There are some tricks to getting the Apps reusable, like naming the templates directory: project/app2/templates/app2/index.html. This lets you refer to app2/index.html from any other app, and get the right template. I picked that one up looking at the built-in reusable apps in Django itself. Pinax is a bit of a monster size-wise but it also demonstrates a nice reusable App structure.
If in doubt, forget about reusable apps for now. Put all your messages and polls in one app and get through one rev. You'll discover during the process what steps feel unnecessary, and could be broken out as something stand-alone in the future.

A good question to ask yourself when deciding whether or not to write an app is "could I use this in another project?". If you think you could, then consider what it would take to make the application as independent as possible; How can you reduce the dependancies so that the app doesn't rely on anything specific to a particular project.
Some of the ways you can do this are:
Giving each app its own urls.py
Allowing model types to be passed in as parameters rather than explicitly declaring what models are used in your views. Generic views use this principle.
Make your templates easily overridden by having some sort of template_name parameter passed in your urls.py
Make sure you can do reverse url lookups with your objects and views. This means naming your views in the urls.py and creating get_absolute_url methods on your models.
In some cases like Tagging, GenericForeignKeys can be used to associate a model in your app to any other model, regardless of whether it has ForeignKeys "looking back" at it.

I'll tell you how I am approaching such question: I usually sit with a sheet of paper and draw the boxes (functionalities) and arrows (interdependencies between functionalities). I am sure there are methodologies or other things that could help you, but my approach usually works for me (YMMV, of course).
Knowing what a site is supposed to be is basic, though. ;)