I've enjoyed using Django quite a bit over the years. Right now I'm working for a company that is building some shared internal libraries for accessing information from our databases. Right now things are terribly sloppy - lots of inline SQL code. Some colleagues were working on doing some accessing code, and it occurred to me that this is the sort of thing that I'm used to Django doing out of the box. I had also heard that Django is fundamentally modular, so that I could just use the ORM system in isolation. I know there are other packages for this like SqlAlchemy, but I also hear that Django does things easier for the average case, which is likely all we'll need. I also know Django, and don't know SQLAlchemy.
So, I can do a proof of concept, like the following. Here is the directory structure:
+-- example.py
+-- transfer
| +-- __init__.py
| +-- models.py
Here is models.py
import django.conf
import django.db.models
django.conf.settings.configure(
DATABASES = ..., # database info
INSTALLED_APPS = ("transfer",),
SECRET_KEY = 'not telling',
)
django.setup()
class TransferItem(django.db.models.Model)
# model info here
example.py
from transfer.models import TransferItem
items = TransferItem.objects.all()
print items
This seems to work fine, as far as it goes. But I'm worried about the bootstrap code in a library context. Specifically:
Is there danger in django thinking of this as an app? In this case, "transfer" is a root module, but in a library context this could be buried deep. For example, "mycompany.data.protocols.transfer". Theoretically, we could have these data models defined throughout the codebase. How can this "app list" scale?
The call to setup really worries me. The django docs specifically say only to call setup once. And yet the nature of a library is that any python application could import whatever type of data model they want. I can't make any assumptions about which django "apps" they might want, or what order they want it in. Would if one type of model is used, data is returned, and only then does the python application decide it needs to import another type of model (quite possibly in a different "app")?
So, the fundamental question is this:
Is there a good way to use django ORM in a python library?
EDIT: This is not a duplicate of the CLI tool question. I know how to run django outside the web server. I even gave code samples showing this. I want to know if there's a way to run django where I don't have "apps" per se - where model files could be imported by client code and used in any order.
OK, here's my own attempt to answer the question after some research.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is still a big concern I have which might be lethal. Django's initialization itself is still singleton in nature. If my library were itself imported by a Django application, it would all crash and burn. I've asked for solutions to this in this follow up question.
Related
I like the Django ORM. It's simple, easy to use, and reasonably powerful.
I'm currently developing some internal sites for the VFX company I work for, for which I've used Django. In the meantime, we are developing other python applications and libraries to be used in various contexts in production. There's a number of places in which our core library needs to be interacting with some databases, and using an ORM like Django would really help things. I'm aware of other options like SqlAlchemy or PeeWee, but I'd like to see if Django will work since I use it on the websites and I like its API better.
Using Django as an ORM in a library is tricky (as I explored in a previous question), because Django expects to be used as a website with "apps". In a library, I might want to define any number of data models, which would exist in appropriate places in the library but not inside any Django app (as we're not using any other parts of the framework). So far so good.
I can create a baseclass for my models anywhere in the library as follows:
from django.db import models
from django.apps import apps
import django.conf
django.conf_settings.configure(
DATABASES = ...
)
apps.populate((__name__,))
class LibModel(models.Model):
class Meta:
abstract = True
app_label = __name__
Then anywhere in the library I can create my own models with this baseclass. Since I'm not relying on the "app" for the database names, I need to state them explicitly.
class SpecificModel(LibModel):
# fields go here
class Meta(LibModel.Meta):
db_table = "specific_model_table_name"
This gets around my concern of having to simulate the structure of an "app". The name property in the base class supplies Django with all it needs, and then Django quits whining about not finding an app. The other model files can live wherever they want.
However, there is a glaring use case where this all falls apart. Say that my Django web application wants to use some functionality from the company core python library, which now uses the Django ORM for various things. Since I make a call to django.conf.settings.configure in the library, Django is going to scream about defining the settings more than once when it tries to run the main application.
So basically, a library using the Django ORM is incompatible with Django. Wonderful.
Is there any way around this? I mean, it's a lovely ORM - is it really this impossible to use in a standalone modular way? Is the Django architecture utterly singleton in nature, making this impossible?
*Not a duplicate
I'm trying to have a company python library that uses Django as an ORM. Some of the things that could depend on it might be Django websites themselves. How do I get around Django's singleton insistence on only setting the settings config once? Or is it possible? None of these answers address this!
You can check if django has already been configured.
from django.apps import apps
from django.conf import settings
if not apps.ready:
settings.configure()
django.setup()
When starting Django application - core python library can be configured as separate app an be loaded on startup.
Also, check this answer on dynamic app loading at runtime.
A simple answer is how to initialize Django in a standalone application and do it compatible with Django applications.
import os
import django
if not 'DJANGO_SETTINGS_MODULE' in os.environ:
os.environ['DJANGO_SETTINGS_MODULE'] = 'mysettings'
# # or without DJANGO_SETTINGS_MODULE directly
# from django.conf import settings
# settings.configure(DATABASES=... other...)
django.setup()
# this shouldn't be before DJANGO_SETTINGS_MODULE or settings.configure(...)
from myapp.models import MyModel
# this shouldn't be called before django.setup()
queryset = MyModel.objects.filter(...)
This is more compatible with Django then the answer by Oleg Russkin (where a risk of cyclic dependency at django.setup() is possible if the code is called inside inside a setup() initiated by another similar code or a normal Django project started by manage. It is similar to manage.py where django.setup() is also called internally by execute_from_command_line(sys.argv). The setup initializes all modules related to INSTALLED_APPS all urls modules and consequently all views etc. Many of them are lazy, but still. If any code called by setup() depends on this then neither the condition not apps.ready doesn't help. The setup() is not reentrant and the startup fails.)
Much more general answer
An important concept of Django is to support writing reusable parts of code ("applications" in Django terminology, that can be developed and tested independently. There can be also a tree of dependencies, but uncontrolled mutual dependencies should be avoided if possible) Reusable applications are expected that they can be easier combined to whole project ("project" in Django terminology is with all settings necessary to run it by Python.)
The only unavoidable and useful "singleton" in Django ORM are database connections django.db.connections and django.conf.settings especially INSTALLED_APPS. Only one connection should be used to the same database from the same process or thread.
Django is very configurable. An extreme example: It is possible to write a single file project where all code like settings, models, URL configs and views is defined in one file. That extreme that is probably useful only for some special tests or very short demos or as an exercise. It is even possible to define a project by one file with two "reusable" applications :-)
Django supports also "legacy databases" where the database structure is shared with existing non Django applications and models can be created by inspectdb command. Table names in such models are explicit and don't contain the app name. On the other hand the app name prefix is useful to prevent a possible conflict of the same table names from independent "applications". An important decision is if you can use it as a "legacy" database or a normal Django database.
You can decide between following two solutions or to combine them:
Use e.g. foo_models or bar.models and import all models from them e.g. to app.models and add only that "app" to INSTALLED_APPLICATIONS. This can be viable if it is only for one company and never otherwise and central name assigment is possible. (easiest but little naive)
Use some coarse separation of namespaces to several apps. You should probably use not more than one app with simple names without app name prefix.
Think ahead about migrations**. They will be probably very complicated and very soon impossible if you will create later more projects for the same database and different subsets of database tables without separating them to more apps and without app namespace.
There is really no "singleton" in Django ORM except of django.db.connections itself. If you use more databases you can direct some tables to a specific database by DATABASE_ROUTERS, even with two different models that use the same table name without a prefix.
I apologize if this seems like a stupid question but I'm still very much a novice Python/Django programmer. Is it normal to create Python objects in a Django application that aren't models that will be saved in the database?
I'm creating what's become a fairly large Django application and, to me, my code is really starting to "smell". What I mean is that my views are becoming very large because I'm taking a procedural rather than object-oriented approach. My intuition tells me that my code might be simpler, easier to test, and more robust in the long run if I were using more objects with their own attributes and behaviors rather than passing information from one function to the next in my views.
What's hanging me up is that these aren't objects I want to save in my database so I don't quite know if I should be using them and, if I should, where I'd put them. Is the approach I'm proposing typical in a Django application? If so, where would I store those objects with respect to the Django model/view/template structure? Also, are there any popular Django modules or libraries that do what I'm describing that I should study?
Thanks in advance for your response.
You can store your objects anywhere. There could be helper functions in your views file or models file or wherever. I prefer to put miscellaneous functions in a utils.py file but that is not a convention, just something I end up doing. I end up putting most of miscellaneous helper functions and base classes in a common app, and more specifically a common.utils file.
In one project I have lots of apps, and each app has an api client. The base class for the client resides in an app called common. Then each app then has their specific client in client.py file
project
common
client
app1
client
app2
client
Then in app1 client
from project.common.client import BaseClient
class ConcreteApp1Client(BaseClient):
pass
Then in my views or management commands or models or wherever the concrete client can be imported and used as normal. from project.app1.client import ConcreteApp1Client
Django also has class-based views if you feel certain variables could best be encapsulated in a class.
https://docs.djangoproject.com/en/dev/topics/class-based-views/
I would like to link django with an existing python application, reusing the modules, the classes and some configurationfiles that were defined in this application. All modules reside in the ./bin directory of this application. In fact the application is a forecasting system, and django will be used as the frontend to visualise the results.
The current directory structure looks like this.
./application/bin/module1.py
/module2.py
/config.txt
/datadir/...
/webresults/run1/myfig.png
/run2/myfig.png
/...
./django/manage.py
/project1/urls.py
/settings.py
/wsgi.py
/app1/views.py
/models.py
/forms.py
/...
/templates/base.html
/showResults.html
/...
An apacheserver is set up to serve static pages and images from the directory ./application/webresults/ and django pages from /django/project1/wsgi.py.
For the moment I have copied all relevant modules from /application/bin/ to django/app1/ so I can reuse them eg. in views.py and models.py. With respect to maintenance of the system not an optimal solution.
So I am looking for a more elegant solution to solve this. Some of the things we would like to achieve:
moving around the backendapplication is harder than moving around django instance, so preferable django is relocated (it was not coded in the most efficient way :-))
reuse classes of application in django
reuse models of django in the application
reuse SQL_objectmapping in the application
use one configfile for settings relevant to both django and the application
The solution we are thinking of, would be to merge all djangocode into /application/bin/ and remap the djangopath in apacheconfiguration
./application/bin/manage.py
/module1.py
/module2.py
/project1/urls.py
/settings.py
/wsgi.py
/app1/views.py
/models.py
/forms.py
/...
/templates/base.html
/showResults.html
/...
Are there any recipes on how this could be handled? Any advice appreciated.
There's nothing wrong with tossing your Django project within another module. So long as your PYTHONPATH's are setup correctly, and your WSGI is configured right, everything should work normally.
My only comment would be that, tautologically, "separate things should be separate."
When I have similar problems, I usually create a simple UNIX socket server (using Tornado), that Django can communicate with for API calls, etc.
If I need to expose the functionality of the socket server, I'll throw in a cheap WSGI server (again, with Tornado) that serves up requests, and stick it under a different subdomain within my hosting configuration.
I think my original question was not very clear. Since then I stumbled upon some other answers and pages that helped me , mainly on how to reuse django functionality in external modules and applications.
http://www.b-list.org/weblog/2007/sep/22/standalone-django-scripts/
why 'list index out of range' in my django code;
https://docs.djangoproject.com/en/dev/ref/models/options/#app-label
With this information I managed to access my models (in apps-subfolder) from the pythonscripts on the root of the application (eg. module1.py).
In short the code to access app1.models.Mymodel from module1.py looks now like this :
os.environ['DJANGO_SETTINGS_MODULE'] = 'project1.settings'
from django.core.management import setup_environ
from project1 import settings
setup_environ(settings)
import app1.models
class MyChildmodel(app1.models.Mymodel):
class Meta:
app_label = 'app1'
def myfunction():
....
I need a place to run an initialization code that is application specific (like connecting to signals).
When I put the code to __init__.py module of an application I've ended up with a circular import of the models.
Is there a way to fire a function when the framework is setup and before any request is executed?
I use quite old version of django 96.6, but I'm also interested in the solutions for the current version.
Regarding the duplication of other questions:
Here is how the question differ from the duplicates suggested by S.Lott in comments:
Correct place to put extra startup code in django?
Django need to be fully initialized when the function is ran. So code in manage.py won't work.
Where should I place the one-time operation operation in the Django framework?
The function initialize the connection between my applications. So the code must be ran in each thread that will actually handle the requests.
Comments to current solutions:
I can't use urls as most of my apps don't have any urls exposed. They just listen to signals and store additional information in the database.
Signals, specifically, are recommended to be put in the models.py of your app.
Try models.py or urls.py and let us know if you have any luck.
The best place for stuff like this... anywhere, just import it in your urls.py file (for obvious reasons urls are loading before any requests).
If you don't provide urls, then you really need to put it in models.py, that's just the way it is.
Now, on to your problems: You want to define it in its own module, great, do that. To avoid a circular import, use django.db.models.get_model to return the model dynamically for you. You can provide an initialisation function for your signals module to import the relevant model and connect the relevant signals. This function would then be called at the end of models.py, being run only ever once and after your model is initialised.
There's still a chance that this wont work (if the models aren't yet ready when you set it up), but give it a try and let us know.
For me, the following works:
In init.py:
from . import models
from . import signals
signals.py imports from models, but not vice versa. signals.py contains module code that is run immediately when it is imported and is thus run when the django server starts.
I have started work on a local app for myself that runs through the browser. Having recently gone through the django tutorial I'm thinking that it might be better to use django rather than just plain python.
There's one problem: I have at least 20 models and each will have many functions. Quite simply it's going to create one huge models file and probably huge views too. How do I split them up?
The models are all related so I can't simply make them into separate apps can I?
This is a pretty common need... I can't imagine wading through a models.py file that's 10,000 lines long :-)
You can split up the models.py file (and views.py too) into a pacakge. In this case, your project tree will look like:
/my_proj
/myapp
/models
__init__.py
person.py
The __init__.py file makes the folder into a package. The only gotcha is to be sure to define an inner Meta class for your models that indicate the app_label for the model, otherwise Django will have trouble building your schema:
class Person(models.Model):
name = models.CharField(max_length=128)
class Meta:
app_label = 'myapp'
Once that's done, import the model in your __init__.py file so that Django and sync db will find it:
from person import Person
This way you can still do from myapp.models import Person
"I have at least 20 models" -- this is probably more than one Django "app" and is more like a Django "project" with several small "apps"
I like to partition things around topics or subject areas that have a few (1 to 5) models. This becomes a Django "app" -- and is the useful unit of reusability.
The overall "project" is a collection of apps that presents the integrated thing built up of separate pieces.
This also helps for project management since each "app" can become a sprint with a release at th end.
The models are all related so I cant's
simply make them into separate apps
can I?
You can separate them into separate apps. To use a model in one app from another app you just import it in the same way you would import django.contrib apps.
Having 20 models in one app might be a sign that you should break it up in smaller ones.
The purpose of a Django app is to have a small single-purpose piece of code, that fits nicelly together.
So, if you had a e-commerce site, you might have a shopping_cart app, a billing app, and so on.
Keep in mind that there is really no problem in apps depending on each other (although it's always better if they can be decoupled), but you should not have an app doing two very distinct things.
The article Django tips: laying out an application might help you. As always, take everything you read with a grain of salt (including this answer).
You can break up the models over multiple files. This goes for views as well.
You can split them into separate files and simply have imports at the top of your main models.py field.
Whether you'd really want to is another question.