We Keep Coding

Django is rendering template from the database efficient

This is not a HOWTO question, as it, as such, has been answered before here
I am trying to integrate Django with modern frontend framework, and I found that it is possible to store and render Django templates from models. Since it is not a standard, I am wondering what are the advantages (or disadvantages if that's the case) of file based templates.
Reading though the documentation, I have seen that it is recommended to actually cache templates and models as much as possible for best performance, so why would it not be recommended to store templates in the database? It seems very convenient to me that in doing so pages can be edited from the admin panel (where you can add a code editor), which, along with the rest framework and a front end framework synergize very well.
From my research, the template tags and template language seem to work and the context can be passed in a view as well. the only thing I cannot figure out is the {include .. } tag, as it does not seem to point to a view. (although a custom tag cam be made to have this function)
Can such a setup be used in production? or are there security/performance/other concerns?

It's not recommended because templates are the developer's responsibility, not the site admin's.
Otherwise, there's not much performance difference between reading files directly from filesystem and from database.
One drawback of this approach is you don't get revision history (using git, mercurial etc). Sure, you can implement something similar to save the revisions in the database, but then why not use the better tools which already exist?
There's a Django flatpages app which lets you save HTML content in database. But the purpose of the app is to allow site admins to edit HTML content, such as information for an "About Us" page, because writing this info is not a developer's responsibility.

template fragment caching doesn't seem to work for some custom template tags

I've been implementing caching in my django application, and used per view caching via the cache API and template fragment caching.
On some of my pages I use a custom django template tag, this tag is provided via a third party developer, it takes some arguments in its template tags, and then make a request to a remote server, gets the response back over XML, and then renders the result in my page.
Great - I thought I could easily cache this using fragment caching, so I :
{% load cache %}
{% cache 500 request.user.username %}
{% load third party custom tags %}
{% expensive custom tag set that gets stuff from a third party server via xml %}
{{ some.stuff}}
{% endcache %}
Trouble is no matter what I do, the requests still get fired off to that remote server, it seems Django doesn't like to cache these custom template tags. I know memcached is working great, for other views and templates it all works just fine. Am I doing something that is incompatible with the fragment caching? Is there a way round it?

If the template fragment you're trying to cache can't be pickled, memcached won't be able to store it and will raise an exception. From what I can gather, exceptions generated when rendering Django templates are suppressed. Since your custom tag is doing HTTP requests, maybe socket objects (which can't be pickled) are getting stored to the template fragment somehow.
If this is the case, the only way around it I can think of would be to modify the custom tag to get rid of any leftover socket objects.

Have you tried to use a different name for the cache fragment? There could be a problem with using request.user.username for a couple of reasons:
If a user is not signed in,
request.user.username could be empty,
resulting in a non-named cache
fragment
If a user is signed in, this will
call the 3rd party template tag at
least once for each user every 3
mintues
Maybe it's worth trying to rename the cache fragment name to test:
{% cache 500 customxml %}
I'd also try loading of the 3rd party template tag outside the cache tag like so:
{% load cache third_party_custom_tags %}
{% cache 500 request.user.username %}
{% expensive custom tag set that gets stuff from a third party server via xml %}
{{ some.stuff}}
{% endcache %}
What I'm not sure of is if the cache framework caches the results of a template tag. If that doesn't work, I'd take a look at what the template tag is doing under the hood and re-implement the template tag using Django's low-level cache.

I think the problem is the custom tag, as you suggested.
I disagree that the request.user.username is a problem, as the documentation for the subject actually gives that as an example, and I have used it with internal caching (number of posts, for instance), in testing, and it worked fine.
The low level cache is potentially useful, but I would take a look at your custom tag to see what wouldn't be caching. Without the code it's hard to guess, but my guess would be something like the time, or some other variable, is being returned whic his causing it to force an update (if the XML pulls any data that changes Django may force an update, depending on other settings). I've had mixed results with Django's caching, so I would have a look at your XML feed(s) to see if it's causing anything to stop caching.

I don't think this has anything to do with the custom tag.
We endded up rewriting the Django caching tag because we needed more control than was possible with the one that was supplied. You might make a copy of it yourself and stick some debugging print statements into it. In particular, check the filename (assuming you are caching to files) and see what is being generated. It could be that it is changing when it shouldn't (for some unknown reason) and that would mean that it is always needing to re-render then enclosed block.
Look in django/templatetags/cache.py. It's only 63 lines of code.

Rendering JSON objects using a Django template after an Ajax call

I've been trying to understand what's the optimal way to do Ajax in Django. By reading stuff here and there I gathered that the common process is:
formulate your Ajax call using some JavaScript library (e.g., jQuery), set up a URL pattern in Django that catches the call and passes it to a view function
in the Python view function retrieve the objects you are interested in and send them back to the client in JSON format or similar (by using the built in serializer module, or simplejson)
define a callback function in JavaScript that receives the JSON data and parses them, so to create whatever HTML is needed to be displayed. Finally, the JavaScript script puts the HTML wherever it should stay.
Now, what I still don't get is how are Django templates related to all of this? Apparently, we're not making use of the power of templates at all.
Ideally, I thought it'd be nice to pass back a JSON object and a template name, so that the data could be iterated over and an HTML block is created. But maybe I'm totally wrong here...
The only resource I found that goes in this direction is this snippet (769) but I haven't tried it yet.
Obviously, what's going to happen in this case is that all the resulting HTML is created on the server side, then passed to the client. The JavaScript-callback function only has to display it in the right place.
Does this cause performance problems? If not, even without using the snippet above, why not formatting the HTML directly in the backend using Python instead of the front-end?
Many thanks!
UPDATE: please use snippet 942 because it is an enhanced version of the one above! I found that the inheritance support works much better this way..

Hey thanks vikingosegundo!
I like using decorators too :-).
But in the meanwhile I've been following the approach suggested by the snippet I was mentioning above. Only thing, use instead the snippet n. 942 cause it's an improved version of the original one. Here's how it works:
Imagine you have a template (e.g., 'subtemplate.html') of whatever size that contains a useful block you can reuse:
........
<div id="results">
{% block results %}
{% for el in items %}
<li>{{el|capfirst}}</li>
{% endfor %}
{% endblock %}
</div><br />
........
By importing in your view file the snippet above you can easily reference to any block in your templates. A cool feature is that the inheritance relations among templates are taken into consideration, so if you reference to a block that includes another block and so on, everything should work just fine. So, the ajax-view looks like this:
from django.template import loader
# downloaded from djangosnippets.com[942]
from my_project.snippets.template import render_block_to_string
def ajax_view(request):
# some random context
context = Context({'items': range(100)})
# passing the template_name + block_name + context
return_str = render_block_to_string('standard/subtemplate.html', 'results', context)
return HttpResponse(return_str)

Here is how I use the same template for traditional rendering and Ajax-response rendering.
Template:
<div id="sortable">
{% include "admin/app/model/subtemplate.html" %}
</div>
Included template (aka: subtemplate):
<div id="results_listing">
{% if results %}
{% for c in results %}
.....
{% endfor %}
{% else %}
The Ajax-view:
#login_required
#render_to('admin/app/model/subtemplate.html')#annoying-decorator
def ajax_view(request):
.....
return {
"results":Model.objects.all(),
}
Of course you can use render_to_response. But I like those annoying decorators :D

There's no reason you can't return a rendered bit of HTML using Ajax, and insert that into the existing page at the point you want. Obviously you can use Django's templates to render this HTML, if you want.

When you are doing Ajax I don't think you have any use for templates.
Template is there so that you can generate dynamic HTML on the server side easily and hence it provides few programming hooks inside HTML.
In case of Ajax you are passing JSON data and you can format it as you want in Python.
and HTML/document elements will be generated on client side using the JSON by some JavaScript library e.g. jQuery on client side.
Maybe if you have a very specific case of replacing some inner HTML from server side HTML then maybe you can use templates but in that case why you would need JSON?
You can just query the HTML page via Ajax and change inner or outer or whatever HTML.

Templates are for the purpose of presentation. Responding with data in format X (JSON, JSONP, XML, YAML, *ml, etc.) is not presentation, so you don't need templates. Just serialize your data into format X and return it in an HttpResponse.

While templates are indeed just for presentation purposes, it shouldn't matter if you are doing it on the serverside or client side. It all comes down to separating the control logic that is performing an action, from the view logic that is just responsible for creating the markup. If your javascript control logic is having to handle how you are rendering or displaying the HTML, then you might be doing it wrong, but if you isolate that rendering logic to another object or function, and just passing it the data necessary for the render, then you should be fine; it mirrors how we separate our controllers, models and views on the server side.
Take a look at the github project: http://github.com/comolongo/Yz-Javascript-Django-Template-Compiler
It compiles django templates into optimized javascript functions that will generate your presentation html with data that you pass it. The compiled functions are in pure javascript, so there are no dependencies on other libraries. Since the templates are compiled instead of being parsed at runtime, the strings and variables are all already placed into javascript strings that just need to be concatenated, so you get a huge speed increase compared to techniques that require you to do dom manipulation or script parsing to get the final presentation. Right now only the basic tags and filters are there, but should be enough for most things, and more tags will be added as people start making requests for them or start contributing to the project.

You can use jquery.load() or similar, generating the HTML on the server and loading it into the DOM with JavaScript. I think someone has called this AJAH.

Unfortunately, Django templates are designed to be executed server side only. There is at least one project to render Django templates using Javascript, but I haven't used it and so I don't know how fast, well supported or up to date it is. Other than this, you have to either use the Django templates on the server or generate dynamic elements on the client without using templates.

Using Django admin look and feel in my own application

I like the very simple but still really elegant look and feel of the django admin and I was wondering if there is a way to apply it to my own application.
(I think that I've read something like that somewhere, but now I cannot find the page again.)
(edited: what I am looking for is a way to do it automatically by extending templates, importing modules, or something similar, not just copy&paste the css and javascript code)

Are you sure you want to take every bit of admin-site's look & feel??
I think you would need to customize some, as in header footer etc.
To do that, just copy base.html from
"djangosrc/contrib/admin/templates/admin/"
and keep it in
"your_template_dir/admin/base.html" or
"your_template_dir/admin/mybase.html"
Just change whatever HTML you want to customize and keep rest as it is (like CSS and Javascript) and keep on extending this template in other templates of your application. Your view should provide what it needs to render (take a look at any django view from source) and you'll have everything what admin look & feel had. More you can do by extending base_site.html in same manner.
(Note: if you keep the name
'base.html' the changes made in
html will affect Django Admin too.
As this is the way we change how
Django Admin look itself.)

{% extends "admin/base_site.html" %}
is usually a good place to start but do look at the templates in contrib/admin/templates and copy some of the techniques there.

Django.contrib.flatpages without models

I have some flatpages with empty content field and their content inside the template (given with template_name field).
Why I am using django.contrib.flatpages
It allows me to serve (mostly) static pages with minimal URL configuration.
I don't have to write views for each of them.
Why I don't need the model FlatPage
I leave the content empty and just supply a template path. Therefore I can take advantage of having the source in a file;
I can edit the source directly from the file system, without the help of a server (such as admin).
I can take advantage of syntax highlightning and other editor features.
With the model I have to maintain fixtures for flatpages.
So the data for the same entity is in two seperate places.
If I move the content inside the fixture it'll be more difficult to edit.
Even if fixture maintenance was a non-issue I'd still need to dump and load these fixtures again and again during development.
What I am looking for
Basically; getting rid of FlatPage model while maintaining contrib.flatpages functionality. I don't have a clear idea how this should be solved. If there's a clean way of modifying (like add_to_class) FlatPages to get the information somewhere other than the database I'd prefer that. Maybe the metadata can be inserted to the templates and then a special manager that reads this data would replace the default manager of FlatPages.
If I don't prefer manual editing over admin functionality for flatpages, how can take the database out of the equation?

Using the direct_to_template generic view would be a lot simpler. You could use the passed in parameters on one view to specify the actual template in urls.py, if you don't want to add an entry for each page:
r'^foo/(?P<template_name>.+)/$','direct_to_template', {'template': 'foo_index.html'}),
Then import the template in your foo_index.html:
{% include template_name %}