Is there a way to detect unused templates in a Django project?
Before Django 1.3, that would have been possible with a simple string-matching function like this one. But since 1.3, there are generic class based views that automatically generate a template_name, if you don't override it (e.g. DetailView).
Also, if you override 3rd party module templates, those templates aren't used anywhere directly in your views.
Maybe it could be done by crawling all URL definitions, loading the corresponding views and getting the template_name from them?
I was curious if you could do this by monkey patching/decorating get_template instead. I think you can, though you have to find all the template loading
functions (I have two in my example below).
I used wrapt when I noticed it went beyond just loader.get_template, but it seems to the trick just fine. Of course, keep this 50000 km away from prod, but...
Now, the thing to follow as well is that I am driving this with unittests and nosetests so, if you have full branch coverage of your template-using Python code, you should be able to get most templates (assuming I didn't miss any get_template-type functions).
in settings.py
This is the "brains" to patch get_template & co.
import wrapt
import django.template.loader
import django.template.engine
def wrapper(wrapped, instance, args, kwargs):
#concatenate the args vector into a string.
# print "\n\n\n\n%s\nI am a wrapper \nusage:%s\n%s\n\n\n\n\n" % ("*"*80, usage, "*"*80)
try:
return wrapped(*args, **kwargs)
finally:
usage = ",".join([unicode(arg) for arg in args if arg])
track_usage(usage)
#you have to wrap whatever is loading templates...
#imported django module + class/method/function path of what needs to be
#wrapped within that module. comment those 2 lines out and you are back to
#normal
wrapt.wrap_function_wrapper(django.template.loader, 'get_template', wrapper)
wrapt.wrap_function_wrapper(django.template.engine, 'Engine.find_template', wrapper)
See safely-applying-monkey-patches-in-python for more details on wrapt. Actually easier to use than to understand the docs, decorators make my brain hurt.
Also, to track which django functions were doing the actual loads I mispelled some template names on purpose in the code and in templates, ran unit tests on it and looked at the stacktraces for missing template exceptions.
This is my rather badly-written function which adds to a set and puts it into
a json output....
def track_usage(usage):
fnp_usage = "./usage.json"
try:
with open(fnp_usage, "r") as fi:
data = fi.read()
#read the set of used templates from the json file
j_data = json.loads(data)
s_used_file = set(j_data.get("li_used"))
except (IOError,),e:
s_used_file = set()
j_data = dict()
s_used_file.add(usage)
#convert the set back to a list for json compatibility
j_data["li_used"] = list(s_used_file)
with open(fnp_usage, "w") as fo:
json.dump(j_data, fo)
and the ouput (with a script to format it):
import sys
import json
fnp_usage = sys.argv[1]
with open(fnp_usage, "r") as fi:
data = fi.read()
#read the set of used templates from the json file
j_data = json.loads(data)
li_used_file = j_data.get("li_used")
li_used_file.sort()
print "\n\nused templates:"
for t in li_used_file:
print(t)
From wrapping the 2 functions above, it seems to have caught extends, %includes and straight get_templates, as well as list-type templates that were being used by class-based views. It even caught my dynamically generated templates which aren't even on the file system but get loaded with a custom loader.
used templates:
bootstrap/display_form.html
bootstrap/errors.html
bootstrap/field.html
bootstrap/layout/baseinput.html
bootstrap/layout/checkboxselectmultiple.html
bootstrap/layout/field_errors.html
bootstrap/layout/field_errors_block.html
bootstrap/layout/help_text.html
bootstrap/layout/help_text_and_errors.html
bootstrap/layout/radioselect.html
bootstrap/whole_uni_form.html
django_tables2/table.html
dynamic_template:db:testdb:name:pssecurity/directive.PrimaryDetails.json
uni_form/layout/div.html
uni_form/layout/fieldset.html
websec/__base.html
websec/__full12.html
websec/__l_right_sidebar.html
websec/bootstrapped_home.html
websec/changedb.html
websec/login.html
websec/requirejs_config.html
websec/topnav.html
websec/user_msg.html
It's not possible to detect unused templates for certain, even in the absence of generic views, because you can always write code like this:
get_template(any_code_you_like()).render(context)
So even prior to Django 1.3 the django-unused-templates application you linked to could only have worked for projects that respected some kind of discipline about the use of templates. (For example, always having a string literal as the template argument to functions like get_template and render_to_response.)
Loading all the views wouldn't be sufficient either: a view may use different templates under different circumstances:
def my_view(request):
if request.user.is_authenticated():
return render(request, 'template1.html')
else:
return render(request, 'template2.html')
And of course templates may not be used by views at all, but by other parts of the system (for example, e-mail messages).
Related
I have a Flask project where the entry point is application.py and then I have several other modules like, e.g. variant.py, etc.
The project structure is:
>my_app_dir/
application.py
views/
__init__.py
users.py
variant.py
...
For variant.py, it's a function like:
import ...
from views import *
def variant(variant_id, subset='all', language='en'):
...
if subset == 'all':
return json.dumps(x)
return json.dumps([{subset: y[subset]} for y in x])
The point is I want to use variant.py like an API, so I am testing via iPython, something like, but it's returning an error:
from views import variant as v
aa = v.variant('22-38212762-A-G')
...
RuntimeError: Working outside of request context.
This typically means that you attempted to use functionality that needed
an active HTTP request. Consult the documentation on testing for
information about how to avoid this problem.
I've tried googling but couldn't find any similar case, yet I experimented several things for no avail.
In the end, I found out a way to get what I was looking for:
from views import application, autocomplete
from views.variant import variant
ctx = application.test_request_context(path='/login',method='POST', data={'user':'demo','password':'demo123'})
ctx.push()
variant('22-38212762-A-G')[:50]
autocomplete.autocomplete('ttll','gene').json
So, essentially, the trick bit is:
ctx = application.test_request_context(path='/login',method='POST', data={'user':'demo','password':'demo123'})
ctx.push()
I need to test a function with different parameters, and the most proper way for this seems to be using the with self.subTest(...) context manager.
However, the function writes something to the db, and it ends up in an inconsistent state. I can delete the things I write, but it would be cleaner if I could recreate the whole db completely. Is there a way to do that?
Not sure how to recreate the database in self.subTest() but I have another technique I am currently using which might be of interest to you. You can use fixtures to create a "snapshot" of your database which will basically be copied in a second database used only for testing purposes. I currently use this method to test code on a big project I'm working on at work.
I'll post some example code to give you an idea of what this will look like in practice, but you might have to do some extra research to tailor the code to your needs (I've added links to guide you).
The process is rather straighforward. You would be creating a copy of your database with only the data needed by using fixtures, which will be stored in a .yaml file and accessed only by your test unit.
Here is what the process would look like:
List item you want to copy to your test database to populate it using fixtures. This will only create a db with the needed data instead of stupidly copying the entire db. It will be stored in a .yaml file.
generate.py
django.setup()
stdout = sys.stdout
conf = [
{
'file': 'myfile.yaml',
'models': [
dict(model='your.model', pks='your, primary, keys'),
dict(model='your.model', pks='your, primary, keys')
]
}
]
for fixture in conf:
print('Processing: %s' % fixture['file'])
with open(fixture['file'], 'w') as f:
sys.stdout = FixtureAnonymiser(f)
for model in fixture['models']:
call_command('dumpdata', model.pop('model'), format='yaml',indent=4, **model)
sys.stdout.flush()
sys.stdout = stdout
In your test unit, import your generated .yaml file as a fixture and your test will automatically use this the data from the fixture to carry out the tests, keeping your main database untouched.
test_class.py
from django.test import TestCase
class classTest(TestCase):
fixtures = ('myfile.yaml',)
def setUp(self):
"""setup tests cases"""
# create the object you want to test here, which will use data from the fixtures
def test_function(self):
self.assertEqual(True,True)
# write your test here
You can read up more here:
Django
YAML
If you have any questions because things are unclear just ask, I'd be happy to help you out.
Maybe my solution will help someone
I used transactions to roll back to the database state that I had at the start of the test.
I use Eric Cousineau's decorator function to parametrizing tests
More about database transactions at django documentation page
import functools
from django.db import transaction
from django.test import TransactionTestCase
from django.contrib.auth import get_user_model
User = get_user_model()
def sub_test(param_list):
"""Decorates a test case to run it as a set of subtests."""
def decorator(f):
#functools.wraps(f)
def wrapped(self):
for param in param_list:
with self.subTest(**param):
f(self, **param)
return wrapped
return decorator
class MyTestCase(TransactionTestCase):
#sub_test([
dict(email="new#user.com", password='12345678'),
dict(email="new#user.com", password='password'),
])
def test_passwords(self, email, password):
# open a transaction
with transaction.atomic():
# Creates a new savepoint. Returns the savepoint ID (sid).
sid = transaction.savepoint()
# create user and check, if there only one with this email in DB
user = User.objects.create(email=email, password=password)
self.assertEqual(User.objects.filter(email=user.email).count(), 1)
# Rolls back the transaction to savepoint sid.
transaction.savepoint_rollback(sid)
Ok so I have my apps, that takes requests from root / Almost everything is using traversal.
But i'd like to make on top of that site a rest api.
So I'm off with two choices. I either separate the that in two different apps and put that rest application to : rest.site.com, Or I can move it to site.com/rest/*traversal
If I'm doing "/rest/*traversal", I guess I'll have to add a route called rest_traversal where the traversal path will be *traversal with the route /rest/*traversal. I did that once for an admin page.
I was wondering if there was a cleanest way to do that. I tried to use virtual_root, but as I understand virtual_root is actually getting added to the path for traversal.
like having virtual_root = /cms and requesting /fun will create the following path /cms/fun
I on the other hand wish to have /cms/fun turned into /fun
I know this has been answered already, but in case someone arrives here looking for another possible way to make "subapps" and using them in pyramid, I wanted to point out that some interesting things can be done with pyramid.wsgi
"""
example of wsgiapp decorator usage
http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/api/wsgi.html
"""
from pyramid.wsgi import wsgiapp2, wsgiapp
from pyramid.config import Configurator
from webob import Request, Response
import pprint
# define some apps
def wsgi_echo(environ, start_response):
"""pretty print out the environ"""
response = Response(body=pprint.pformat({k: v for k, v in environ.items()
if k not in ["wsgi.errors",
"wsgi.input",
"SCRIPT_NAME"]}))
return response(environ, start_response)
print Request.blank("/someurl").send(wsgi_echo).body
# convert wsgi app to a pyramid view callable
pyramid_echo = wsgiapp(wsgi_echo)
pyramid_echo_2 = wsgiapp2(wsgi_echo)
# wire up a pyramid application
config = Configurator()
config.add_view(pyramid_echo, name="foo") # /foo
config.add_view(pyramid_echo, name="bar") # /bar
config.add_view(pyramid_echo_2, name="foo_2") # /foo
config.add_view(pyramid_echo_2, name="bar_2") # /bar
pyramid_app = config.make_wsgi_app()
#call some urls
foo_body = Request.blank("/foo").send(pyramid_app).body
bar_body = Request.blank("/bar").send(pyramid_app).body
foo_body_2 = Request.blank("/foo_2").send(pyramid_app).body
bar_body_2 = Request.blank("/bar_2").send(pyramid_app).body
# both should be different because we arrived at 2 different urls
assert foo_body != bar_body, "bodies should not be equal"
# should be equal because wsgiapp2 fixes stuff before calling
# application in fact there's an additional SCRIPT_NAME in the
# environment that we are filtering out
assert foo_body_2 == bar_body_2, "bodies should be equal"
# so how to pass the path along? like /foo/fuuuu should come back
# /fuuuu does it
foo_body = Request.blank("/foo_2/fuuuu").send(pyramid_app).body
assert "'/fuuuu'," in foo_body, "path didn't get passed along"
# tldr: a wsgi app that is decorated with wsgiapp2 will recieve data
# as if it was mounted at "/", any url generation it has to do should
# take into account the SCRIPT_NAME variable that may arrive in the
# environ when it is called
If you're using traversal already, why not just use it to return your "rest API root" object when Pyramid traverses to /rest/? From there, everything will work naturally.
class ApplicationRoot(object):
def __getitem__(self, name):
if name == "rest":
return RestAPIRoot(parent=self, name=name)
...
If your "application tree" and "API tree" have the same children and you want to have different views registered for them depending on which branch of the tree the child is located in, you can use containment view predicate to register your API views, so they will only match when the child is inside the "API branch":
containment
This value should be a reference to a Python class or interface that a
parent object in the context resource’s lineage must provide in order
for this view to be found and called. The resources in your resource
tree must be “location-aware” to use this feature.
If containment is not supplied, the interfaces and classes in the
lineage are not considered when deciding whether or not to invoke the
view callable.
Another approach would be not to build a separate "API tree" but to use your "main" application's "URI-space" as RESTful API. The only problem with this is that GET and possibly POST request methods are already "taken" on your resources and mapped to your "normal" views which return HTML or consume HTTP form POSTs. There are numerous ways to work around this:
register the API views with a separate name, so, say GET /users/123 would return HTML and GET /users/123/json would return a JSON object. Similarly, POST /users/123 would expect HTTP form to be posted and POST /users/123/json would expect JSON. A nice thing about this approach is that you can easily add, say, an XML serializer at GET /users/123/xml.
use custom view predicates so GET /users/123 and GET /users/123?format=json are routed to different views. Actually, there's a built-in request_param predicate for that since Pyramid 1.2
use xhr predicate to differentiate requests based on HTTP_X_REQUESTED_WITH header or accept predicate to differentiate on HTTP_ACCEPT header
When I am adding a static view like this:
cfg = config.Configurator(...)
cfg.add_static_view(name='static', path='MyPgk:static')
# And I want to add a view for 'favicon.ico'.
cfg.add_route(name='favicon', pattern='/favicon.ico')
cfg.add_view(route_name='favicon', view='MyPgk.views.mymodule.favicon_view')
I am trying to add that favicon.ico annoying default path of /favicon.ico called by the browser if it's undefined in the webpage. I would like to use the example at http://docs.pylonsproject.org/projects/pyramid_cookbook/en/latest/files.html and modify it to have:
def favicon_view(request, cache=dict()):
if (not cache):
_path_to_MyPkg_static = __WHAT_GOES_HERE__
_icon = open(os.path.join(_path_to_MyPkg_static, 'favicon.ico')).read()
cache['response'] = Response(content_type='image/x-icon', body=_icon)
return cache['response']
Since, I can't really define the _here proposed in the example, how can I make it dependent to request to get the actual full path at runtime? Or do I really have to deal with:
_here = os.path.dirname(__file__)
_path_to_MyPkg_static = os.path.join(os.path.dirname(_here), 'static')
and having to be careful when I decide to refactor and put the view in another pkg or subpackage, or where-ever?
Something equivalent to request.static_path() but instead of getting the url path, to actually get a directory path:
request.static_file_path('static') -> /path/to/site-packages/MyPkg/static
Thanks,
You can use the pkg_resources module to make paths that are relative to Python modules (and thus, independent of the module that retrieves them). For example:
import pkg_resources
print pkg_resources.resource_filename('os.path', 'static/favicon.ico')
# 'C:\\Python27\\lib\\static\\favicon.ico'
Just substitute os.path with whatever module that is the parent of your static files.
EDIT: If you need to remember that 'static' route mapped to 'MyPkg:static', then the easiest way is to save it in some dictionary in the first place:
STATIC_ROUTES = {'static': 'MyPkg:static'}
for name, path in STATIC_ROUTES.iteritems():
cfg.add_static_view(name=name, path=path)
and then simply retrieve the path:
static_path = STATIC_ROUTES['static']
package, relative_path = static_path.split(':')
icon_path = pkg_resources.resource_filename(
package, os.path.join(relative_path, 'favicon.ico'))
If that's impossible, though (e.g. you don't have access to the cfg object), you can retrieve this path, it's just quite painful. Here's a sample function that uses undocumented calls (and so may change in future Pyramid versions) and ignores some additional settings (like route_prefix configuration variable):
def get_static_path(request, name):
from pyramid.config.views import StaticURLInfo
registrations = StaticURLInfo()._get_registrations(request.registry)
if not name.endswith('/'):
name = name + '/'
route_name = '__%s' % name
for _url, spec, reg_route_name in registrations:
print ':', reg_route_name
if reg_route_name == route_name:
return spec
In your case, it should work like this:
>>> get_static_path(request, 'static')
MyPkg:static/
I want to generate for my users temporary download link.
Is that ok if i use django to generate link using url patterns?
Could it be correct way to do that. Because can happen that I don't understand some processes how it works. And it will overflow my memory or something else. Some kind of example or tools will be appreciated. Some nginx, apache modules probably?
So, what i wanna to achieve is to make url pattern which depend on user and time. Decript it end return in view a file.
A simple scheme might be to use a hash digest of username and timestamp:
from datetime import datetime
from hashlib import sha1
user = 'bob'
time = datetime.now().isoformat()
plain = user + '\0' + time
token = sha1(plain)
print token.hexdigest()
"1e2c5078bd0de12a79d1a49255a9bff9737aa4a4"
Next you store that token in a memcache with an expiration time. This way any of your webservers can reach it and the token will auto-expire. Finally add a Django url handler for '^download/.+' where the controller just looks up that token in the memcache to determine if the token is valid. You can even store the filename to be downloaded as the token's value in memcache.
Yes it would be ok to allow django to generate the urls. This being exclusive from handling the urls, with urls.py. Typically you don't want django to handle the serving of files see the static file docs[1] about this, so get the notion of using url patterns out of your head.
What you might want to do is generate a random key using a hash, like md5/sha1. Store the file and the key, datetime it's added in the database, create the download directory in a root directory that's available from your webserver like apache or nginx... suggest nginx), Since it's temporary, you'll want to add a cron job that checks if the time since the url was generated has expired, cleans up the file and removes the db entry. This should be a django command for manage.py
Please note this is example code written just for this and not tested! It may not work the way you were planning on achieving this goal, but it works. If you want the dl to be pw protected also, then look into httpbasic auth. you can generate and remove entries on the fly in a httpd.auth file using htpasswd and the subprocess module when you create the link or at registration time.
import hashlib, random, datetime, os, shutil
# model to hold link info. has these fields: key (charfield), filepath (filepathfield)
# datetime (datetimefield), url (charfield), orgpath (filepathfield of the orignal path
# or a foreignkey to the files model.
from models import MyDlLink
# settings.py for the app
from myapp import settings as myapp_settings
# full path and name of file to dl.
def genUrl(filepath):
# create a onetime salt for randomness
salt = ''.join(['{0}'.format(random.randrange(10) for i in range(10)])
key = hashlib('{0}{1}'.format(salt, filepath).hexdigest()
newpath = os.path.join(myapp_settings.DL_ROOT, key)
shutil.copy2(fname, newpath)
newlink = MyDlink()
newlink.key = key
newlink.date = datetime.datetime.now()
newlink.orgpath = filepath
newlink.newpath = newpath
newlink.url = "{0}/{1}/{2}".format(myapp_settings.DL_URL, key, os.path.basename(fname))
newlink.save()
return newlink
# in commands
def check_url_expired():
maxage = datetime.timedelta(days=7)
now = datetime.datetime.now()
for link in MyDlink.objects.all():
if(now - link.date) > maxage:
os.path.remove(link.newpath)
link.delete()
[1] http://docs.djangoproject.com/en/1.2/howto/static-files/
It sounds like you are suggesting using some kind of dynamic url conf.
Why not forget your concerns by simplifying and setting up a single url that captures a large encoded string that depends on user/time?
(r'^download/(?P<encrypted_id>(.*)/$', 'download_file'), # use your own regexp
def download_file(request, encrypted_id):
decrypted = decrypt(encrypted_id)
_file = get_file(decrypted)
return _file
A lot of sites just use a get param too.
www.example.com/download_file/?09248903483o8a908423028a0df8032
If you are concerned about performance, look at the answers in this post: Having Django serve downloadable files
Where the use of the apache x-sendfile module is highlighted.
Another alternative is to simply redirect to the static file served by whatever means from django.