We Keep Coding

Correct way to validate GET parameters in django

I'm building a social website that uses django templates/dynamic pages (no SPA technology in place).
I have some ajax calls that check the users news feed or new messages.
Example GET web request of those looks as follows:
GET /feeds/check/?last_feed=3&feed_source=all&_=1500749662203 HTTP/1.1
This is how I receive it in the view:
#login_required
#ajax_required
def check(request):
last_feed = request.GET.get('last_feed')
feeds = Feed.get_feeds_after(last_feed)
It all works, but I want to protect it so the function get_feeds_after does not crash when a malicious user sets the GET parameter to last_feed="123malicious4556". Currently it crashes because in the Feed model the function does this:
#staticmethod
def get_feeds_after(feed):
feeds = Feed.objects.filter(parent=None, id__gt=float(feed))
return feeds
and crashes with the error:
ValueError at /feeds/check/
invalid literal for float(): 2fff2
I currently solve this by directly performing checks on the GET variable and handling exception on int() casting:
def check(request):
last_feed = request.GET.get('last_feed')
try:
feed_source = int(request.GET.get('last_feed'))
except ValueError:
return HttpResponse(0)
My question is what is the best django-recommended way to address this?
I know django has special support forms validation. But this does not seem quite right here, as the GET calls are more of an api rather than forms so it seems like a bad idea to define forms for those GET parameters.
Thanks

All you actually need are the form fields which do all basic validation for you.
If you need custom validation you can write your own validator or even better your own custom form field.
To use the field alone without the form you can do like that for example:
evalType = forms.CharField().clean(request.GET.get('eval-type'))
Because calling this way is not very human friendly I prefer to write a function to deal with it:
def cleanParam(params, paramName, FieldType, *args, **kwargs):
field = FieldType(*args, **kwargs)
cleaned = field.clean(params.get(paramName))
return cleaned
Which we use this way:
evalType = cleanParam(request.GET, 'eval-type', forms.CharField)
This will save you a form class. But I don't think it's very ugly to create a django form for that. A bit too much for the problem but no great concern IMHO.
The advantage of having a form is that you declare the fields you expect in your api call and can check all at once then see the result of is_valid().
I hope this helps.

You can also use a django Validator, which is
a callable that takes a value and raises a ValidationError if it does not meet some criteria.
In your specific case, you can use a combination of validate_slug() - which does not raise a ValidationError only if the passed input is composed of letters, numbers, underscores or hyphens - and int() functions in this way:
from django.core.validators import validate_slug
...
last_feed = request.GET.get("last_feed", "")
try:
validate_slug(last_feed)
last_feed = int(last_feed)
...
except ValueError:
print("invalid literal passed to int()")
except ValidationError:
print("invalid input passed to validate_slug()")
...

Request params can be validated with the DRF serializer.
Create a serializer with all the parameters that are required to be validated.
from rest_framework import serializers
class OrderIDSerializer(serializers.Serializer):
order_id = serializers.CharField(required=True)
def validate(self, attrs):
order_id = attrs.get('order_id')
if not Order.objects.filter(
id=order_id).exists():
raise error
attrs['order_id'] = order_id
return attrs
Import serializer in the views:
from order.serializers.order_serializer import OrderIDSerializer
serializer = OrderIDSerializer(data=request.query_params)
if serializer.is_valid():
order_id = serializer.data['order_id']
// your logic after the validation

What is the proper way of testing throttling in DRF?

What is the proper way of testing throttling in DRF? I coulnd't find out any answer to this question on the net. I want to have separate tests for each endpoint since each one has custom requests limits (ScopedRateThrottle).
The important thing is that it can't affect other tests - they have to somehow run without throttling and limiting.

An easy solution is to patch the get_rate method of your throttle class. Thanks to tprestegard for this comment!
I have a custom class in my case:
from rest_framework.throttling import UserRateThrottle
class AuthRateThrottle(UserRateThrottle):
scope = 'auth'
In your tests:
from unittest.mock import patch
from django.core.cache import cache
from rest_framework import status
class Tests(SimpleTestCase):
def setUp(self):
cache.clear()
#patch('path.to.AuthRateThrottle.get_rate')
def test_throttling(self, mock):
mock.return_value = '1/day'
response = self.client.post(self.url, {})
self.assertEqual(
response.status_code,
status.HTTP_400_BAD_REQUEST, # some fields are required
)
response = self.client.post(self.url, {})
self.assertEqual(
response.status_code,
status.HTTP_429_TOO_MANY_REQUESTS,
)
It is also possible to patch the method in the DRF package to change the behavior of the standard throttle classes: #patch('rest_framework.throttling.SimpleRateThrottle.get_rate')

Like people already mentioned, this doesn't exactly fall within the scope of unit tests, but still, how about simply doing something like this:
from django.core.urlresolvers import reverse
from django.test import override_settings
from rest_framework.test import APITestCase, APIClient
class ThrottleApiTests(APITestCase):
# make sure to override your settings for testing
TESTING_THRESHOLD = '5/min'
# THROTTLE_THRESHOLD is the variable that you set for DRF DEFAULT_THROTTLE_RATES
#override_settings(THROTTLE_THRESHOLD=TESTING_THRESHOLD)
def test_check_health(self):
client = APIClient()
# some end point you want to test (in this case it's a public enpoint that doesn't require authentication
_url = reverse('check-health')
# this is probably set in settings in you case
for i in range(0, self.TESTING_THRESHOLD):
client.get(_url)
# this call should err
response = client.get(_url)
# 429 - too many requests
self.assertEqual(response.status_code, 429)
Also, regarding your concerns of side-effects, as long as you do user creation in setUp or setUpTestData, tests will be isolated (as they should), so no need to worry about 'dirty' data or scope in that sense.
Regarding cache clearing between tests, I would just add cache.clear() in tearDown or try and clear the specific key defined for throttling.

I implemented my own caching mechanism for throttling based on the user and the parameters with which a request is called. You can override SimpleRateThrottle.get_cache_key to get this behavior.
Take this throttle class for example:
class YourCustomThrottleClass(SimpleRateThrottle):
rate = "1/d"
scope = "your-custom-throttle"
def get_cache_key(self, request: Request, view: viewsets.ModelViewSet):
# we want to throttle the based on the request user as well as the parameter
# `foo` (i.e. the user can make a request with a different `foo` as many times
# as they want in a day, but only once a day for a given `foo`).
foo_request_param = view.kwargs["foo"]
ident = f"{request.user.pk}_{foo_request_param}"
# below format is copied from `UserRateThrottle.get_cache_key`.
return self.cache_format % {"scope": self.scope, "ident": ident}
In order to clear this in a TestCase I call the following method in each test method as required:
def _clear_throttle_cache(self, request_user, foo_param):
# we need to clear the cache of the throttle limits already stored there.
throttle = YourCustomThrottleClass()
# in the below two lines mock whatever attributes on the request and
# view instances are used to calculate the cache key in `.get_cache_key`
# which you overrode. Here we use `request.user` and `view.kwargs["foo"]`
# to calculate the throttle key, so we mock those.
pretend_view = MagicMock(kwargs={foo: foo_param})
pretend_request = MagicMock(user=request_user)
# this is the method you overrode in `YourCustomThrottleClass`.
throttle_key = throttle.get_cache_key(pretend_request, pretend_view)
throttle.cache.delete(user_key)

This is an amendment on yofee's post which got me 90% there. When using a throttle, custom or otherwise, with a set rate, get_rate is never called. As shown below from the source.
def __init__(self):
if not getattr(self, 'rate', None):
self.rate = self.get_rate()
Hence when one is mocking a throttle with a set rate that is not None, I would recommend patching the rate attribute directly.
...
with mock.patch.object(AuthRateThrottle, 'rate', '1/day'):
...

How to use Pyramid i18n outside of views and templates?

Pyramid documentation shows us how to use i18n inside views (and templates as well). But how to does one use it outside of views and templates where we have no access to current request (for example, in forms and models)?
#Michael said to pass request to models and forms. But is it right? I mean if form fields defines before __init__() method calls, the same with models. They don't see any parameters from views...
In Pylons we could simply use get_lang() and set_lang() and define preferable language in parent controller and then use ugettext() and ungettext() in any place we want without calling it from request directly every possible time (in views).
How to do that in Pyramid? Note that the language must be set from user's settings (session, cookies, db, etc).

My solution is to create the form class when it's needed with localizer as parameter. For example
forms.py
class FormFactory(object):
def __init__(self, localizer):
self.localizer = localizer
_ = self.localizer
self.required_msg = _(u'This field is required.')
self.invalid_email_msg = _(u'Invalid email address.')
self.password_not_match_msg = _(u'Password must match')
def make_contact_form(self):
_ = self.localizer
class ContactForm(Form):
email = TextField(_(u'Email address'), [
validators.Required(self.required_msg),
validators.Email(self.invalid_email_msg)
])
content = TextAreaField(_(u'Content'), [
validators.Required(self.required_msg)
])
return ContactForm
When you need to use the form
#view_config(route_name='front_pages.contact_us',
renderer='myweb:templates/front_pages/contact_us.genshi')
def contact_us(request):
"""Display contact us form or send mail
"""
_ = get_localizer(request)
factory = FormFactory(_)
ContactForm = factory.make_contact_form()
form = ContactForm(request.params)
return dict(form=form)
As you can see, we get the localizer in the view, and pass it to the FormFactory, then create a form with that factory. By doing that, all messages in the form was replaced with current locale language.
Likewise, you can do the same with model.

Have you found pyramid.18n.get_localizer yet?
http://docs.pylonsproject.org/projects/pyramid/en/1.3-branch/narr/i18n.html#using-a-localizer

Actually I had this very same problem. What I ended up doing was to see how the default locale negotiator works - it looks for a LOCALE property on the given request object. So just use a dummy to create the localizer. You may cache this value too, if you want
def my_get_localizer(locale=None):
request = Request({})
request._LOCALE_ = locale
return get_localizer(request)
Alternatively, join the irc channel #pyramid # freenode and pester the guys enough there to split the functionality of get_localizer in 2 separate documented functions (get_localizer and get_localizer_for_locale_name) for us to enjoy ;)
Also, notice that Pyramid TranslationStrings are lazy, so you can translate them as late as you want, e.g.
class MyModel(Base):
description = TranslationString("My model number ${number}")
...
def view(request):
m = MyModel()
localizer = get_localizer(request)
description = localizer.translate(m.description, mapping={'number': 1})
Sidenote: pylons' i18n was the worst can of worms I had opened in ages. The set_lang, get_lang hack was really awful, and pain in the ass as we needed to send emails to users in their native languages and then tried to restore the language back... also, it was IMPOSSIBLE to translate anything outside of a request in a pylons program, as a translator or the registry did not exist then.

You can make a localizer, and then translate a template accordingly.
When making the localizer, you can pass the lang you want (whether you have it from db or else). Hope it can help.
For the sake of clarity, I will set it as 'fr' below
from pyramid.i18n import make_localizer, TranslationStringFactory
from mako.template import Template
from mako.lookup import TemplateLookup
import os
absolute_path = os.path.dirname(os.path.realpath(__file__))
tsf = TranslationStringFactory('your_domain')
mako_lookup = TemplateLookup(directories=['/'])
template = Template(filename=template_path, lookup=mako_lookup)
localizer = make_localizer("fr", [absolute_path + '/../locale/'])
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
# Pass _ pointer (translate function) to the context
_ = auto_translate
context.update({
"_": _
})
html = template.render(**context)
EDIT
You can also put this logic into a small function
def get_translator(lang):
"""
Useful when need to translate outside of queries (no pointer to request)
:param lang:
:return:
"""
localizer = make_localizer(lang, [absolute_path + '/../locale/'])
def auto_translate(*args, **kwargs):
return localizer.translate(tsf(*args, **kwargs))
_ = auto_translate
return _

Django, how to see session data in the admin interface

I'm using Django sessions and I would like a way of seeing the session data in the admin interface. Is this possible?
I.e. for each session I want to see the data stored in the session database (which is essentially a dictionary as far as I can gather).
Currently I can just see a hash in the Session data field, such as:
gAJ9cQEoVQ5zb3J0aW5nX2Nob2ljZXECVQJQT3EDVQxnYW1lc19wbGF5ZWRxBH1xBVgLAAAAcG9z
dG1hbi1wYXRxBksDc1UKaXBfYWRkcmVzc3EHVQkxMjcuMC4wLjFxCFUKdGVzdGNvb2tpZXEJVQZ3
b3JrZWRxClUKZ2FtZV92b3Rlc3ELfXEMdS4wOGJlMDY3YWI0ZmU0ODBmOGZlOTczZTUwYmYwYjE5
OA==
I have put the following into admin.py to achieve this:
from django.contrib.sessions.models import Session
...
admin.site.register(Session)
In particular I was hoping to be able to see at least an IP address for each session. (Would be nice too if I could count how many sessions per IP address and order the IPs based on number of sessions in total for each.)
Thank you for your help :-)

You can do something like this:
from django.contrib.sessions.models import Session
class SessionAdmin(ModelAdmin):
def _session_data(self, obj):
return obj.get_decoded()
list_display = ['session_key', '_session_data', 'expire_date']
admin.site.register(Session, SessionAdmin)
It might be even that get_decoded can be used directly in list_display. And in case there's some catch that prevents this from working ok, you can decode the session data yourself, based on the linked Django source.

Continuing from Tomasz's answer, I went with:
import pprint
from django.contrib.sessions.models import Session
class SessionAdmin(admin.ModelAdmin):
def _session_data(self, obj):
return pprint.pformat(obj.get_decoded()).replace('\n', '<br>\n')
_session_data.allow_tags=True
list_display = ['session_key', '_session_data', 'expire_date']
readonly_fields = ['_session_data']
exclude = ['session_data']
date_hierarchy='expire_date'
admin.site.register(Session, SessionAdmin)

Session data is contained in a base64 encoded pickled dictionary. That's is what you're seeing in the admin because that data is stored in a TextField in the Session model.
I don't think any distributed django code stores the ip address in the session but you could do it yourself if you can access it.
In order to display the real session information, you may write your own form field that presents the decoded information. Keep in mind that you'll have to also overwrite the save method if you want to modify it. You can take a look at the encode and decode methods in django/contrib/sessions/models.py.

EB's otherwise great answer left me with the error "Database returned an invalid value in QuerySet.dates(). Are time zone definitions and pytz installed?". (I do have db tz info and pytz installed, and my app uses timezones extensively.) Removing the 'date_hierarchy' line resolved the issue for me. So:
import pprint
from django.contrib.sessions.models import Session
class SessionAdmin(admin.ModelAdmin):
def _session_data(self, obj):
return pprint.pformat(obj.get_decoded()).replace('\n', '<br>\n')
_session_data.allow_tags=True
list_display = ['session_key', '_session_data', 'expire_date']
readonly_fields = ['_session_data']
exclude = ['session_data']
admin.site.register(Session, SessionAdmin)

Adding to previous answers, We can also show the user for that session which is helpful for identifying the session of users.
class SessionAdmin(admin.ModelAdmin):
def user(self, obj):
session_user = obj.get_decoded().get('_auth_user_id')
user = User.objects.get(pk=session_user)
return user.email
def _session_data(self, obj):
return pprint.pformat(obj.get_decoded()).replace('\n', '<br>\n')
_session_data.allow_tags = True
list_display = ['user', 'session_key', '_session_data', 'expire_date']
readonly_fields = ['_session_data']

Django models: Only permit one entry in a model?

I want to make some of my Django global settings configurable through the admin interface.
To that end, I've decided to set them as database fields, rather than in settings.py.
These are the settings I care about:
class ManagementEmail(models.Model):
librarian_email = models.EmailField()
intro_text = models.CharField(max_length=1000)
signoff_text = models.CharField(max_length=1000)
These are one-off global settings, so I only ever want there to be a single librarian_email, intro_text etc floating around the system.
Is there a way I can prevent admin users from adding new records here, without preventing them from editing the existing record?
I guess I can do this by writing a custom admin template for this model, but I'd like to know if there's a neater way to configure this.
Could I use something other than class, for example?
Thanks!

Please see this question on "keep[ing] settings in database", where the answer seems to be django-dbsettings
Update
Just thought of another option: you can create the following model:
from django.contrib.sites.models import Site
class ManagementEmail(models.Model):
site = models.OneToOneField(Site)
librarian_email = models.EmailField()
intro_text = models.CharField(max_length=1000)
signoff_text = models.CharField(max_length=1000)
Because of the OneToOneField field, you can only have one ManagementEmail record per site. Then, just make sure you're using sites and then you can pull the settings thusly:
from django.contrib.sites.models import Site
managementemail = Site.objects.get_current().managementemail
Note that what everyone else is telling you is true; if your goal is to store settings, adding them one by one as fields to a model is not the best implementation. Adding settings over time is going to be a headache: you have to add the field to your model, update the database structure, and modify the code that is calling that setting.
That's why I'd recommend using the django app I mentioned above, since it does exactly what you want -- provide for user-editable settings -- without making you do any extra, unnecessary work.

I think the easiest way you can do this is using has_add_permissions function of the ModelAdmin:
class ContactUsAdmin(admin.ModelAdmin):
form = ContactUsForm
def has_add_permission(self, request):
return False if self.model.objects.count() > 0 else super().has_add_permission(request)
You can set the above to be any number you like, see the django docs.
If you need more granularity than that, and make the class a singleton at the model level, see django-solo. There are many singleton implementations also that I came across.
For StackedInline, you can use max_num = 1.

Try django-constance.
Here are some useful links:
https://github.com/jezdez/django-constance
http://django-constance.readthedocs.org/en/latest/

I'd take a page out of wordpress and create a Model that support settings.
class Settings(models.Model):
option_name = models.CharField(max_length=1000)
option_value = models.CharField(max_length=25000)
option_meta = models.CharField(max_length=1000)
Then you can just pickle (serialize) objects into the fields and you'll be solid.
Build a little api, and you can be as crafty as wordpress and call. AdminOptions.get_option(opt_name)
Then you can just load the custom settings into the runtime, keeping the settings.py module separate, but equal. A good place to write this would be in an __init__.py file.

Just set up an GlobalSettings app or something with a Key and Value field.
You could easily prevent admin users from changing values by not giving them permission to edit the GlobalSettings app.
class GlobalSettingsManager(models.Manager):
def get_setting(self, key):
try:
setting = GlobalSettings.objects.get(key=key)
except:
raise MyExceptionOrWhatever
return setting
class GlobalSettings(models.Model):
key = models.CharField(unique=True, max_length=255)
value = models.CharField(max_length=255)
objects = GlobalSettingsManager()
>>> APP_SETTING = GlobalSettings.objects.get_setting('APP_SETTING')
There are apps for this but I prefer looking at them and writing my own.

You can prevent users from adding/deleting an object by overriding this method on your admin class:
ModelAdmin.has_add_permission(self, request)
ModelAdmin.has_delete_permission(self, request, obj=None)

Modification of #radtek answer to prevent deleting if only one entry is left
class SendgridEmailQuotaAdmin(admin.ModelAdmin):
list_display = ('quota','used')
def has_add_permission(self, request):
return False if self.model.objects.count() > 0 else True
def has_delete_permission(self, request, obj=None):
return False if self.model.objects.count() <= 1 else True
def get_actions(self, request):
actions = super(SendgridEmailQuotaAdmin, self).get_actions(request)
if(self.model.objects.count() <= 1):
del actions['delete_selected']
return actions

I had basically the same problem as the original poster describes, and it's easily fixed by overriding modelAdmin classes. Something similar to this in an admin.py file easily prevents adding a new object but allows editing the current one:
class TitleAdmin(admin.ModelAdmin):
def has_delete_permission(self, request, obj=TestModel.title):
return False
def has_add_permission(self, request):
return False
def has_change_permission(self, request, obj=TestModel.title):
return True
This doesn't prevent a user from posting a form that edits data, but keeps things from happening in the Admin site. Depending on whether or not you feel it's necessary for your needs you can enable deletion and addition of a record with a minimum of coding.