I have a website using Django. Each post is an object called Article. I want to retrieve the post's HTML after saving it so I wrote the following post_save hook:
#receiver(models.signals.post_save, sender=Article)
def _send_article_mentions(sender, instance, **kwargs):
import requests
from django.contrib.sites.models import Site
from urlparse import urljoin
from ParallelTransport.settings import ARTICLES_URL
SITE_URL = 'http://'+Site.objects.get_current().domain
article_url = urljoin( SITE_URL, instance.get_absolute_url() )
import time
time.sleep(20)
r = requests.get(article_url)
error_file = open(ARTICLES_URL+'/'+'error.txt','w')
error_file.write('file started1\n')
m = r.status_code
error_file.write(str(m))
error_file.close()
It basically waits for 20s (added as a test) then tries to retrieve the HTML of the post using its URL, and writes the request status code to a file for debugging.
The problem is I always get status = 404 on the first save, it does work on 2nd and subsequent saves. I thought the way Django works would be, in order:
save instance to database using save(). At this point the post would get a URL
send post_save signal
But then I should be able to retrieve the HTML in post_save. Am I understanding post_save incorrectly?
Added notes:
Putting this code in save() method does not work. Nor should it. The post is added to database at the end of the save() method and so should not have any URL until save() ends.
This is on a production site, not on the development server.
I want to use the links in the HTML to send 'pingbacks' or actually webmention. But all my pingbacks are being rejected because the post does not have a URL yet. This is the bare minimum code that does not work.
Although this is a completely wrong approach(*), the problem is probably in database transactions. Current thread saves the article but within this uncommited transaction you are trying to get these data through another thread (through web server). In that case, this behaviour is fully correct. Either you need to commit before retrieving through another thread or get the HTML by another way.
(*) should be done asynchronously on the background (Celery or other more lightweight async queue app) or you can call the view directly if you want to get the HTML (depending on your view, you may have to forge the request; if too complicated, you can create a helper function that cherrypicks minimal code to render the template). If you only need to call a 3rd party API after you save something, you want to do it asynchronously. If you don't do it, the success of your "save() code" will depend on the availability of your connection or the 3rd party service and you will need to deal with transactions on place where you won't to deal with transactions ;)
Have you tried overriding the object's save method, calling super, waiting and then trying to retrieve the HTML?
Also are you using the development server? It may have issues handling the second request while the first one is still going. Maybe try it on a proper server?
I had similar problem caused probably by the same issue (Asked differently, https://plus.google.com/u/0/106729891586898564412/posts/Aoq3X1g4MvX).
I did not solve it in a proper way, but you can try playing with the database cache, or (seen it in another django database problem) close all of the database connections and requery.
Edit 2:
I have created a simple example (using Django 1.5.5) to test whether this works as intended. As far as I can tell, it does. pre_save fires before a database commit and post_save fires after.
Example detailed:
Two example models.
Article is used for triggering signals.
ArticleUrl is used to log responses from Article.get_absolute_url().
# models.py
from django.db import models
from django.core.urlresolvers import reverse
class Article(models.Model):
name = models.CharField(max_length=150)
def get_absolute_url(self):
"""
Either return the actual url or a string containing '404' if reverse
fails (Article is not saved).
"""
try:
return reverse('article:article-detail', kwargs={'pk': self.pk})
except:
return '404'
class ArticleUrl(models.Model):
article_name = models.CharField(max_length=150)
url = models.CharField(max_length=300)
def __unicode__(self):
return self.article_name + ': ' + self.url
Example views.py and urls.py were omitted as they are simple. I can add them if needed.
# views.py, url.py
Creating pre_save and post_save signals for Article:
# signals.py
from django.db.models import signals
from django.dispatch import receiver
from article.models import Article, ArticleUrl
#receiver(signals.pre_save, sender=Article)
def _log_url_pre(sender, instance, **kwargs):
article = instance
article_url = ArticleUrl(
article_name = 'pre ' + article.name,
url = article.get_absolute_url()
)
article_url.save()
#receiver(signals.post_save, sender=Article)
def _log_url_post(sender, instance, **kwargs):
article = instance
article_url = ArticleUrl(
article_name = 'post ' + article.name,
url = article.get_absolute_url()
)
article_url.save()
Importing my signals.py so Django can use it:
# __init__.py
import signals
After defining the above I went ahead and created a new Article in Django shell (python.exe manage.py shell).
>>> from article.models import *
>>> a = Article(name='abcdd')
>>> a.save()
>>> ArticleUrl.objects.all()
[<ArticleUrl: pre abcdd: 404>, <ArticleUrl: post abcdd: /article/article/8>]
The above example does seem to show that pre_save did indeed not return a url, but post_save did. Both appear to be behaving as intended.
You should check whether your code either deviates from the above example or interferes with program execution in some way.
Edit 1:
Having said that (below), according to What Happens When You Save, the post_save signal should run after a database save.
Could there be other parts of your app/site somehow interfering with this?
Original post:
According to the Django documentation, the post_save signal is sent at the end of save(), not after it.
As far as I understand, Django Signals are synchronous (in-process) so they would stall the actual save(). It won't fully complete until the signals are done.
This isn't always applicable, but have you considered a custom signal you could call after save()?
Related
I've run into a strange issue with Django Rest Framework testing engine. The weird thing is that everything used to work fine with Django 3 and this issue turned up after I migrated to Django 4. Apart from testing, everything works well, and responds to queries as expected.
The problem
I'm using DRF APIClient to make queries for unit tests. While GET requests perform predictably, I fail to make POST requests work.
Here is some minimalistic example code I created to figure out the issue. The versions I'm using:
Python 3.9
Django==4.0.3
djangorestframework==3.13.1
from django.db import models
from django.urls import include, path
from django.utils import timezone
from rest_framework import routers, serializers, viewsets
router = routers.DefaultRouter()
# models.py
class SomeThing(models.Model):
created_at = models.DateTimeField(default=timezone.now)
title = models.CharField(max_length=100, null=True, blank=True)
# serializers.py
class SomeThingSerializer(serializers.ModelSerializer):
class Meta:
fields = "__all__"
model = SomeThing
# views.py
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
# urls.py
router.register("some-things", SomeThingViewSet, basename="some_thing")
app_name = 'question'
urlpatterns = (
path('', include(router.urls)),
)
Here is my test case:
import json
from django.contrib.auth import get_user_model
from rest_framework import status
from rest_framework.test import APITestCase, APIClient
class TestUserView(APITestCase):
self.some_user = get_user_model().objects.create(login="some_user#test.ru")
#staticmethod
def get_client(user):
client = APIClient()
client.force_authenticate(user=user)
return client
def test_do_something(self):
client = self.get_client(self.compliance_chief)
url = reverse('question:some_things-list')
resp = client.post(
path=url,
data=json.dumps({"title": "Created Something"}),
content_type="application/json",
)
assert resp.status_code == status.HTTP_201_OK
(Yes, I have to use some authentication to get access to the data, but I don't think it is relevant to the problem.) To which I receive a lengthy traceback, ending with an assertion error:
File "/****/****/****/venv/lib/python3.9/site-packages/django/test/client.py", line 82, in read
assert (
AssertionError: Cannot read more than the available bytes from the HTTP incoming data.
As it is really fairly long, I'll leave it just in case in a gist without posting it here.
Steps to fix
The problem clearly happens after the correct response is returned by the viewset. To make sure the response is correct I made a slight customisation in the create method to print out the response before it is returned, like so:
class SomeThingViewSet(viewsets.ModelViewSet):
queryset = SomeThing.objects.all().order_by('id')
serializer_class = SomeThingSerializer
def create(self, request, *args, **kwargs):
response = super().create(request, *args, **kwargs)
print("THIS IS THE RESPONSE FROM THE VIEWSET", response)
return response
And, sure enough, the result is correct:
THIS IS THE RESPONSE FROM THE VIEWSET <Response status_code=201, "text/html; charset=utf-8">
Which makes me think something goes wrong at the parsing stage (actually, the traceback implies the same). I tried to tweak the way I build the query, namely:
using format instead of content type like so: resp = client.post(path=url, data={"title": "Created Something"}, format="json")
using the .generic method instead of .post like so: resp = client.generic(method="POST", path=url, data=json.dumps({"title": "Created Something"}), content_type="application/json")
The result is the same.
From googling I found out that this error indeed has occasionally occurred in connection with DRF APIClient and Django, but really long ago (like this discussion, which claims that the issue was fixed in the later versions of Django).
I'm sure the reason for this behaviour is rather obvious (some stupid mistake most likely) and the solution must be very simple, but so far I've failed to find it. I would be very grateful if somebody shared their experience, if there is any, of dealing with such an issue, or their considerations as to where to move from this deadlock.
Alright, the mystery's been resolved and I'm going to share it here in case somebody runs into something similar, although it would take quite a coincidence, so it is unlikely.
Long story short: I messed up the source code of my Django4.0.3. installed in this project.
Now, how it happened. While I was testing some stuff, I ran into an error, which I failed to locate, so I went along the whole chain of events checking if the output was what I expected it to be. Soon enough I found myself checking the output from functions in the libraries installed under my virtual environment. I realise it's a malpractice to directly modify their code, but as I was working in my local environment with an option to reinstall everything at any moment, I decided it was fine to play with them. As it resulted in nothing, I removed all the code I had added (or so I thought).
After a while I realised what caused the initial error (an overlooked condition in my testing setup), fixed it and tried to run the test. That's when the problem in question showed up.
Later I found out that the same very test performs correctly in an identical environment. Then I suspected that I broke something in my local library code. Next I simply compared the code I had dealt with in my local environment with the code from the official source and soon enough I established the offending line. It happened to be in django/test/client.py, in the definition of the RequestFactory.generic method. Something like this:
...
if not r.get("QUERY_STRING"):
# WSGI requires latin-1 encoded strings. See get_path_info().
query_string = parsed[4].encode().decode("iso-8859-1")
r["QUERY_STRING"] = query_string
req = self.request(**r)
return self.request(**r)
...
The offending line (which I added and forgot to remove) was req = self.request(**r). After I deleted it, everything returned back to normal.
I am using the Django Test client (django.test.Client) to run view tests. Upon attempting to use the Test Client on my index function that handles post requests for logins, it continually fails the test even though the authentication successfully occurs.
Heres my test:
def test_login(self):
response = self.client.post('/login/', {'username':'user', 'password':'pass'})
print response.content
self.assertIn(SESSION_KEY, self.client.session)
So the reason i know the login process successfully works is because response.content yields HTML data from another view that can only be access if request.user.is_authenticated() is true. In other words, they must be logged in for response.content to yield the "logged in page". So given this, i can tell that the function obviously works for its practical purpose of logging the user in, however, i've been scouring the docs for hours trying to figure out why i can't access SESSION_KEY from the client session. All my reading suggests that the django test client is in fact stateful in nature and should store the session.
Can someone shed some light on this?
Ok after much searching and asking around on #django, i made a working solution for Django 1.6.x
from django.contrib.auth import SESSION_KEY, get_user_model
from django.test import Client
def setUp(self):
self.client = Client()
def test_login_view(self):
user_pk = get_user_model()._default_manager.get(username__exact='test_username_here').pk
response = self.client.post('/login/', {'username':'test_username_here', 'password':'test_password_here'})
self.assertEqual(self.client.session[SESSION_KEY], user_pk)
The test_login_view function will be the one evaluating the view in my app that handles user logins from the template form. First, i grab user_pk which is the real primary key of the given user in the database. I used get_user_model() instead of User.objects.get() because the former allows you to reference regardless of whether the User model is modified or not. Of course you can use the latter as well. Second, i go ahead and send the post request using the test client just like a standard user's browser would. Finally, i discovered that self.client.session[SESSION_KEY] contains the primary key of the logged in user. (If the login was successful, otherwise, it will simply yield a KeyError)
I'm completely new to testing in Django. I have started by installing nose and selenium and now I want to test the following code (below) It sends an SMS message.
This is the actual code:
views.py
#login_required
def process_all(request):
"""
I process the sending for a single or bulk message(s) to a group or single contact.
:param request:
"""
#If we had a POST then get the request post values.
if request.method == 'POST':
batches = Batch.objects.for_user_pending(request.user)
for batch in batches:
ProcessRequests.delay(batch)
batch.complete_update()
return HttpResponseRedirect('/reports/messages/')
So where do I start? This is what I have done so far...
1) created a folder called tests and added init.py.
2) created a new python file called test_views.py (I'm assuming that's correct).
Now, how do I go about writing this test?
Could someone show me with an example of how I write the test for the view above?
Thank you :)
First of all, you don't need selenium for testing views. Selenium is a tool for high-level in-browser testing - it's good and useful when you are writing UI tests simulating a real user.
Nose is a tool that makes testing easier by providing features like automatic test discovery, supplies a number of helper functions etc. The best way to integrate nose with your django project is to use django_nose package. All you have to do is to:
add django_nose to INSTALLED_APPS
define TEST_RUNNER = 'django_nose.NoseTestSuiteRunner'
Then, every time you run python manage.py test <project_name> nose will be used to run your tests.
So, speaking about testing this particular view, you should test:
login_required decorator work - in other words, unauthenticated user will be redirected to the login page
if request.method is not POST, no messages sent + redirect to /reports/messages
sending SMS messages when POST method is used + redirect to /reports/messages
Testing first two statements is pretty straight-forward, but in order to test the last statement you need to provide more details on what is Batch, ProcessRequests and how does it work. I mean, you probably don't want to send real SMS messages during testing - this is where mocking will help. Basically, you need to mock (replace with your own implementation on the fly) Batch, ProcessRequests objects. Here's an example of what you should have in test_views.py:
from django.contrib.auth.models import User
from django.core.urlresolvers import reverse
from django.test.client import Client
from django.test import TestCase
class ProcessAllTestCase(TestCase):
def setUp(self):
self.client = Client()
self.user = User.objects.create_user('john', 'lennon#thebeatles.com', 'johnpassword')
def test_login_required(self):
response = self.client.get(reverse('process_all'))
self.assertRedirects(response, '/login')
def test_get_method(self):
self.client.login(username='john', password='johnpassword')
response = self.client.get(reverse('process_all'))
self.assertRedirects(response, '/reports/messages')
# assert no messages were sent
def test_post_method(self):
self.client.login(username='john', password='johnpassword')
# add pending messages, mock sms sending?
response = self.client.post(reverse('process_all'))
self.assertRedirects(response, '/reports/messages')
# assert that sms messages were sent
Also see:
https://docs.djangoproject.com/en/dev/topics/testing/
Django Testing Guide
Getting Started with Python Mock
Hope that helps.
I am trying to log the IP address of the user that is trying to login using signals. How do I do this?
I have already captured the datetime for the login.
#models.py
class UserLogin(models.Model):
"""user details when logging in"""
user = models.ForeignKey(User)
timestamp = models.DateTimeField(auto_now=True)
This is for the signal:
#models.py
def user_login_save(sender, instance, **kwargs):
if instance.last_login:
old = instance.__class__.objects.get(pk=instance.pk)
if instance.last_login != old.last_login:
instance.userlogin_set.create(timestamp=instance.last_login)
models.signals.post_save.connect(user_login_save, sender=User)
Although I know how to get the IP Address using: request.META[REMOTE_ADDR] my problem is that I cannot use the request instance in my model. I am not also sure if getting something from the request is good practice.
What is the recommended way of doing this?
Any reply will be greatly appreciated.
Regards,
Wenbert
Although I know how to get the IP Address using: request.META[REMOTE_ADDR] my problem is that I cannot use the request instance in my model.
That's why you have view functions.
I am not also sure if getting something from the request is good practice.
It is perfectly fine.
That's why the request is provided to every view function.
That's why you have view functions.
Just do it in the view function. Don't mess with signals unless you're writing a new kind of database interface.
As the instance is passed into the signal and the instance is really the same object that is saved, you can attach a request object or IP to the instance upon saving it.
user_login.request=request
user_login.save()
And retrieve it in the signal like
instance.request
Django has a signal that fires whenever a user logs in that is called user_logged_in. By attaching a signal handler to this signal you can be notified of login events. The handler is passed the user and request objects. As you note, you can get the IP address from the request object.
This is a very old question, but I had the same problem today and this was the first hit at Google. So maybe this helps other people as well. This is how it works for me.
from django.contrib.auth.signals import (
user_logged_in,
user_logged_out,
user_login_failed,
)
from django.dispatch import receiver
import logging
user_logger = logging.getLogger("user")
#receiver(user_logged_in)
def log_user_login(sender, user, **kwargs):
"""log user "login" to log-file setup in django settings.py"""
# if the IP-ADRESS is 127.0.0.1 from external users then check this
# https://stackoverflow.com/questions/4581789/how-do-i-get-user-ip-address-in-django
request = kwargs["request"]
ip_address = request.META.get("REMOTE_ADDR")
user_logger.info(f"{user} login successful. IP-Address: {ip_address}")
# other functions deleted for brevity
I am looking into adding RSS feeds to one of my Django apps and I would like to be able to have them be authenticated.
I want to use the new syndication framework in Django 1.2. I've read the docs on how to do this and have the basic feeds setup.
I am new to authenticating feeds, so I am not sure what the best approach to take is or what my options really are.
Each user has a unique sub domain and I would like the URL structure to look something like this: http://mysubdomain.mysite.com/myapp/rss/ if possible.
I don't want the feeds to be publicly available, is it possible to use the users username and password for the authentication? Have you found that most feed readers support this? If it's not possible to authenticate for each user, should I try to use a uuid to give them a unique url or is that not secure enough?
As you can probably tell I am not sure what direction to take with this, so any advice on the best way to do this would be very much appreciated.
Thanks
This is an old thread, but I recently encountered the same question. I solved it by overloading the __call__ method of the Feed object:
from django.http import HttpResponse
class ArticleFeed(Feed):
"snip [standard definitions of title, link, methods...]"
def __call__(self,request,*args,**kwargs):
if not request.user.is_authenticated():
return HttpResponse(status=401)
else:
return super().__call__(request,*args,**kwargs)
Have you tried wrapping the syndication view django.contrib.syndication.views.feed into a view that requires login? RSS feeds should normally be fetched over HTTP, so this should work!
# Import Django's standard feed view.
from django.contrib.auth.decorators import login_required
from django.django.contrib.syndication.views import feed
# Wrap it in a new feed view that requires authentication!
private_feed = login_required(feed)
Caveat: I've never tried this!
Edit!
To be safe with RSS readers that don't support redirection, return a HTTP 401 status code with the following:
authentication_url = '/accounts/login'
def feed_safe_login_required ( view ):
def _ ( request, *args, **kwargs ):
if not request.user.is_authenticated:
return HttpResponseNotAuthorized, authentication_url
return _
feed = feed_safe_login_required(django.contrib.syndication.views.feed)
Where HttpResponseNotAuthorized is as defined in this django snippet.