We Keep Coding

Exporting queryset data using Django without have to re-query data

I have set up a Django app that queries from a PostgreSQL database. Exporting all the data within a model seems relatively straight forward but what if I wanted to export only the data from a returned queryset.
Currently I'm managing this by having a button on my page and changing the url when the user wishes to export data. The code within the view that I'm rendering is as follows.
def export_data(request,pk):
resource_class = MDPResource()
queryset = MDP.objects.filter(mdp_id=pk)
dataset = person_resource.export(queryset)
response = HttpResponse(dataset.csv, content_type='text/csv')
response['Content-Disposition'] = 'attachment; filename="mdpbasic.csv"'
return response
However, this means that I'm going to be making one query when the user initially opens the page and then a second query (in effect 2 extra) when the user wishes to export the data which I have already queried for in the previous view.
Moreover, in a section of my page I'm also using Q models for complex filtering. The above method is not sufficient for exporting data in that case (I don't want to have to reconstruct the query).
I also do not want to have to write the data to a file (if I can help it) because then that raises the issue of having to delete it using os once the user has exported their data.
Generating CSV file with Django (dynamic content)
I have already checked out this question but I'm not sure how passing a dictionary to a urlpattern works and it still seems as though the query is being made one more time anyways (still one less than my current method).
I don't know if I should be checking out apps for this particular purpose. I've looked into a couple such as django-queryset-csv and django-import-export but none of them seem to do what I've mentioned.
django: export current queryset to csv by button click in browser
I've seen this example that makes use of class-based views but I wanted to see if there was a way to do it with function-based views so I don't have to migrate all my views to class-based ones.

Saving data on server side using REST

I'm using django rest framework and trying to save some data so it will be accessible by GET, PUT, DELETE.
So when user send GET request server send some information (a random number, for example) and that information is needed after user sends PUT request on the same url. How would one save such information? I'm using class-based views.
So i want to save that information on GET method.
I tried saving that information to class variable self.information, but the problem is self.information is empty when PUT method is getting called.
I also tried saving it to session, but like class variable, session is also empty when PUT method is being executed.
class SampleClass(mixins.RetrieveModelMixin, mixins.UpdateModelMixin, generics.GenericAPIView):
serializer_class = SampleSerializer
def get(self, request):
random_number = random.randint(0, 10)
request.session['number'] = random_number;
content = {'random_number': random_number}
return Response(content)
def put(self, request):
number = request.session['number'] # key doesn't exists
process_number(number)
# ...

Before I begin, it's important to note that HTTP is a stateless protocol, and you are looking to add state into the mix. If you can rework what you are doing to not depend on previous requests, that will probably be better in the long run.
I tried saving that information to class variable self.information, but the problem is self.information is empty when PUT method is getting called.
This is because the class is re-initialized for each request. Because of that, the class variables don't persist across requests. Even if they did, that would mean everyone would get access to the persisted value, and it isn't made clear if that is what you are looking for.
I also tried saving it to session, but like class variable, session is also empty when PUT method is being executed
This doesn't work because Django sessions are persisted through the use of cookies. While this might work for SessionAuthentication, it won't work for any authentication that happens outside of the browser. This is because the session cookies won't be included, so Django will think the new requests are under a different session.
Now, just because HTTP is mostly stateless and doing this might lead to future trouble, that doesn't mean that you should never do it. The Django sessions wouldn't exist if there wasn't a need for it, and there are ways to save state without Django sessions.
Create a new model for the state - This is usually the best way to save state per-user and ensure that it doesn't fade away. The model needs a user field along with the fields that the state will be stored in, and all you need to do is have a query that retrieves the state object for the user.
Use the Django cache - This is the way I would recommend it for the case that you specified in your question. When you don't need to store much state, the state is shared among everyone, or you can live with it not existing (expiring), storing the data in a simple cache environment will probably work the best. You have much more control over what is stored, but at the expense of having to do more work.

Django Rest Framework Pagination Since ID

I have built an API with Django Rest Framework. I want to change pagination for a better user experience.
The problem:
The client makes a call to request all posts. The request looks like:
http://website.com/api/v1/posts/?page=1
This returns the first page of posts. However, new posts are always being created. So when the user requests:
http://website.com/api/v1/posts/?page=2
The posts are almost always the same as page 1 (since new data is always coming in and we are ordering by -created).
Possible Solution?
I had the idea of sending an object id along with the request so that when we grab the posts. We grab them with respect to the last query.
http://website.com/api/v1/posts/?page=2&post_id=12345
And when we paginate we filter where post_id < 12345
But this only works assuming our post_id is an integer.
Right now I'm currently only using a basic ListAPIView
class PostList(generics.ListAPIView):
"""
API endpoint that allows posts to be viewed
"""
serializer_class = serializers.PostSerializer # just a basic serializer
model = Post
Is there a better way of paginating? so that the next request for that user's session doesn't look like the first when new data is being created.

You're looking for CursorPagination, from the DRF docs:
Cursor Pagination provides the following benefits:
Provides a consistent pagination view. When used properly CursorPagination ensures that the client will never see the same item twice when paging through records, even when new items are being inserted by other clients during the pagination process.
Supports usage with very large datasets. With extremely large datasets pagination using offset-based pagination styles may become inefficient or unusable. Cursor based pagination schemes instead have fixed-time properties, and do not slow down as the dataset size increases.
You can also use -created as the ordering field as you mentioned above.

How about caching the queryset? So that the next page is served from the same query set, and not from a new one. And then you could use a parameter to get a new queryset when you want.
Something like this:
from django.core.cache import cache
class PostList(generics.ListAPIView):
def get_queryset(self):
qs_key = str(self.request.user.id) + '_key'
if 'refresh' in self.request.QUERY_PARAMS:
# get a new query set
qs = Post.objects.all()
cache.set(qs_key, qs)
return cache.get(qs_key)
So basically, only when your url will be like this:
http://website.com/api/v1/posts/?refersh=whatever
the request will return new data.
UPDATE
In order to provide each user with it's own set of posts, the cache key must contain an unique identifier (which might be the user's ID):
I also updated the code.
The downside to this approach is that for a very large number of users and a large number of posts for each user, it might not work very well.
So, here is my second idea
Use a TimeStamped model for the Post model, and filter the query set based on the created field.
I don't know much about your models and how exactly they are built, so I guess you will have to choose which solution is best for your app :)

Maybe you can add a field to every object like "created_at/updated_at". Then you can save the timestamp when the user made the request and filter out everything that came after it.
Haven't tried it myself but I guess it might work on your case

Submitting Multiple Forms At The Same Time (Edit Profile Page)

My question I suppose is rather simple. Basically, I have a profile. It has many variables being passed in. For instance, name, username, profile picture, and many others that are updated by their own respective pages. So one page would be used to update the profile picture, and that form would submit data from the form to the handler, and put() it to the database. What i'm trying to do here, is put all of the forms used to edit the profile on one single page at the same time.
Would I need one huge handler to deal with that page? When I hit 'save' at the bottom of the page, how do I avoid overwriting data that hasn't been modified? Currently, say I have 5 profile variables, they map to 5 handlers, and 5 separate pages that contain their own respective form.
Thanks.

I've used django on most of my webapps, but the concept should be the same; I use ajax to send the data to the backend whenever the user hits submit (and the form returns false) so the user can keep editing it. With ajax, you can send the data to different handlers on the backend. Also, using jQuery, you can set flags to see if fields have been changed, to avoid sending the ajax message in the first place. Ajax requests behave almost exactly like standard HTTP requests, but I believe the header indicates AJAX.
If you're looking at strictly backend, then you will need to do multiple "if" statements on the backend and check one field at a time to see if it has been changed. On the backend you should still be able to call other handlers (passing them the same request).

Better way of passing form parameters into hidden form in Pyramid

In a previous question, I was trying to figure out the right strategy for to passing data between forms in Pyramid. Based on the answer I received, I decided the approach of using a hidden form.
I started implementing this and think there must be a better way of passing along the data. Specifically, passing parameters through the url results in a tuple that is messy to parse.
I want it to be general enough to not to know what parameters the form has and also it needs to handle file fields as well.
How I'm currently attempting to pass the form data to the confirmation page:
#view_config(renderer="templates/derived/load/error.mak", route_name='process_model_route')
def process_model(self):
#processing logic and validaton, failiure in validation sends user to error.mak
return HTTPFound(route_url('confirm_model_route', self.request, fparams=self.request.POST))
Route: config.add_route('confirm_model_route', 'rnd2/model/confirm/*fparams')
#view_config(renderer="templates/derived/confirm/model.mak", route_name='confirm_model_route')
def confirm_model(self):
form_dict = self.request.matchdict['fparams']
#need to decode and pass to template
return dict({'load_route':load_route, 'form_dict':form_dict})
The confirm/model.mak template would contain the hidden form.

The idea with this method is:
Client visits page.
Server renders the form.
Client fills in form and POSTs to URL.
Server renders a new page that contains a hidden form with all of the data it just received in the POST.
Client POSTs to a URL, confirming the submission.
Server persists the data from the hidden form and redirects.
Now depending on usability, it's up to you to decide how many different URLs you actually want here and how many views in Pyramid. You have to think about what happens with invalid data?
Notice in the outline above, once the user POSTs the form to a URL, that URL must return the confirmation page containing a hidden form. If you try to redirect the user to a confirmation page instead, you must persist the data somehow, either in a session or through the hack you showed in your example (shoving all of the data into the GET). The second solution is very bad because it abuses the true purpose of GET in HTTP.
There is also the convention that every POST should result in a redirect to avoid a client submitting the form multiple times. With this in mind you might consider the simple solution of rejecting POSTs that do not have a "confirmed" flag and simply setting the "confirmed" flag in javascript after prompting the user. This allows you to keep your form handling logic simple.
If you don't want to rely on javascript and you don't want to persist the form data in a session, then you run into the issue of not redirecting after the first POST but other than that it should be simple from the outline above.