We Keep Coding

Having one single generic Viewset that can be resolved through its router basename on Django Rest Framework

My question is more about refactoring, best practices and potential vulnerability while organizing my routes and views in DRF.
My app is really a simple one and all the views are to be described in the same fashion. Take my views.py file for example:
# views.py
class TrackView(viewsets.ModelViewSet):
queryset = Track.objects.all()
serializer_class = TrackSerializer
class AlbumView(viewsets.ModelViewSet):
queryset = Album.objects.all()
serializer_class = TrackSerializer
class ArtistView(viewsets.ModelViewSet):
queryset = Artist.objects.all()
serializer_class = ArtistSerializer
class ChartView(viewsets.ModelViewSet):
queryset = Chart.objects.all()
serializer_class = ChartSerializer
And in urls.py I can do:
#urls.py
router = DefaultRouter()
router.register(r'tracks', TrackView)
router.register(r'albums', AlbumView)
router.register(r'artists', ArtistsView)
router.register(r'charts', ChartsView)
urlpatterns = [
path('', include(router.urls)),
path('api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]
In this scenario every new View is a copied logic of each other only changing the name of the model they use, and for every new model added that needs a view I would basically have to create the same repeated code in urls.py and views.py.
Instead of doing like that I refactored it like this:
# views.py
from rest_framework import viewsets
class APIView(viewsets.ModelViewSet):
queryset = None
def get_queryset(self):
return eval(self.basename).objects.all()
def get_serializer_class(self):
return eval(self.basename+'Serializer')
# urls.py
from django.urls import path, include
from rest_framework.routers import DefaultRouter
from top_charts.views import *
from .routes import ROUTE_NAMES
router = DefaultRouter()
for route in ROUTE_NAMES:
router.register(r''+route.lower()+'s', APIView, basename=route)
urlpatterns = [
path('', include(router.urls)),
path('api-auth/', include('rest_framework.urls', namespace='rest_framework'))
]
# Here in routes.py would be the only place I'd need to update when I have a new model
ROUTE_NAMES = [
'Track',
'Album',
'Artist',
'Chart'
]
It works, and I can access the urls of all endpoints just fine and it has a much cleaner code. My question is: is this a good practice or do you see any problem with my implementation?
There are always concerns when it comes to the use of eval() but I could not think pragmatically what could go wrong in here since ROUTE_NAMES is the one that would define the evaluation and that would totally be under my control, so it doesn't seem like a place where I leave for potential vulnerability.
Also I don't see this sort of implementation suggestion anywhere in DRF documentation. They always go for the repeated code logic of views, which left me suspicious and wondering why.

Part by part:
About the URLs, I don't think what you have done is very wrong, but it is certainly not a good practise. I see three main issues.
Having the routes in the URLs file makes you able to see the views and paths directly. Separating those into two files is not really useful. You save 2 lines of code, but that doesn't make the code more efficient and it doesn't make it cleaner either.
A problem that you might have is that whenever you need to add a route that doesn't follow the same pattern as all other ones you have, you'll have to add it separately. Or if you have to modify one of the already existing ones you'll have to remove the item from the list and, again, add another route separately. You've got more files and data to maintain but every route could be clearly defined in one line in urls.py.
Suppose you have the following:
ROUTE_NAMES = [
'Track',
'Album',
'Artist'
'Chart'
]
It looks the same, right? But a comma is missing after the third element, and now you've got an app that works in a completely different without crashing while running the server, because the code is actually completely fine. Having the routes that way is a source of errors that would be easily avoided just by having one router.register per line.
About the views, I also have 2 main concerns.
The first one is basically the points 1 and 2 I've explained about the URLs combined: you save lines of code but that doesn't make the code that more efficient, and the moment you need to add more functionalities to one view you'll have to remake it because there's no way for you to modify only one leaving the others the way they are.
A big concern is on the use of eval, as you had already guessed. Yes, I too fail to see a way to exploit that, but isn't it better to just not use it? If you really need to save those few lines of code, there are other ways to do what you intend without using eval or other dangerous methods. For example, using getattr:
from rest_framework import viewsets
from . import models, serializers
class APIView(viewsets.ModelViewSet):
queryset = None
def get_queryset(self):
return getattr(models, self.basename).objects.all()
def get_serializer_class(self):
return getattr(serializers, self.basename + 'Serializer')
This way, the exact same is achieved without using a too dangerous method, limiting the items you can get to whatever has been declared or imported in models.py and serializers.py, instead of allowing the execution of any expression. However, this is still not recommended because the problem I've stated in point 1 remains.
At the start of your question you talk about refactoring. I would say that Django and DRF don't really consider refactoring in their guides and docs because they provide very simple examples and use cases for their software. Your example application is very small too, and the code is very clean, so there's no real reason to refactor that. I could talk for hours about when and how to refactor code, but as a general rule don't refactor anything that is clearly understandable if efficiency is not critical.

Django 3.1 - "OperationalError: no such table" when using an ORM Class Model before making or applying a migration

TL;DR If I have a Django model for which I create a migration, I can run the migration and create the table in the DB, then I can proceed to reference the model in the code. Now, when I merge the code into master, it will both contain the migration files and the code that references the Django model. When I pull this onto a new machine (e.g. staging/prod), I will want to run the migrations in this new environment so as to have an updated DB. correct? This operation seems impossible though because the code that references the model will want the table to already exist even to just perform the migration itself, otherwise the migration command fails. How to solve this issue?
The Problem
Whenever I create a new model class inside products/models.py (products is the name of my Django app), I cannot reference such model (e.g. Product model) in any method or class before having made and run the migrations. Otherwise, if I reference the new model in the code and then I run any of the following commands/procedures, I receive the OperationalError: no such table error:
python manage.py runserver
python manage.py makemigrations [appname]
python manage.py migrate
python manage.py showmigrations
Delete all existing migrations and try to regenerate them with python manage.py makemigrations [appname]
Delete current db.sqlite3 file; delete all existing migrations and try to regenerate them with python manage.py makemigrations [appname]
The error looks like this:
...
File "/Users/my_user/Work/django_proj_1/config/urls.py", line 21, in <module>
path('', include('products.urls')),
...
File "/Users/my_user/Work/django_proj_1/products/urls.py", line 1, in <module>
from products.api_views import ProductList3
File "/Users/my_user/Work/django_proj_1/products/api_views.py", line 7, in <module>
class ProductList3(ListAPIView):
File "/Users/my_user/Work/django_proj_1/products/api_views.py", line 8, in ProductList3
queryset = get_products()
File "/Users/my_user/Work/django_proj_1/products/data_layer/product_data_layer.py", line 4, in get_products
all_prods = list(Product.objects.values())
...
File "/Users/my_user/.local/share/virtualenvs/django_proj_1-a2O6RBaf/lib/python3.8/site-packages/django/db/backends/sqlite3/base.py", line 413, in execute
return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: products_product
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
The Real Question
This behavior may seem normal, but if you think about it, how would a developer make a release in production where a new model is created and also used in the code simultaneously (given that running migrations will fail if you referenced the created model into the code)?
I found similar questions (which I linked below), but none of them seems actually relevant for a real-world large scale production project. The only 'solution' in the similar questions is to:
comment out the code (all functions in the layer/chain) that end up interacting with the new model that does not exist in the DB yet
(make and) run the migrations
revert the comments
This seems pretty manual and not feasible for real-world production applications. I can understand if the issue was for local development, in which I can
create a model,
quickly make and run the migrations
and then start coding and use that model.
However, how is it possible to go about this in a real-world production scenario? By this I mean, how is it possible to release a single PR of code where I create a model as well as use the model in the code and make sure I can successfully run the migration on the production machine upon release of such code?
More Context of my problem
Let's assume I have the following app in Django called products with, among others, the following files:
...
|_products/ # one of the Django apps in the project
|
|_ models.py # where I defined the Product model
|_ data_layer/ # a python module to separate all DB logic
| |_ __init__.py
| |_ product_data_layer.py # file where I put the function that references the Product model
|_ api_views.py # file where I call a function from the data layer
|_ urls.py # file where I create an API endpoint that references api_views.py
|
...
Code inside the above files:
# products/urls.py content
from products.api_views import ProductList3 # THORWS ERROR
from django.urls import path
urlpatterns = [
path('api/v1/products', ProductList3)
]
# products/api_views.py content
from rest_framework.generics import ListAPIView
from products.serializers import GenericProductSerializer
from products.data_layer.product_data_layer import get_products
class ProductList3(ListAPIView): # THORWS ERROR
queryset = get_products() # THORWS ERROR
serializer_class = GenericProductSerializer
# products/data_layer/product_data_layer.py content
from products.models import Product
def get_products():
all_prods = list(Product.objects.values()) # THORWS ERROR (main problem)
# all the logic I need
return all_prods
# products/models.py content
from django.db import models
class Product(models.Model):
product_name = models.CharField(max_length=100)
product_description = models.TextField('Main Product Description')
def __str__(self):
return self.product_name
Similar questions
The following questions are similar questions that I looked at, but besides some dev-process solutions, I found no substantial answer on how to tackle this issue once for all.
django migration no such table 1 - unclear as to how not to execute code on import
django migration no such table 2 - no answers as of 18/03/2021
django migration no such table 3 - does not work in my case

You write in your class declaration:
queryset = get_products()
And this function has this line:
all_prods = list(Product.objects.values())
What happens here? Anything written directly inside a class is executed when the class is being created. If it were a method of the class then that would not be evaluated. So get_products() is called when the class is being created. Next list(Product.objects.values()) is called where calling list on the queryset forces the query to be evaluated. Now obviously you are making migrations and the tables don't exist yet giving you an error.
Firstly this code of yours is bad design even if there are changes to the table Product your view will still only display the products that were gotten in the first query made when the server was started up. This is obviously unintended and not what you want to do.
What is the solution? Obviously your query should reside somewhere where it is actually meant to be. Mostly in Django with classes that get a queryset there is a method named get_queryset that does this task, so you should be overriding that:
class ProductList3(ListAPIView):
serializer_class = GenericProductSerializer
def get_queryset(self):
return get_products() # More sensible to write it here.
# If concerned about overriding something you can use the below which would be equivalent
# self.queryset = get_products()
# return super().get_queryset()

Python Django Rest Framework UnorderedObjectListWarning

I upgraded from Django 1.10.4 to 1.11.1 and all of a sudden I'm getting a ton of these messages when I run my tests:
lib/python3.5/site-packages/rest_framework/pagination.py:208:
UnorderedObjectListWarning:
Pagination may yield inconsistent results with an unordered object_list:
<QuerySet [<Group: Requester>]>
paginator = self.django_paginator_class(queryset, page_size)
I've traced that back to the Django Pagination module:
https://github.com/django/django/blob/master/django/core/paginator.py#L100
It seems to be related to my queryset code:
return get_user_model().objects.filter(id=self.request.user.id)
How can I find more details on this warning? It seems to be that I need to add a order_by(id) on the end of every filter, but I can't seem to find which code needs the order_by added (because the warning doesn't return a stack trace and so it happens randomly during my test run).
Thanks!
Edit:
So by using #KlausD. verbosity tip, I looked at a test causing this error:
response = self.client.get('/api/orders/')
This goes to OrderViewSet but none of the things in get_queryset cause it and nothing in serializer class causes it. I have other tests that use the same code to get /api/orders and those don't cause it.... What does DRF do after get_queryset?
https://github.com/encode/django-rest-framework/blob/master/rest_framework/pagination.py#L166
If I put a traceback into pagination then I get a whole bunch of stuff related to django rest framework but nothing that points back to which of my queries is triggering the order warning.

So in order to fix this I had to find all of the all, offset, filter, and limit clauses and add a order_by clause to them. Some I fixed by adding a default ordering:
class Meta:
ordering = ['-id']
In the ViewSets for Django Rest Framework (app/apiviews.py) I had to update all of the get_queryset methods as adding a default ordering didn't seem to work.

I was getting this warning when i used objects.all() in my view.py
profile_list = Profile.objects.all()
paginator = Paginator(profile_list, 25)
to fix this i changed my code to :
profile_list = Profile.objects.get_queryset().order_by('id')
paginator = Paginator(profile_list, 25)

In my case, I had to add order_by('id') instead of ordering.
class IntakeCaseViewSet(viewsets.ModelViewSet):
schema = None
queryset = IntakeCase.objects.all().order_by('id')
Ordering needs to be in the model using Class Meta (not View).

Let me give an answer updated to new developments...
https://code.djangoproject.com/ticket/6089
The default ordering of the User model has been removed in Django. If you found yourself at this page because of an upgrade, it's very likely connected to this change.
There are 2 versions of this problem you might be dealing with.
your own model does not have a default ordering in its Meta (see accepted answer)
you are using a model from an app you are using as a dependency which does not have a default ordering
Since literally the Django User model itself does not adhere to ordering, it's very clear that the second scenario cannot be resolved by asking the maintainers of those dependencies to put in a default ordering. Okay, so now you either have to override the model being used for whatever your doing (sometimes a good idea, but not good for addressing such a minor issue).
So you're left with addressing it on the view level. You also want to do something that will play nicely with any ordering filter class you have applied. For that, set the view's ordering parameter.
class Reviewers(ListView):
model = User
paginate_by = 50
ordering = ['username']
Also see Is there Django List View model sort?

Another option is to add OrderingFilter
http://www.django-rest-framework.org/api-guide/filtering/#orderingfilter

Including this didn't work for me.
class Meta:
ordering = ['-id']
But changing get_queryset(self) and sorting the list with .order_by('id') did. Maybe worked because I'm using filters, I don't know
class MyView(viewsets.ModelViewSet):
queryset = MyModel.objects.all()
serializer_class = MySerializerSerializer
def get_queryset(self):
user = self.request.user
return MyModel.objects.filter(owner=user).order_by('id')

Update the model meta class instead.
class UsefulModel(models.Model):
class Meta:
ordering='-created' # for example
you can still override the ordering from the view attribute 'ordering' As advised by AlanSE previously.
class UsefulView(ListView):
ordering = ['-created']

In my case, it expected a tuple and that tuple has to contain a comma even when you are parsing just an item in it.
class Meta:
ordering = ('-name',)

Django: Is separating views.py into its own module a good idea?

Dilemma
My views.py gets pretty unwieldy, so I want to separate it into a separate views module inside of my app. However, I'm not sure this is a good idea, for two reasons:
If my views file is the same name as the app name, I cannot import the model without using django.db.get_model, therefore I am worried my approach may be flawed. I have heard it is best practice to avoid name collision within modules; should I rename my view files?
I'm not sure if creating a views module is considered good practice within the Django community in general.
Example
For example, for an app named blogs, with a Blog model and a Post model:
blogs/
__init__.py
models.py
urls.py
views/
__init__.py
blogs.py
posts.py
Here is my blogs.views.blogs:
# project/blogs/views/blogs.py
from django.db.models import get_model
from django.shortcuts import get_object_or_404
from django.views.generic import ListView, DetailView
# Cannot import model directly, results in `ImportError: No module named models`.
# This can be resolved if I resolve the name collision between this file and
# the app itself.
#
# from blogs.models import Blog
class BlogListView(ListView):
model = get_model('blogs', 'Blog')
def get_queryset(self):
return self.model.objects.all()
class BlogDetailView(DetailView):
model = get_model('blogs', 'Blog')
def get_object(self):
blog_pk = self.kwargs.get('blog_pk')
return get_object_or_404(self.model.objects, pk=blog_pk)
Question
My question is twofold:
Should I be separating my views in the first place?
If so, is using get_model a good idea, or is there a way I can import my model directly without using this method? Or should I change my view file names, for example by adding the suffix _views (e.g.: blogs.views.blogs_views.py) in order to avoid the problem altogether?

I cannot import the model without using django.db.get_model
You can: from project_name.app_name.models import MyModel And it's preferable way, 'relative imports for intra-package imports are highly discouraged', - as said in PEP-8.
There shouldn't be any problems with names, views.py has no special meaning in Django, it's just a convention.
You can keep your views in any file in any module under any name you want. So there is no special rules here, if you think that separating the module into submodules will be good, do it.

As DrTyrsa points out, views has no special meaning. So as an alternative to creating a subpackage, you could just create several files at the same level as the existing views.py - blog_views.py, posts_views.py, etc. As long as you use the correct reference in urls.py, this works fine.

Django admin site not displaying ManyToManyField relationship

I'm working on what I think is a pretty standard django site, but am having trouble getting my admin section to display the proper fields.
Here's my models.py:
class Tech(models.Model):
name = models.CharField(max_length = 30)
class Project(models.Model):
title = models.CharField(max_length = 50)
techs = models.ManyToManyField(Tech)
In other words, a Project can have different Tech objects and different tech objects can belong to different Projects (Project X was created with Python and Django, Project Y was C# and SQL Server)
However, the admin site doesn't display any UI for the Tech objects. Here's my admin.py:
class TechInline(admin.TabularInline):
model = Tech
extra = 5
class ProjectAdmin(admin.ModelAdmin):
fields = ['title']
inlines = []
list_display = ('title')
admin.site.register(Project, ProjectAdmin)
I've tried adding the TechInline class to the inlines list, but that causes a
<class 'home.projects.models.Tech'> has no ForeignKey to <class 'home.projects.models.Project'>
Error. Also tried adding techs to the fields list, but that gives a
no such table: projects_project_techs
Error. I verified, and there is no projects_project_techs table, but there is a projects_tech one. Did something perhaps get screwed up in my syncdb?
I am using Sqlite as my database if that helps.

I've tried adding the TechInline class to the inlines list, but that causes a
'TechInLine' not defined
Is that a straight copy-paste? It looks like you just made a typo -- try TechInline instead of TechInLine.
If your syncdb didn't create the proper table, you can do it manually. Execute this command:
python manage.py sqlreset <myapp>
And look for the definition for the projects_project_techs table. Copy and paste it into the client for your database.

Assuming your app is called "projects", the default name for your techs table will be projects_tech and the projects table will be projects_project.
The many-to-many table should be something like projects_project_techs

#John Millikin - Thanks for the sqlreset tip, that put me on the right path. The sqlreset generated code that showed me that the projects_project_techs was never actually created. I ended up just deleting my deb.db database and regenerating it. techs then showed up as it should.
And just as a sidenote, I had to do an admin.site.register(Tech) to be able to create new instances of the class from the Project page too.
I'll probably post another question to see if there is a better way to implement model changes (since I'm pretty sure that is what caused my problem) without wiping the database.