I am working on a project which requires me to create a table of every user who registers on the website using the username of that user. The columns in the table are same for every user.
While researching I found this Django dynamic model fields. I am not sure how to use django-mutant to accomplish this. Also, is there any way I could do this without using any external apps?
PS : The backend that I am using is Mysql
An interesting question, which might be of wider interest.
Creating one table per user is a maintenance nightmare. You should instead define a single table to hold all users' data, and then use the database's capabilities to retrieve only those rows pertaining to the user of interest (after checking permissions if necessary, since it is not a good idea to give any user unrestricted access to another user's data without specific permissions having been set).
Adopting your proposed solution requires that you construct SQL statements containing the relevant user's table name. Successive queries to the database will mostly be different, and this will slow the work down because every SQL statement has to be “prepared” (the syntax has to be checked, the names of table and columns has to be verified, the requesting user's permission to access the named resources has to be authorized, and so on).
By using a single table (model) the same queries can be used repeatedly, with parameters used to vary specific data values (in this case the name of the user whose data is being sought). Your database work will move along faster, you will only need a single model to describe all users' data, and database management will not be a nightmare.
A further advantage is that Django (which you appear to be using) has an extensive user-based permission model, and can easily be used to authenticate user login (once you know how). These advantages are so compelling I hope you will recant from your heresy and decide you can get away with a single table (and, if you planning to use standard Django logins, a relationship with the User model that comes as a central part of any Django project).
Please feel free to ask more questions as you proceed. It seems you are new to database work, and so I have tried to present an appropriate level of detail. There are many pitfalls such as this if you cannot access knowledgable advice. People on SO will help you.
This page shows how to create a model and install table to database on the fly. So, you could use type('table_with_username', (models.Model,), attrs) to create a model and use django.core.management to install it to the database.
Related
Just a heads up, I am new to using web frameworks. My only experience so far comes from completing the VSCode, Django and Mozilla MDN tutorials. But I'm making my way through these along with my own project.
I'm creating a web app that has an "official" database table that all website/app users can view. But, I want to be able to let them add their own data entries to the table, which only they can view and edit. And would be able to grant other users/friends access to their created data entries, to expand the total number of entries made available. Without making everyone using the site having to work out which data entries are the "official" and which are user created.
Hopefully a better way of understanding what I'm planning
So, what would be the best method for setting up user accounts to have access to the main database table and their own data set, which they can grant access for others to view?
Would this mean creating a table for each user, and if so how can this be set up automatically upon account creation?
I've read that creating a new table in the database can be cumbersome later on if lots of accounts with their own tables of data are created.
I've looked through the Django documentation, but it seems to be more focussed on user account creation and authorisation. And regarding databases, I can't find any questions/posts that relate to what I'm trying to make. Especially with creating a personal list of data entries for each user upon account creation.
Thank you, for taking the time to read this, even if you don't have an answer!
So we have been using Apache Superset, It's a great tool.
The only frustration come from that there are a few dashboards we want to share with users outside the company.
I believe right now the way to do it is go from the Gamma user then create a read only role (Correct me if I'm wrong)
There are a few downside of this:
we need to create a view per user on each table to make sure that they do not see the records that they are not supposed to.
the access is given by datasource, so they will be able to see any dashboard that use the same datasource, which can be a problem sometimes.
all of these authentication is a lot of work to maintain.
I'm wondering if there is any way (or even hack) to simply share the graphs and tables as a dashboard, without any database access granted.
Like a frozen or snapshot of dashboard,
like the way Redash does it:
https://redash.io/help/user-guide/dashboards/sharing-dashboards
What you are looking for can be achieved through a combination of the public user and appending ?standalone=true to the dashboard url.
You also don't need the entire list of Gamma user permissions, the most important ones are can explore on superset, explore json on superset and datasource access and csrf token. This basically renders the dashboards without the superset menu and should make everything readonly.
We can achieve this by creating a custom role.
1. Can remove all the menu items
2. Can disable the dashboard edit button
3. Can give access to specific tables.
So a user cannot access any other dashboard or charts
Eg. Dashboard
Public dashboards
This is not meant for production. It’s for experiments or while doing a proof of concept.
#superset_config.py
PUBLIC_ROLE_LIKE_GAMMA = True
or
PUBLIC_ROLE_LIKE: Optional[str] = "Gamma"
After this, we need to re-run the init user (if already run)
docker-compose exec superset superset-init
Dashboards & charts can be embedded without superset header (Nav bar etc) by adding standalone=true parameter to the url, like this :
http://localhost:9000/superset/dashboard/world_health/?standalone=true
We need to grant database source permissions to public role for the data to be visible.
please see: https://sairamkrish.medium.com/apache-superset-custom-authentication-and-integrate-with-other-micro-services-8217956273c1
Superset is great, I'm glad people are talking about it since the days when it was AirBnB's Caravel. It has come a long way.
There is no 'official' solution for what you're looking for but there is a way to effectively get the same result. You said you wouldn't mind a 'hack' so...
Creating a table or a data source and exposing it to the 'public' group should do what you're looking to accomplish.
Can you take form data and change database schema? Is it a good idea? Is there a downside to many migrations from a 'default' database?
I want users to be able to add / remove tables, columns, and rows. Making schema changes requires migrations, so adding in that functionality would require writing a view that takes form data and inserts it into a function that then uses Flask-Migrate.
If I manage to build this, don't migrations build the required separate scripts and everything that goes along with that each time something is added or removed? Is that practical for something like this, where 10 or 20 new tables might be added to the starting database?
If I allow users to add columns to a table, it will have to modify the table's class. Is that possible, or a safe idea? If not, I'd appreciate it if someone could help me out, and at least get me pointed in the right direction.
In a typical web application, the deployed database does not change its schema at runtime. The schema is only changed during an upgrade, and only the developers make these changes. Operations that users perform on the application can add, remove or modify rows, but not modify the tables or columns themselves.
If you need to offer your users a way to add flexible data structures, then you should design your database schema in a way that this is possible. For example, if you wanted your users to add custom key/value pairs, you could have a table with columns user_id, key_name and value. You may also want to investigate if a schema-less database fits your needs better.
Need a way to improve performance on my website's SQL based Activity Feed. We are using Django on Heroku.
Right now we are using actstream, which is a Django App that implements an activity feed using Generic Foreign Keys in the Django ORM. Basically, every action has generic foreign keys to its actor and to any objects that it might be acting on, like this:
Action:
(Clay - actor) wrote a (comment - action object) on (Andrew's review of Starbucks - target)
As we've scaled, its become way too slow, which is understandable because it relies on big, expensive SQL joins.
I see at least two options:
Put a Redis layer on top of the SQL database and get activity feeds from there.
Try to circumvent the Django ORM and do all the queries in raw SQL, which I understand can improve performance.
Any one have thoughts on either of these two, or other ideas, I'd love to hear them.
You might want to look at Materialized Views. Since you're on Heroku, and that uses PostgreSQL generally, you could look at Materialized View Support for PostgreSQL. It is not as mature as for other database servers, but as far as I understand, it can be made to work. To work with the Django ORM, you would probably have to create a new "entity" (not familiar with Django here so modify as needed) for the feed, and then do queries over it as if it was a table. Manual management of the view is a consideration, so look into it carefully before you commit to it.
Hope this helps!
You said redis? Everything is better with redis.
Caching is one of the best ideas in software development, no mather if you use Materialized Views you should also consider trying to cache those, believe me your users will notice the difference.
Went with an approach that sort of combined the two suggestions.
We created a master list of every action in the database, which included all the information we needed about the actions, and stuck it in Redis. Given an action ID, we can now do a Redis look up on it and get a dictionary object that is ready to be returned to the front end.
We also created action id lists that correspond to all the different types of activity streams that are available to a user. So given a user id, we have his friends' activity, his own activity, favorite places activity, etc, available for look up. (These I guess correspond somewhat to materialized views, although they are in Redis, not in PSQL.)
So we get a user's feed as a list of action ids. Then we get the details of those actions by look ups on the ids in the master action list. Then we return the feed to the front end.
Thanks for the suggestions, guys.
I'm starting a Django project and need to shard multiple tables that are likely to all be of too many rows. I've looked through threads here and elsewhere, and followed the Django multi-db documentation, but am still not sure how that all stitches together. My models have relationships that would be broken by sharding, so it seems like the options are to either drop the foreign keys of forgo sharding the respective models.
For argument's sake, consider the classic Authot, Publisher and Book scenario, but throw in book copies and users that can own them. Say books and users had to be sharded. How would you approach that? A user may own a copy of a book that's not in the same database.
In general, what are the best practices you have used for routing and the sharding itself? Did you use Django database routers, manually selected a database inside commands based on your sharding logic, or overridden some parts of the ORM to achive that?
I'm using PostgreSQL on Ubuntu, if it matters.
Many thanks.
In the past I've done something similar using Postgresql Table Partitioning, however this merely splits a table up in the same DB. This is helpful in reducing table search time. This is also nice because you don't need to modify your django code much. (Make sure you perform queries with the fields you're using for constraints).
But it's not sharding.
If you haven't seen it yet, you should check out Sharding Postgres with Instagram.
I agree with #DanielRoseman. Also, how many is too many rows. If you are careful with indexing, you can handle a lot of rows with no performance problems. Keep your indexed values small (ints). I've got tables in excess of 400 million rows that produce sub-second responses even when joining with other many million row tables.
It might make more sense to break user up into multiple tables so that the user object has a core of commonly used things and then the "profile" info lives elsewhere (std Django setup). Copies would be a small table referencing books which has the bulk of the data. Considering how much ram you can put into a DB server these days, sharding before you have too seems wrong.