I have a mssql schema with the django ORM / pymssql extension. I have some classes build via the inspectdb function. A lot of the Primarykeys in the tables are UUID fields / mssql uniqueidentifier, which the ORM inspected as CharFields with length 36.
I am concerned now with possible duplicates for the primary keys since the tables are growing very fast.
The tables have a default constraint for any new primary key on the database site. So basically I have two (different) sources of UUID generation (the database server and the application server)
How is it possible to insert via the ORM from django performantly?
Am I save with generating the UUIDs via pythons uuid module or do I have to ask the database everytime for a new UUID before creating a object with django?
The primary key cannot be duplicated, so it will raise a "duplicated pk duplicated" exception. In addition, the odds of getting a duplicated uuid is quite close to 0, you will get 3 lottery prizes before you get a duplicated uuid.
Related
I'm currently developing a Python Flask app that will allow users to write out paragraphs and store them in a MySQL database. Are there any Python libraries that will let users have the benefits of version control? Ideally users would be able to track edits so that users can revert to previous versions of the text they've written.
If you're using SQL Alchemy, checkout: sqlalchemy-continuum.
Features:
Does not store updates which don’t change anything
Supports alembic migrations
Can revert objects data as well as all object relations at given transaction even if the object was deleted
Transactions can be queried afterwards using SQLAlchemy query syntax
Querying for changed records at given transaction
Querying for versions of entity that modified given property
Querying for transactions, at which entities of a given class changed
History models give access to parent objects relations at any given point in time
Or check versioning-objects in SQLAlchemy documentation.
I also found this tutorial Database Content Versioning Using SQLAlchemy by Googling: sqlalchemy history table, you may find other solutions.
For a specific security reason, a client has asked if we can integrate a 'write-only' DB into a Django web application. I have tried creating a DB then restricting access to one of its tables in psql via:
REVOKE SELECT ON TABLE testapp_testmodel FROM writeonlyuser;
But then trying to save a model in the Django shell...
p = new TestModel(test_field="testvalue")
p.save(using="writeonlydb")
...generates this error:
ProgrammingError: permission denied for relation testapp_testmodel
Which I assume is because the ORM generated SQL includes a return of the newly created object's id, which counts as a read:
INSERT INTO "testapp_testmodel" ("test_field") VALUES ('testvalue') RETURNING "testapp_testmodel"."id"
My question is therefore, is this basically impossible? Or is there perhaps some other way?
I'm designing a bulk import tool from an old system into a new Django based system.
I'd like to retain all of the current IDs of objects (they are just 5 digit strings), now due to the design in the current system there are lots of references between these objects.
To import I can see 2 possible techniques - import a known object, and carefully recurse through these relationships making sure to import in the right way and only set relationships as soon as I know they exist
... or start at item 00001 set the foreignkeys to values I know exist and just grab everything in order, knowing that once we get to item 99999 all the relationships will exist.
So is there a way to set a foreignkey to the ID of an item that doesn't exist, but will, even for imports only?
To add further complexity, not all of these relationships are straightforward foreignkeys, some are ManyToMany relationships as well.
To be able to handle any database that Django supports and not have to deal with peculiarities of the backend, I'd export the old database in the format that Django loaddata can read, and then give this exported file to loaddata. This command has no issue importing the type of structure you are talking about.
Creating the file that loaddata will read could be done by writing your own converter that reads the old database and dumps an appropriate file. However, a way which might be easier would be to create a throwaway Django project with models that have the same structure as the tables in the old database, point the Django project to the old database, and use dumpdata to create the file. If table details between the old database and the new database have changed, you'd still have to modify the file but at least some of the conversion work would have already been done.
A more direct way would be to bypass Django completely to do the import in SQL but turn off foreign key constraints for the time of the import. For MySQL this would be done by setting foreign_key_checks to 0 for the time of the import, and then back to 1 when done. For SQLite this would be done by using PRAGMA foreign_keys = OFF; and then ON when done.
Postgresql does not allow just turning off these constraints but Django creates foreign key constraints as DEFERRABLE INITIALLY DEFERRED, which means that the constraint is not checked until the end of a transaction. So initiating a transaction, importing and then committing should work. If something prevents this, then you have to drop the constraint before importing and add it back afterwards.
Sounds like you need a database migration tool like South, the standard for Django. Worth noting that Django 1.7 Beta 1 was released recently and it provides in-built migration.
I have a postgreSQL database that has a table foo that I've created outside of django. I used manage.py inspectdb to build the model for table foo for me. This technique worked fine when I was using MySQL but with PostgreSQL it is failing MISERABLY. The table is multiple gigabytes and I build it from a text file with PostgreSQL 'COPY'.
I can run raw queries on table foo and everything executes and expected.
For example
foo.objects.raw('bar_sql')
executes as expected.
But running queries like:
foo.objects.get(bar=bar)
throw
ProgrammingError column foo.id does not exist LINE 1: SELECT "foo"."id", "foo"."bar1", "all_...
foo doesn't innately have an id field. As I understand it django is suppose to create one. Have I some how subverted this step when creating the tables outside of django?
Queries run on models whose table was populated threw django run as expected in all cases.
I'm missing something very basic here and any help would be appreciated.
I'm using django 1.6 with postgreSQL 9.3.
Django doesn't modify your existing database tables. It only creates new tables. If you have existing tables, it usually doesn't touch them at all.
"As I understand it django is suppose to create one." --> It only adds a primary key to a table when it creates it, which means you don't need to specify that explicitly in your model, but it won't do anything to an existing table.
So if for example you later on decide to add fields to your models, you have to update your databases manually.
What you need to do in your case is that by doing manual database administration make sure that your table has a primary key, and also that the name of the primary key is "id" (although I am not sure if this is necessary, it is better to do it.) So use a database administration tool, modify your table and add the primary key, and name it id. Then it should start working.
I have Django application that using MongoDB with Mongoengine. I need to create several sqlite databases. Every SQLite database associated with one of all documents of some MondoDB collection (for example MainDocument). SQLite duplicate some collections documents that depend on MainDocument with one-to-many relation. So... Number of documents on MainDocument is 1074! I need to create 1074 sqlite databases with different names.
I fill one default sqlite database with django orm then copy it with other name, then truncate all tables and start again for another document. It takes a long time. How to do it faster?
It's possible with django orm without configuring 1074 database routers?
I've tried use ramfs, but it still so slowly.