We Keep Coding

Unit tests with an unmanaged external read-only database

I'm working on a project which involves a huge external dataset (~490Gb) loaded in an external database (MS SQL through django-pyodbc-azure). I've generated the Django models marked managed=False in their meta. In my application this works fine, but I can't seem to figure out how to run my unit tests. I can think of two approaches: mocking the data in a test database, and giving the unit tests (and CI) read-only access to the production dataset. Both options are acceptable, but I can't figure out either of them:
Option 1: Mocked data
Because my models are marked managed=False, there are no migrations, and as a result, the test runner fails to create the database.
Option 2: Live data
django-pyodbc-azure will attempt to create a test database, which fails because it has a read-only connection. Also I suspect that even if it were allowed to do so, the resulting database would be missing the required tables.
Q How can I run my unittests? Installing additional packages, or reconfiguring the database is acceptable. My setup uses django 1.9 with postgresql for the main DB.

After a day of staring at my screen, I found a solution:
I removed the managed=True from the models, and generated migrations. To prevent actual migrations against the production database, I used my database router to prevent the migrations. (return False in allow_migrate when for the appropriate app and database).
In my settings I detect whether unittests are being run, and then just don't define the database router or the external database. With the migrations present, the unit tests.

Using web2py for a user frontend crud

I was asked to port a Access database to MySQL and
provide a simple web frontend for the users.
The DB consists of 8-10 tables and stores data about
clients consulting (client, consultant,topic, hours, ...).
I need to provide a webinterface for our consultants to use,
where they insert all this information during a session into a predefined mask/form.
My initial thought was to port the Access-DB to MySQL, which I have done
and then use the web2py framework to build a user interface with login,
inserting data, browse/scroll through the cases and pulling reports.
web2py with usermanagment and a few samples views & controllers and
MySQL-DB is running. I added the DB to the DAL in web2py,
but now I noticed, that with web2py it is mandatory to define every table
again in web2py for it being able to communicate with the SQL-Server.
While struggeling to succesfully run the extract_mysql_models.py script
to export the structure of the already existing SQL DB for use in web2py
concerns about web2py are accumulating.
This double/redundant way of talking to my DB strikes me as odd and
web2py does not support python3.
Is web2py the correct way to fulfill my task or is there better way?
Thank you very much for listening/helping out.

This double/redundant way of talking to my DB strikes me as odd and web2py does not support python3.
Any abstraction you want to use to communicate with your database (whether it be the web2py DAL, the Django ORM, SQLAlchemy, etc.) will have to have some knowledge of the database schema in order to construct queries.
Even if you programmatically generated all the SQL statements yourself without use of an ORM/DAL, your code would still have to have some knowledge of the database structure (i.e., somewhere you have to specify names of tables and fields, etc.).
For existing databases, we aim to automate this process via introspection of the database schema, which is the purpose of the extract_mysql_models.py script. If that script isn't working, you should report an issue on Github and/or open a thread on the web2py Google Group.
Also, note that when creating a new database, web2py helps you avoid redundant specification of the schema by handling migrations (including table creation) for you -- so you specify the schema only in web2py, and the DAL will automatically create the tables in the database (of course, this is optional).

Restricted database user for Django

I'm running Django with Postgres database. On top of application-level security checks, I'm considering adding database-level restrictions. E.g. the application code should only be able to INSERT into log tables, and not UPDATE or DELETE from them.
I would manually create database user with appropriate grants for this. I would also need a more powerful user for running database migrations.
My question is, do people practice things like this? Any advice, best practices on using restricted database users with Django?
Edit: To clarify, there's no technical problem, I'm just interested to hear other people's experiences and takeaways. One Django-specific thing is, I'll need at least two DB users: for normal operation and for running migrations. Where do I store credentials for the more privileged user? Maybe make manage.py migrate prompt for password?
As for the reasoning, suppose my app has a SQL injection vulnerability. With privileged user, the attacker can do things like drop all tables. With a more limited user there's slightly less damage potential and afterwards there's some evidence in insert-only log tables.

The way I would try to solve this problem is:
Setup two database connection for the same database but different users:
'default' for unprivileged db user.
'master' for user with full access to the db.
Configure routers to your liking: https://docs.djangoproject.com/en/dev/topics/db/multi-db/
Perform migrations: ./manage.py migrate --database=master
Now that being said, if you have a vulnerability, like a SQL injection, even with a user that can read, it is already a disaster. If you have db backup setup, you shouldn't worry about lost/corrupted data, you should be worrying about data leakage!
On the other hand, there might be other use cases for the proposed problem.

For storing the credentials to the privileged user for management commands, when running manage.py you can use the --settings flag, which you would point to another settings file that has the other database credentials.
Example migrate command using the new settings file:
python manage.py migrate --settings=myapp.privileged_settings

Yes, this is practiced sometimes, but not commonly. The best way to do it is to grant specific privileges on user, not in django. Making such restrictions means that we should not trust application, because it might change some files / data in db in the way that we do not expect it to do so.
So, to sum up: create another user able to create / modify data and user another one with restrictions to use normally.
It's also quite common in companies to create one user to insert data and another one for employees / scripts to access it.

django database routing with transactions

Referring to the example in Django documentation for multiple databases in one application,
https://docs.djangoproject.com/en/dev/topics/db/multi-db/#an-example
" It also doesn’t consider the interaction of transactions with the database utilization strategy. "
How do I handle the interaction stated above.
The scenario is this:
I am using postgresql as my database. I have setup up a replica and want all the reads to "auth" tables to go to replica. Following the documentation I wrote a database router. Now whenever I try to log in to my application is throws the following error.
DatabaseError: cannot execute UPDATE in a read-only transaction.
This happens when Django tries to save the "last_login" time. Because, in the same view it first fetches the record from replica, and then tries to update the last_login time. Since it happens in one transaction so the same database is used, i.e. replica.
How do I handle this?
Thoughts?

Does Django's south (migration tool) work for innodb?

$ py manage.py migrate turkey
Running migrations for turkey:
- Migrating forwards to 0001_initial.
> turkey:0001_initial
! Error found during real run of migration! Aborting.
! Since you have a database that does not support running
! schema-altering statements in transactions, we have had
! to leave it in an interim state between migrations.
! You *might* be able to recover with: = DROP TABLE `turkey_demorecs` CASCADE; []
! The South developers regret this has happened, and would
! like to gently persuade you to consider a slightly
! easier-to-deal-with DBMS.
! NOTE: The error which caused the migration to fail is further up.
For some reason I get this when I try it.
But my other setups are in MyISAM.
Why doesn't it work in Innodb?

InnoDB has constraints on Foreign Keys which ensure you are not breaking the database model when doing a migration. (see http://dev.mysql.com/doc/refman/5.5/en/innodb-foreign-key-constraints.html)
MyISAM does not have native support for constraints (although it seems you can implement this if you choose to do do http://dev.mysql.com/tech-resources/articles/mysql-enforcing-foreign-keys.html)
Because MyISAM is not checking your FK relationships, you do not get the error. InnoDB however is doing a check and it seems that you have a problem with your migration.

See also https://code.djangoproject.com/wiki/AlterModelOnSyncDB
I have had the same kind of error happen to me when working with a mysql setup whose default table storage engine is MyISAM and I wanted to use InnoDB (using the recipe found in above link, we used the post_syncdb signal to trigger the conversion code). However, when using South to create new tables they were first created using MyISAM engine then later converted. I was mistakenly believing InnoDB tables weren't doing what they were supposed to, when those were actually MyISAM; because the table were converted by the signal, any migration error would fail to unapply :-/
If you need to use or create InnoDB tables where the default is MyISAM, this be solved with:
# add at the beginning of your migration
if db.backend_name == 'mysql':
db.execute('SET storage_engine=INNODB')
or if you do not mind the performance hit:
# add this to settings.py
DATABASE_OPTIONS = {
"init_command": "SET storage_engine=INNODB", # XXX: performance hit...
}

Yes, South does support InnoDB. Can you delete the contents of your "migrations" folder, and re-run schemamigration, migrate, and post the results and contents of the 0001_initial file here? PS: Make sure you have your migrations folder backed up or in source control first.
rm -fr app/migrations/*
./manage.py schemamigration app --initial
./manage.py migrate app

You could try adding to your first migration:
if db.backend_name == 'mysql':
db.execute('SET foreign_key_checks=0')
This will disable the foreign key check constraints.
You don't have to set it back to 1 since it's a session variable.
By the way, it doesn't work if you set to 0 at the beggining and back to 1 at the end of your migration method, because south generates SQL with them, but executes it when they return.