I put the first steps in BlueBream framework. In my project I must get data from RDBMS - MySQL, PostgreSQL and MS Server. For now, I made a simple tutorial helloworld :) I know how to write Interfaces and Implementations, etc.
My question is: How to set up a connections to RDBMS and multiple connections? Could you give me a simple "step by step" tutorial ?
How to bake a database? Is in BlueBream something like a command "syncdb" in Django?
Take a look at zope.sqlalchemy; it integrates SQLAlchemy into the Zope transaction manager. SQLAlchemy in turn lets you access many different databases, including MySQL, PostgreSQL and MS Server.
The zope.sqlalchemy package explains how to obtain a SQLAlchemy session. From there on out you use bog-standard SQLAlchemy operations for which there are plenty of tutorials and help here on SO. SQLAlchemy can also take care of setting up the database schema for you, if that is needed.
Related
I was asked to port a Access database to MySQL and
provide a simple web frontend for the users.
The DB consists of 8-10 tables and stores data about
clients consulting (client, consultant,topic, hours, ...).
I need to provide a webinterface for our consultants to use,
where they insert all this information during a session into a predefined mask/form.
My initial thought was to port the Access-DB to MySQL, which I have done
and then use the web2py framework to build a user interface with login,
inserting data, browse/scroll through the cases and pulling reports.
web2py with usermanagment and a few samples views & controllers and
MySQL-DB is running. I added the DB to the DAL in web2py,
but now I noticed, that with web2py it is mandatory to define every table
again in web2py for it being able to communicate with the SQL-Server.
While struggeling to succesfully run the extract_mysql_models.py script
to export the structure of the already existing SQL DB for use in web2py
concerns about web2py are accumulating.
This double/redundant way of talking to my DB strikes me as odd and
web2py does not support python3.
Is web2py the correct way to fulfill my task or is there better way?
Thank you very much for listening/helping out.
This double/redundant way of talking to my DB strikes me as odd and web2py does not support python3.
Any abstraction you want to use to communicate with your database (whether it be the web2py DAL, the Django ORM, SQLAlchemy, etc.) will have to have some knowledge of the database schema in order to construct queries.
Even if you programmatically generated all the SQL statements yourself without use of an ORM/DAL, your code would still have to have some knowledge of the database structure (i.e., somewhere you have to specify names of tables and fields, etc.).
For existing databases, we aim to automate this process via introspection of the database schema, which is the purpose of the extract_mysql_models.py script. If that script isn't working, you should report an issue on Github and/or open a thread on the web2py Google Group.
Also, note that when creating a new database, web2py helps you avoid redundant specification of the schema by handling migrations (including table creation) for you -- so you specify the schema only in web2py, and the DAL will automatically create the tables in the database (of course, this is optional).
I'm aiming to apply the skills to hosting an app on Heroku with Python + SQLAlchemy + Xeround + Redis/Memcache.
What is the minimal software stack that I need? I'm looking at the following:
Python
A web application framework, like Flask
MySQL
MySQLdb <-- do I need this?
SQLAlchemy
It's obvious from the question that I do not know anything about SQL yet, that it would be preposterous to look at SQLAlchemy already. That's fine. I'm planning to learn the basics then immediately apply them using a Python API, if "immediately" is possible at all.
What I have accomplished so far
For an idea of where I stand:
Hosted an app on Google App Engine, using my own custom Models and Propertys for the datastore, memcache, task queue.
Hosted an app on Heroku, but I haven't used a database with it yet.
What I'm aiming for with this question
I want to know the software stack that I need to begin using MySQL. I just want to avoid installing stuff that I don't need.
For MySQL on Heroku, I suggest https://addons.heroku.com/cleardb
I am using Django 0.97 version with postgresql 9.0. I have configured hot streaming replication on a master db server and a slave db server. My application has heavy bot-driven writes on the DB and reads only from the users. So, it makes it very optimized if I make the read-access slave db for the users' and write-access master db for the bot write access. Unfortunately only Django 1.2 has multiple database support and its a huge effort to upgrade in my application. I got some leads through the following link : http://www.eflorenzano.com/blog/post/easy-multi-database-support-django/ However, this also requires me to change all instances of db access in my application. Is there any simpler way to assign separate db servers for read access and write access by fiddling with the the django core db module?
Your best bet is to upgrade to 1.2 as it will be significantly less work than hacking together features that already exist. If you stick with 0.97 for much longer your life will only be more difficult down the road.
I'm guessing you may have some misconceptions on how using multiple DBs works in Django 1.2. You do not have to "change all instances of db access in [your] application" if you use the Database Routers feature of Django.
With a router, you can specify which database to use for reads and writes. All of your existing django models should work and begin sending requests to the proper database. It's pretty simple to set up a router, just check the docs. All that is required is to create the router class, put it somewhere, then add a line in your settings.
It works really nicely and is not as much work as you may expect. You may have other issues with upgrading that you aren't telling us, but as far as models go you shouldn't have many problems.
There's an API for Twisted apps to talk to a database in a scalable way: twisted.enterprise.dbapi
The confusing thing is, which database to pick?
The database will have a Twisted app that is mostly making inserts and updates and relatively few selects, and then other strictly-read-only clients that are accessing the database directly making selects.
(The read-only users are not necessarily selecting the data that the Twisted app is inserting; its not as though the database is being used as a message-queue)
My understanding - which I'd like corrected/adviced - is that:
Postgres is a great DB, but almost all the Python bindings - and there is a confusing maze of them - are abandonware
There is psycopg2 for postgres, but that makes a lot of noise about doing its own connection-pooling and things; does this co-exist gracefully/usefully/transparently with the Twisted async database connection pooling and such?
SQLLite is a great database for little things but if used in a multi-user way it does whole-database locking, so performance would suck in the usage pattern I envisage; it also has different mechanisms for typing column values?
MySQL - after the Oracle takeover, who'd want to adopt it now or adopt a fork?
Is there anything else out there?
Scalability
twisted.enterprise.adbapi isn't necessarily an interface for talking to databases in a scalable way. Scalability is a problem you get to solve separately. The only thing twisted.enterprise.adbapi really claims to do is let you use DB-API 2.0 modules without the blocking that normally implies.
Postgres
Yes. This is the correct answer. I don't think all of the Python bindings are abandonware - psycopg2, for example, seems to be actively maintained. In fact, they just added some new bindings for async access which Twisted might eventually offer an interface.
SQLite3 is pretty cool too. You might want to make it possible to use either Postgres or SQLite3 in your app; your unit tests will definitely be happier running against SQLite3, for example, even if you want to deploy against Postgres.
Other?
It's hard to know if another database entirely (something non-relational, perhaps) would fit your application better than Postgres. That depends a lot on the specific data you're going to be storing and the queries you need to run against it. If there are interesting relationships in your database, Postgres does seem like a pretty good answer. If all your queries look like "SELECT foo, bar FROM baz" though, there might be a simpler, higher performance option.
There is the txpostgres library which is a drop in replacement for twisted.enterprise.dbapi, —instead of a thread pool and blocking DB IO, it is fully asynchronous, leveraging the built in async capabilities of psycopg2.
We are using it in production in a big corporation and it's been serving us very well so far. Also, it's actively developed—a bug we reported recently was solved very quickly.
http://pypi.python.org/pypi/txpostgres
https://github.com/wulczer/txpostgres
You could look at nosql databases like mongodb or couchdb with twisted.
Scaling out could be rather easier with nosql based databases than with mysql or postgres.
Is there any python library that can keep a client-side SQLite database in sync with a server-side PostgreSQL database?
There are solutions for Java, such as Daffodil or SymmetricDS. Is there something similar for python?
SymmetricDS is a server-side solution for synchronization that gets triggered regardless of which language is being used to access the database. You should still be able to use that to synchronize the databases, while using Python libraries to actually query them. I would recommend sqlalchemy as a good database-independent query layer for Python.