I have two apps that both access the same database. The first has clients connecting via TCP and writes to the db using SQLAlchemy. The second is a more typical webapp using Django. Both have read/write requirements.
I would like to unify the database access layer, but picking just SQLAlchemy or just Django is unattractive because:
I would like to use django auth, permissions, and maybe third party plugins, which require the Django ORM (correct me if I'm wrong).
For the first app, using SQLAlchemy (so far) is much simpler than trying to use the Django ORM outside of a Django app - it is a TCP/IP server app, not a HTTP/web app.
Are there any problems with mixing these two ORMs on the same database?
In which system (Django, SQLA) should I create the models, vs using some kind of introspection such as Django inspectdb?
Firstly - it's not very hard to use Django ORM outside a manage.py, WSGI handlers and other HTTP related stuff. You can use it any python script, but it needs some initialization (example).
Secondly - SQLA is a very powerfull tool and it's able to do stuff which is very hard to achive in Django ORM (like genuine polymorphism and polymorphic queries). If I had to choose, I'd personally choose to use Django ORM as a platform to create models, then manually map them in SQLA since it is much more flexibile and hopefully will be able to adopt. Which may not work in the opposite case.
Finally, since you can use Django ORM on both sides, and you just have to use a Django ORM because of the plugins, I suggest to abandon the SQLA. It's a powerfull tool, but also rather complicated. Having two different ORMs operating on one database may result in unexpected problems in the future and increases the complexity of your app, so it'll be harder to maintenance.
Related
I am setting a web server based on Django Framework with PostgreSQL. I need to update the models in my Django app every hour and I am wondering if is it better way to update the database from another script file using python and psycopg2 or using script that directly connects to Django models and update them using Django API for models? I use this Database only in Django web server.
Summarizing... What is better logic?
update_db.py that contains:
connect to database via psycopg2, update the tables
update_db.py that contains:
connect to Django app models, update them using Django API for models
I will run script every hour using Cron, and the Django Project is already connected to that specific database and it works fine. I am just asking for better performance and better way in logic sense. Data won't be updated very often but it would have big jsons.
Unless you have very compelling reasons to do otherwise, the obvious solution here is to use a custom management command and the ORM - your code will be at the obvious place (for someone else having to work on the project), can be tested as part of your whole project's test suite, and won't require having to mentally "translate" from raw SQL to Django ORM code (not to mention that you'll have more chance to catch a mismatch between your Django models and your script's code when your schema changes - from experience, raw sql scripts that aren't under test are usually forgotten when doing a schema migration).
FWIW, optimizing "for performances" doesn't really makes sense in your case, unless your updates are long and complex and do block access to the site - but even then the overhead of the ORM is certainly not going to be the main bottleneck (just make sure you use properly).
I noticed that most examples for accessing mysql from flask suggest using a plugin that calls init_app(app).
I was just wondering why that is as opposed to just using a mysql connector somewhere in your code as you need it?
Is it that flask does better resource management with request life cycles?
Using packages like flask-mysql or Flask-SQLAlchemy, they provided useful defaults and extra helpers that make it easier to accomplish common CRUD tasks.
All of such package are good at handling relationships between objects. You only need to create the objects and then the objects contain all the functions and helpers you needed to deal with the database, you don't have to implement such code by yourself and you don't need to worry about the performance of the queries.
I had worked on a Django project（I believe the theory in Flask is similar） and its ORM is really amazing, all i need to do is writing Models and encapsulate business logic. All CRUD commands are handled by the built-in ORM, as a developer we don't worry about the SQL statements.
Another benefit is that it makes database migration much easier. You can switch it from MySQL to PostgresSQL with minimal code modifications which will speed up development.
I have my existing Django web application that uses a MySQLDB without memcaching.
I would like to implement memcaching to improve the responsiveness of this site. I see the instructions here.
However, these instructions leave me with some unanswered question(s). Is this all I need to do to get memcache working after I setup the memcached server? Or do I need to alter any of my code outside of settings.py? Does Django nicely handle all the memcaching operations behind the scenes for me whenever models are read or written? (If so, that's very cool!) How can I see what improvement the memcaching is having on the number of DB accesses?
What you've done is just a set up of a Cache Backend.
In order to benefit from caching you need to find the places where it is appropriate and would have a positive impact on performance: your views, templates..you can cache the whole views, templates, template fragments etc.
If you want some automation to help you, take a look at Johnny Cache package:
Johnny Cache is a caching framework for django applications. It works
with the django caching abstraction, but was developed specifically
with the use of memcached in mind. Its main feature is a patch on
Django’s ORM that automatically caches all reads in a consistent
manner.
Or django-cache-machine package:
Cache Machine provides automatic caching and invalidation for Django
models through the ORM.
There is also an interesting project called django-cacheops that is aiming to improve Django ORM caching, but it uses Redis backend.
Also, django_debug_toolbars caching panel can help you in the future.
Note that django querysets have a built-in internal cache, but it has nothing to do with a cache framework.
Further reading:
Using Django querysets effectively
Caching and QuerySets
Basically, I am trying to share as much database layer code as possible between a Flask application (for a REST API) and a non-Flask API.
Is it a good idea to use the same Flask-SQLAlchemy layer in both the pure Python API (intended to be imported in non-web Python applications) and in the REST API Flask daemon?
I guess another way to phrase, though I am not sure on the terminology, "how do I best share database model between Flask app and a separate Python import library?"
Or from another angle, is there any point in using Flask-SQLAlchemy in a Flask REST API, if you also want to share the SQL abstraction with an import library. Is it better then to use plain SQLAlchemy?
Use case: we have a large database with many tables, and want to build both a REST API (for customer access) and a Python import library (for performant internal tools) for accessing the database, but of course share as much code between them as possible.
Related:
Using Flask-SQLAlchemy in Blueprint models without reference to the app
What's your folder layout for a Flask app divided in modules?
SQLAlchemy with Flask -- hybrid models?
Using Flask-SQLAlchemy models out of a web context is a matter of creating a Flask application and call
app.test_request_context().push()
The point here is what you will do with your "non web" library.
If it's not a problem to have the whole Flask library installed when you need to use the library then there's no problem at all using it in that way.
If you plan to make performance improvements in the library data accessing code, like using different sessions, concurrency and such, then you're modifying your initial code so it's a totally different scenario. In this case a pure-SQLAlchemy approach might be better but it really depends on the differences between the two patterns.
Normally with models comes methods and using 2 different ORM patterns (Flask-SQLAlchemy wrapper models and pure SQLAlchemy) means duplicating code.
This may seem like a subjective question. But it is not (that's not the idea, at least).
I'm developing an Advertising software (like AdWords, AdBrite, etc) and i've decide to use Python. And would like to use one of those well known web frameworks (Django, Cherrypy, pylons, etc).
The question is:
Given that it will have just a few Models (seven or eight), which has the best cache support? and What is the most efficient retrieving data from a MySQL database?
Thanks!
check out Flask. Its easy, its fast, works on top of Werkzeug, uses Jinja2 templating and SQLAlchemy for the model domain. http://flask.pocoo.org/
Performance should be more or less equal. If you want to keep it simple look at cherrypy, pylons and other lightweight frameworks.
-> http://wiki.python.org/moin/WebFrameworks gives a nice overview.
If you want to use Python to do complex SQL queries on your database, e.g. eagerloading or filtering on the fly you might be wanting SQLAlchemy.
TurboGears 2 is a framework which comes with SQLAlchemy as standard, check out their caching page for more info on the second part of your answer.
CherryPy is the only framework I'm aware of that does real HTTP caching out of the box (but look at "beaker" for a WSGI component solution). Many of the others give you tools to store arbitrary objects in Memcached or other storage.
I am only familiar with Django, and can tell you that it has a very robust middleware handler and very straightforward cache management. Also, the ORM (object-relational mapper; connect objects to databases) can have Postgre or MySQL as the engine, so you are free to choose the fastest one (I think that other frameworks use SQLAlchemy's ORM, which is also super cool and fast)
Check:
Middleware
Cache