This is a bit of a strange question, I know, but bear with me. We've developed a RESTful platform using Python for one of our iPhone apps. The webapp version has been built using Django, which makes use of this API as well. We were thinking it would be a great idea to use Django's built-in control panel capabilities to help manage the data.
This itself isn't the issue. The problem is that everyone has decided it would be best of the admin center was essentially a client that sits on top of the RESTful platform.
So, my question is, is there a way to manipulate the model layer of Django to access our API directly, rather than communicated directly with the database? The model layer would act as the client passing requests and responses to and from the admin center.
I'm sure this is possible, but I'm not so sure as to where I would start. Any input?
I remember I once thought about doing such thing. At the time, I created a custom Manager using a custom QuerySet. And I overrode some methods such as _filter_or_exclude(), count(), exists(), select_related(), ... and added some properties. It took less than a week to become a total mess that had probably no chance to work one day. So I immediately stopped everything and found a more suitable solution.
If I had to do it once again, I would take a long time to consider alternatives. And if it really sounds like the best thing to do, I'd probably create a custom database backend. This backend would, rather than converting Django ORM queries to SQL queries, convert them to HTTP requests.
To do so, I think the best starting point would be to get familiar with django source code concerning database backends.
I also think there are some important things to consider before starting such development:
Is the API able to handle any Django ORM request? Put another way: Will any Django ORM query be translatable to an API request?
If not, may "untranslatable" queries be safely ignored? For instance, an ORDER BY clause might be safe to ignore. While a GROUP BY clause is very unlikely to be safely dismissed.
If some queries can't be neither translated nor ignored, may them be reasonably emulated. For instance, if your API does not support a COUNT() operation, you could emulate it by getting the whole data and count it in python with len(), but is this reasonable?
If they are still some queries that you won't be able to handle (which is more than likely): Are all "common" queries (in this case, all queries potentially used by Django Admin) covered and will it be possible to upgrade the API if an uncovered case is discovered lately or is introduced in a future version of Django?
According to the use case, there are probably tons of other considerations to take, such as:
the integrity of the data
support of transactions
the timing of a query which will be probably much higher than just querying a local (or even remote) database.
Related
I'm building a web app using Django and couchdb 2.0.
The new version of couchdb doesn't support temporary views. They recommend using a Mongo query but I couldn't find any useful documentation.
What is the best approach or library to use couchdb 2.0 with Django?
Temporary views were indeed abandoned in CouchDB 2.0. With mango, you could emulate them using a Hack, but that's just as bad (read: performance-wise). The recommendation is to actually use persistent views. As only the delta of new or updated documents need indexing, this will likely need significantly less resources.
As opposed to relational DBs, the created view (which is a persisted index by keys), is meant to be queried many times with different parameters (there is no such a thing as a query optimizer taking your temp view definition or something). So, when you're built heavily on temporary views, you might consider changing the way you query in the first place. One place to start is thinking about which attribute will collapse the result set most quickly to what you're looking for and build a view for that. Then, go query this view with keys and post-filter for the rest.
The closest thing you can do to a temporary view (when you really, really need it) is creating a design doc (e.g. _design/temp<uuid>) and use it for the one query execution.
Just to add a link (not new - but timeless) on the details: http://guide.couchdb.org/draft/views.html
I am quite new to Python and I have seen that both
Entries.objects.raw("SELECT * FROM register_entries")
and
Entries.objects.filter()
do the same query.
In which cases is better to use one or the other?
It depends on the database backend that you are using. The first assume that you have a SQL-based database engine. That is not always true. At the opposite, the second one will work in any case (if the backend is designed for). There was for instance few years ago a LDAP backend which was designed so, but LDAP queries do not use SQL language at all.
In all cases, I advice you to use the second one. It is the better way to go if you want to make reusable and long-term code.
There are also other ideas to prefer the second one to the first one
avoiding possible SQL injections ;
no need to know about database design (table's name, fields' name) ;
generic code is better than specific one ;
and moreover, it is shorter...
But you sometimes will have to use the first one when you do specific operations (calling specific backend's functions), but avoid them as much as possible.
In a nutshell, use the second one!!!
From django documentation:
When the model query APIs don’t go far enough, you can fall back to writing raw SQL
For all aspects, django queryset api offers you many options to customize your queries. But in some cases, you need to use very specific queries where django api become insufficient. But before you go for raw SQL, it is best to read all Queryset Api docs and learn everything about django queryset api.
At my work, we use Oracle for our database. Which works great. I am not the main db admin, but I do work with it. One thing I like is that the DB has a built in logic layer using PL/SQL which ca handle logic related to saving the data and retrieve it. I really like this because it allows our MVC application (PHP/Zend Framework) to be lighter, and makes it easier to tie in another platform into the data, such as desktop or mobile.
Although, I have a personal project where I want to use couchdb or mongodb, and I want to try and accomplish a similar goal. outside of the mvc/framework, I want to have an API layer that the main applications talk to. they dont actually talk directly to the database. They specify the design document (couchdb) or something similar for mongo, to get the results. And that API layer will validate the incoming data and make sure that data itself is saved and updated properly. Such as saving a new user, in the framework I only need to send a json obejct with the keys/values that need to be saved and the api layer saves the data in the proper places where needed.
This API would probably have a UI, but only for administrative purposes and to make my life easier. In general it will always reply with json strings, or pre-rendered/cached html in some cases. Since each api layer would be specific to the application anyways.
I was wondering if anyone has done anything like this, or had any tips on nethods I could accomplish this. I am currently looking to write my application in python, and the front end will likely be something like Angularjs. Although I am also looking at node.js for a back end.
We do this exact thing at my current job. We have MongoDB on the back end, a RESTful API on top of it and then PHP/Zend on the front end.
Most of our data is read only, so we import that data into MongoDB and then the RESTful API (in Java) just serves it up.
Some things to think about with this approach:
Write generic sorting/paging logic in your API. You'll need this for lists of data. The user can pass in things like http://yourapi.com/entity/1?pageSize=10&page=3.
Make sure to create appropriate indexes in Mongo to match what people will query on. Imagine you are storing users. Make an index in Mongo on the user id field, or just use the _id field that is already indexed in all your calls.
Make sure to include all relevant data in a given document. Mongo doesn't do joins like you're used to in Oracle. Just keep in mind modeling data is very different with a document database.
You seem to want to write a layer (the middle tier API) that is database agnostic. That's a good goal. Just be careful not to let Mongo specific terminology creep into your exposed API. Mongo has specific operators/concepts that you'll need to mask with more generic terms. For example, they have a $set operator. Don't expose that directly.
Finally after having a decent amount of experience with CouchDB and Mongo, I'd definitely go with Mongo.
I am currently in the planning stages of building a simple Django web app (for learning proposes). Basically, teacher’s can login, input student grades, and then the students can login and view their grades. Each class has a group number.
Here is a template of a user:
Username: Johny098
Password: Y98uj*?877!(
Name: John Doe
Gender: Male
Group: 32
Secondary: 5
Taking into consideration that I am starting with django and web developpement in general, I find confusing the number of database systems that are available to me: MySQL, CouchDB, MongoDB, SQLite, etc. And I am having a hard time deciding which database system to use for my purpose (I have no prior experience with databases).
After some research I found Couchdb (and SQLite) which seems fairly simple to pick up and fun to use, but that's just me, and that's why I need help. I know there are numerous debates on SQL vs NoSQL, but I don't really know if this will have an impact for my use of the databases. Ideally, the database system should integrate well with django and be easy enough to pick up in a couple of days.
So, coming back to the question: What database system should I use for my web app?
Any resources would also be appreciated.
Thanks,
-Raphael
For learning purposes I'd recommend SQLite: no setup, no background daemons running, everything is bundled with Python, it's SQL (so it maps well with Django ORM), and it's a very simple DBMS. In fact, some people use SQLite for prototyping and then switch to MySQL/PostgreSQL in production.
As for NoSQL, I would not recommend to use it unless you know exactly why and for what purpose you need it.
Oh, and one last thing: store password hashes (md5 or sha1), not the raw passwords. It's not necessary in your case, but in real-world apps it's mandatory.
The Django website has a page that lists the database backends supported. However, Django's database access layer is designed to work more or less the same no matter which database backend you're using. There are some differences described on the page I linked, but they shouldn't come up in normal usage if you're writing just a basic web app. So as far as effects on how you write your web app, among the choices listed it really doesn't matter.
Note that all the database backends Django supports are SQL-based, I believe. Accessing the database through Django does eliminate some of the security issues that I believe prompted the NoSQL movement... in any case, NoSQL is something you can pretty much ignore for now.
In your case, I would suggest picking SQLite, simply because it's easier to set up, and you don't want to spend time worrying about how to configure the database when you should be worrying about how to build your web app. The difference between SQLite and most other DBMSs (database management systems) is that SQLite stores each database in a regular file, and the SQLite client works directly with that file. Other DBMSs (like MySQL, PostgreSQL, Oracle, etc.) have a central location for the databases, a server to manage them, and a client that connects to the server and handles all the database access. A server-based DBMS works well for a busy web app, because it has features to handle many simultaneous requests to the database, but since you're just using this as a learning project, you don't need those features.
I would go for MySQL ... why? because among the ones you named, it is the most popular and, better than that, it will give you more credit when looking for a job (for instance, you can go to monster dot com and find how many jabs require you to know MySQL vs how many SQL-Lite).
Also, MySQL is simple and easy to learn, there are many GUI clients (some of which are Open Source) and it has pretty good documentation.
brand new on this forum and this is my first post!
At work we're starting a project which uses Apache Solr and i'm in charge of the frontend system (Django-based).
Our solr database isn't related to any other db engine nor to any models' class, so Haystack isn't good for us (since its strictly related to the models).
I was looking at http://code.google.com/p/pysolr/ and http://code.google.com/p/solrpy/
Basically, they're similar. I like more solrpy, since it uses POST requests and we can mask our users queries, but this makes its paginator harder to use (i guess..).
Other side, pysolr, thanks to the GET method, performs better (lower query timing), but so far i couldn't execute a query without getting a badrequest error.
Before choosing one, i wanted to ask the community any opinion. Users need to do only searches, our data is handled by a java process, no other db is used (except for storing user informations), and we need to use all solr features (faceting, highlight, word stopping, analyzers...).
What will you choose? And why? Any good code example you can point me at? I was looking throu the haystack source to see how they did implement all...
Thanks all!
We have used 'solrpy', but encountered some problems with it.
Sunburnt is actually an interesting API:
https://github.com/tow/sunburnt/
Actively developed, and easy to use. Unfortunately it introduces some additional dependencies.