We Keep Coding

Django and Node – how does it work? [closed]

Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 7 years ago.
Improve this question
In order to make an web application with RESTful capabilities I have read and watched tons of articles and videos and I still do not get a complete picture of how it works. And which I should chose. Every other answer is the not helpful ”it depends”. I have boiled it down to a first choice between Django and Node. But nowhere I find the whole ”picture” of how the pieces works together and which modules are needed. Therefore I have tried to put all into a rough illustration. Note that I am a complete newbie on this.
I develop an ERP application with accounting modules. Basically it is mainly about CRUD besides viewing diagrams, printing and storing documents. So this is the ”it depends”
The only thing I have managed to make decisions about is to use nginx, Postgresql and Debian 8 as tools/os. These are the fixed stars.
My questions are not really the common Django vs Node.js and it is not just an opinion I want:
Is the picture below correct? Any comments?
Is there any further components that will be needed? To get started?

You have a lot of questions - and on StackOverflow there should be one question that can be answered without generating a lot of debate or have opinions rather than facts.
As such, I think your question might be closed as "too broad"; however I think it deserves an answer.
I am not going to say "it depends", although that's really all it boils down to - but here is my attempt to explain it.
nodejs is a runtime. It is an environment which allows you to develop code on the server using javascript. In order to do anything useful with nodejs, beyond "hello world"; you'll need to use a framework, and there are tons of those around and various stacks have been developed by the community to tie in all the components together. An example of such a stack is MEAN, which is MongoDB for the database, Express for the framework, Angular to assist with the front-end, and Node to run it all.
django is a framework - it is not a runtime. This means that it is one step removed from the node world. The runtime for django is Python. django also is not a "stack" like MEAN, you can develop your own stack on top of it - but since django is a "batteries included" framework, you only really need to add a database to it - it includes everything else you need.
REST is just a way of designing web-services. Its not a language, or a platform or a library. Its a set of rules that describe a way to design APIs such that they take advantage of the semantic verbs of HTTP.
You can use any library and programming language to develop a RESTful service. All you really need is two things [a] a library to communicate over HTTP [b] a way to serialize data, preferably in JSON (but even that's not a requirement).
nginx is just a very fast webserver and a reverse proxy. The reason it is mentioned often - is because it is very expensive for a framework to serve static media. All requests to a framework (either in django world, or in nodejs world) have to go through a large chain of components that help decode the HTTP request and create a data structure that is easy for developers to use. This chain of components is often called middleware. Since each and every request has to go through this middleware, it is better for performance reasons that requests that don't need the "power" of the application to execute (like a request for an image, a stylesheet, a video file) be handled by something else. This is what nginx is used for, since its a very fast webserver.
Now that those are explained, you need to see what stack works best for your application. To do that, you need to know a bit about the philosophy/justification or problem that each stack is trying to solve.
For django - this is easy. Django was created by a team working on multiple newspapers to help them manage content that was published on different sites. As such, it is designed so that the management of content is of primary concern. That is why it has a very robust administration console as a standard component; and a built-in quite robust ORM and its own templating engine. Django leaves it up to you to figure out how best to actually run and deploy it; although they do provide a lot of suggestions and examples - but in the end, its upto you to decide which database to use, which web server to use, and how to deploy the application.
In the nodejs world - the primary focus is nonblocking I/O and speed of response. Nodejs excels at being able to serve a lot of simultaneous requests on limited resources. Therefore, it provides you a very powerful foundation to develop applications that need to quickly respond to requests ... and that's it. When you program in node or any other specialized lower-level library, you need to make sure your code is taking complete advantage of the library. So, if you start writing blocking code in node, you'll find that the performance that you expect hasn't been achieved.
nodejs doesn't care what the application actually does. Think of it like a very fast, very powerful tool. You can build anything with it, but you need to know what the tool is designed to do best in order to know when to use it.
nodejs has you working at a lower level - which is why there are a lot of packages that help you do all sorts of things with node; and multiple ways you can take components and create your own stack - depending on what you are building on top of node. Think of it like Lego building blocks.
nodejs and django are not mutually exclusive. You can utilize both in your application and exploit their strengths and take advantage of what each does best.
As far as your specific questions:
Did I get the picture? Any comments?
I don't know. Did you?
Is there any further components that will be needed? To get started?
The answer to this is yes, because you don't want to build everything from scratch. Each stack has its own libraries components for developing services. For django, there is django rest framework (DRF).
Which framework are best for CRUD?
Which framework are best for RESTful? Any other module needed?
Best report generator for printing?
Best diagram tools?
There is nothing that is "best" for anything. This question is just asking for opinions. Its like asking, what is the best fruit juice?
Which framework are fastest and most reliable for CRUD using Postgresql
People have developed many robust applications on top of postgresql; however as nodejs is bound to javascript - there is still a lot of work being done in this area.
Can I lock the library (trade secrets) in both environments?
Yes.
Is there better tools for creating ERP/Accounting?
ERP and accounting are two very different things. There are tons of accounting packages/applications written in Python. There are very few ERP systems written in Python.
You cannot combine the two and lump it together.
What is the benefits using Angular on top of Node.js?
The same benefit of using Angular on top of _____ (insert your favorite backend). Angular is just a front end library.
An anecdotal benefit is that both Angular and Node use the same programming language.

Converting and Unifying API data in Python

I'm trying to pull similar data in from several third party APIs, all of which have slightly varying schemas, and convert them all into a unified schema to store in a DB and expose through a unified API. This is actually a re-write of a system that already does this, minus storing in the DB, but which is hard to test and not very elegant. I figured I'd turn to the community for some wisdom.
Here are some thoughts/what I'd like to achieve.
An easy way to specify schema mappings from the external APIs schema to the internal schema. I realize that some nuances in the data might be lost by converting to a unified schema, but that's life. This schema mapping might not be easy to do and perhaps overkill from the academic papers I've found on the matter.
An alternative solution would be to allow third parties to develop the interfaces to the external APIs. The code quality of these third parties may or may not be known, but could be established via thorough tests.
Therefore the system should be easy to test, I'm thinking by mocking the external API calls to have reproducible data and ensure that parsing and conversion is being done correctly.
One of the external API interfaces crashing should not bring down the rest of them.
Some sort of schema validation/way to detect if the external API schemas have changed without warning
This will end up being integrated into a Django project, so it could be written as a Django app, which would likely make unit and integration testing easier. On the other hand, I would like to keep it as decoupled as possible from Django. Although the API interfaces would have to know what format to convert to, could this be specified at runtime?
Am I missing anything in the wishlist? Unrealistic? Headed down the wrong path? Would love to get some feedback.
I'm not sure if there are libraries/OS project which already do some of this. The less wheels I have to reinvent the better. Would any part of this be valuable as an OS project?
In the previous version I spawned a bunch of threads that would handle individual requests. Although I've never used it, I've been told I should look at gevent as a way to handle this.

For your second bullet point you should check out Temboo. Temboo normalizes access to over 100 APIs, meaning that you can talk to them all using a common syntax in the language of your choice. In this case you would use the Temboo Python SDK - available here.
(Full disclosure: I work at Temboo)

need a python based solution for web based display of tabular data

I need to make a tabular data structure (tab delimited text file) available for viewing as a web based solution. I am a bioinformatics programmer with almost no experience in web based development. I know that django is very hot in the python community but I wanted to ask here before I went ahead and buy a book on django. What would be your choice of technology stack to accomplish something like this.
I need to display a table of 40-50 columns and 100.000 rows and hopefully let the user filter the data based on certain data items ( i.e only show rows that have a certain value in a certain column , show only data that was recorded on Monday and hide all other weekdays)
I am sorry if this question is too vague or stupid but I really need some basic guidance here.
Thanks

Django can do this fairly easily.

Django can do this, but I think the best way to go is to use a Javascript framework ontop of django, I am currently doing this. ExtJS has various types of grids in your situation I think a 'Live' grid would be perfect.
It loads x amount of rows, so that you dont have to load 100,000 rows everytime, just what the user sees. Also, filters etc are built in as well as many other features
Other javascript frameworks that do similar things are YUI and in my opinion JQuery to a lesser extent
Edit/Elaborate
So obviously here isnt the place for a beginners crash course, but in my opinion there is a couple of things you need to do and know.
This will work by firstly creating a django view that returns a JSON string. (If that sentence didnt make a whole bunch of sense, I would recommend skimming over the Django tutorial...actually, you probably should do that anyway) Python has methods to turn datatypes such as dictionaries/csv's (in your case, I guess a TSV lol) to this format. THEN, when you have this (can be pointed to by a url...when you dive into Django it will make more sense) then you create the ExtJS grid and point it to that url.
There is a whole bunch of tutorials about ExtJS grids here, notably the Tutorial:Grid PHP SQL I think would be helpful. Obviously not php, but the concept is the same.
Unfortunately I dont have any examples of my own to show you, but there are TONS of resources about this stuff, I wouldnt bother buying a book

I think this could be done easily without JavaScript. What neolaser is outlining is my preferred solution as well, but django could do this no sweat. You would need
to configure your models.py to match your database
a view that accepts get requests and makes queries based off of their contents. http://docs.djangoproject.com/en/dev/ref/models/querysets/
a template that displays the results of those queries and allows you make get requests which your view will interpret.
Because django is such a well-used framework, it's pretty easy to find a run down on the various terms (google: "django views", "django models", etc).

If what you describe is really all you're doing, then I'd say Django may be overkill. Maybe first try a simpler basic framework like Cherrypy (see tutorial) to serve your simple page/form (you don't even need templates, just spit back HTML yourself). Now all you need is a bit of code to read, filter and/or page, and format your CSV.

If you want to put something like this together very quickly and easily and you don't have much web development experience, I think your best bet would be web2py. It requires no installation or configuration, has no dependencies, and includes a web server, a relational database, a web-based integrated development environment and admin interface (demo), and jQuery integration (for Javascript and Ajax). It's very easy to learn and was designed for ease of use and developer productivity. You can get a lot done with very little code thanks to the included scaffolding app along with many sensible default behaviors.
As for table/grid displays, you could probably use:
The jqGrid plugin - a web2py plugin for jqGrid
powerTable (source code) - a web2py plugin for DataTables
If you need help getting started or have any questions, you'll get lots of help from the very friendly and responsive mailing list.

Is django-piston mature enough?

I'm developing an advertising site and want to use web services for the requests. I mean, a publisher site will put a JavaScript snippet and it will pull a banner through a REST GET.
Is the django-piston framework mature enough to implement this functionality?

I've been looking into finding the "best" Django REST package and came across this table, which is useful:
http://www.djangopackages.com/grids/g/api/
At this point (mid-2011) Django-Tastypie is the clear winner for number of authors, updated codebase, documentation, and overall activity.
EDIT, Jan.2012: I think the two leading contenders are now Django-Tastypie, and also, Django-REST-framework. The latter has a really useful feature of web-browsable APIs, which I've yet to see in other packages.

I am currently using it and it's good enough for my needs, which are fairly simple (mostly just an easy way to set up a read-only API for model data). I do have a couple of criticisms:
It doesn't seem that jespern, the code's creator, is doing much in response to issues posted to bitbucket. Possibly an unfair criticism, and activity in the django-piston Google Group. Still it's frustrating to post an issue and encounter zero response whatsoever.
It appears there are some inconsistencies or errors in the coding. For example, this issue that I posted. (I may pursue it further in the group).

I can't answer "is it mature enough." We are still evaluating our options ourselves.
However, you might want to check out these:
http://github.com/toastdriven/django-tastypie -- perhaps an up and coming competitor to piston.
http://ericholscher.com/blog/2009/nov/11/large-problems-django-mostly-solved-rest-api/

Django is certainly mature enough. It's powering more than a dozen high-profile sites in production. Piston on the other hand, I wouldn't know, except for bitbucket.org. I don't know of any other "high-profile" sites that use it. It doesn't sound like you need a complex framework on top of Django just to handle REST. You could easily implement a RESTful service on top of Django without using any other specific library.

Python/GAE social network / cms?

After much research, I've come up with a list of what I think might be the best way of putting together a Python based social network/cms, but have some questions about how some of these components fit together.
Before I ask about the particular components, here are some of the key features of the site to be built:
a modern almost desktop-like gui
future ability to host an advanced html5 sub-application (ex.http://www.lucidchart.com)
high scalability both for functionality and user load
user ability to password protect and permission manage content on per item/group basis
typical social network features
ability to build a scaled down mobile version in the future
Here's the list of tools I'm considering using:
Google App Engine
Python
Django
Pinax
Pyjamas
wxPython
And the questions:
Google App Engine -- this is an attempt to cut to the chase as many pieces of the puzzle seem to be in place.
Question: Am I limiting my options with this choice? Example: datastore not being relational? Should I wait
for SQL support under the Business version?
Python -- I considered 'drupal' at first, but in the end decided that being dependent on modules that may or
may not exist tomorrow + limitations of its templating system are a no-no. Learning its API, too, would be useless elsewhere
whereas Python seems like a swiss army knife of languages -- good for almost anything.
Question: v.2.5.2 is required by GAE, but python.org recommends 2.5.5. Which do I install?
Django -- v.0.96 is built into GAE. You seem to be able to upgrade it.
Questions: Any reason not to upgrade to the latest version? Ways to get around the lack of HTML5 support?
Pinax (http://pinaxproject.com) Rides on top of Django and appears to provide most of the social network functionality
anyone would want.
Question: Reasons NOT to use it? Alternatives?
Pyjamas and wxPython -- this is the part that gets a little confusing. The basic idea behind these is the ability
to build a GUI. I've considered Silverlight and Flash, before the GAE/Python route, but a few working versions of
HTML5 apps convinced me that enough of it ALREADY runs on the latest batch of browsers to chose the HTML5/Javascript
route instead.
Question: How do I extend/supplement Python/Django to build an app-like HTML5 interface? Are Pyjamas and wxPython
the way to go? Or should I change my thinking completely?
Answers to some/any of these questions would be of great help. Please excuse my ignorance if any of this doesn't make much sense.
My last venture into web programming was a decent sized LAMP website some 5-6 years ago. On the desktop side of things,
my programming experience boils down to very high level scripting languages that I keep on learning to accomplish very specific
tasks :)

As someone who has deployed a Django site to GAE, I can tell you that you are not going to reach the ideal solution. Django on GAE misses some of the best aspects of Django because the ORM doesn't work right. The best compromise may be to use Django-nonrel to add the features back in.
This introduces it's own problems though: because of the large number of files and memory used by a Django app you're code will be unloaded from memory quickly after the app becomes idle. That means that visitors will frequently hit an approximately 6 second delay on the first page view after the site's code has been unloaded from memory while GAE uncompresses the zipped modules. Once your site is busy this won't be a problem, but while your site is still young and unknown it will cause the appearance of performance problems. :-(
Second, I've also worked for a company that built a custom CMS and can tell you that the first 80% is pretty easy, especially with modern frameworks. However, the rest can be quite challenging. For example, user roles and custom content types are two challenging aspects. Therefore strongly consider standing on the backs of giants and finding a CMS or CMS framework that almost perfectly meets your needs and then extend it to do that extra bit you need.
So, that said, answering your points:
Yes, you're limiting your options but that may be OK. Most developers are more comfortable with the relational model than the nosql model. Therefore more open source software is built with it in mind. Also, GAE is a closed source platform which is also a deterrent to open source developers. App Engine Oil is a CMS framework that may suit you well and is optimized for App Engine. Also look at web2py which has support for GAE.
I've found myself to be extremely productive with Python. I used to write a lot of PHP now I find it ugly. That said, think about the total line count of code you'll have to write. If you can make Drupal work with high quality pre-made modules you may find yourself only needing 1/10th of the code. By the way, the trick with Drupal is to mainly use only high quality modules. Look at the history, make sure not to use development versions. Try to contact the authors on IRC. I'm not saying you should use Drupal but it is possible to have a reliable site with it (for example, whitehouse.gov)
You're in the classic GAE/Django problem. If you use 0.96 you get great performance but you miss a lot of the great 1.0+ features and you don't get the ORM and all of it's benefits. If you use a newer version of Django you get the performance/memory problems mentioned above.
I'm about to investigate pinax for my company. I've done a very cursor glance at it. I don't know if it has good support for non relational model backends. You'll probably need to look at django-nonrel. However know that you're going to be investing in relatively untried solutions here. A small percentage of Django users use Pinax and an even smaller percentage, if any, use it on a nonrelational backend. Therefore you're going to be in the highly experimental scenario you mentioned in point 2 above.
I can't offer personal experience on it. I've investigated pyjamas a few times. However I like writing HTML CSS and JS. I like to have control. I like progressive enhancement and knowing what users will see if they don't have the full capabilities. Also, I think any new app that doesn't explicitly address mobile clients is implicitly shooting themself in the foot. As many as 15% of Internet users only use the Internet via their smart phone. What kind of experience will they get with pyjamas?
You didn't mention this, but one thing I consider when choosing a platform is vendor lockin and portability. If you develop your solution for GAE and find that you're not able to do what you want, will you be able to port it to another solution elsewhere? How much work will it take? If you code heavily for GAE or make commitments to its architecture, you're stuck with it or with rewriting to move. Using Django or Web2py can help mitigate this.
That said, the big benefit of Python GAE is that you get to be very productive, see your results instantly, get hosting for free while your site is small and get excellent scalability. These are not small things. There is great value there.