Thinking through an idea, wanted to get feedback/suggestions:
Having had great success with url rewriting and nginx, I'm now thinking of a more capable reverse proxy/router that would do the following:
Map requests to handlers based on regex matching (ala Django)
Certain requests would simply be routed to backend servers - eg. static media, memcached, etc
Other requests would render templates that pull in data from several backend servers
For example, a template could consist of:
<body>
<div>{% remote http://someserver/somepage %}</div>
<div>{% remote http://otherserver/otherpage %}</div>
</body>
The reverse proxy would make the http requests to someserver/somepage and otherserver/otherpage and pull the results into the template.
Questions:
Does the idea make sense or is it a bad idea?
Is there an existing package that implements something like this?
How about an existing server+scripting for implementing this - eg. lighttpd+lua, nginx+??
How about nginx+SSI? Looks pretty capable, if you have experience / recommendations please comment.
How about something like a scripting language+eventlet ?
Twisted?
My preferences are python for scripting and jinja/django style templates, but I'm open to alternatives.
This already exist an is called Deliverance: http://deliverance.openplans.org/
So instead of doing something an AJAXy call into an iframe or something, you're doing it on the server side.
I think it's something I'd only do if the external site was totally under my control, purely for the security implications. It'd also hit your response times quite a bit.
Am I missing the point completely, or would this be quite simple to do with some functions & urllib?
Related
What is the best way to implement isomorphic react app using webpack with python as backend to serve
Following features are required to be implemented
1. React router
2. Redux store
By Isomorphic, I understand you mean both rendered on the server & on the client - so the first access of the site will return static HTML and a javascript bundle which then replaces the static HTML with the react-rendered version.
Usually this would mean running javascript on the server too with Node.js. This means you can render your JSX / React templates / components on the server and send them as static HTML to the user. If you're working with Python on the server, you cannot use your JSX / Javascript templating & logic on the server, so you'll need to duplicate it in Python.
So since you have to do it all twice - you first need to work out which one you want to do first. Either in Python first, or Javascript.
You can either start by building the whole application server-side-rendered, and then have javascript take over, or build a fully javascript application, and render the first views with Python.
Personally, for content-driven sites, I prefer building a server-side rendered application with Django (the most used Python Web Framework), and then adding Javascript on top once it's all fully working as a javascriptless site. This has the advantage of working even when javascript is disabled, and ensures you have good URLs, etc.
Of course, if it's a really complex application in terms of user interaction, you'll need to do it the other way.
I recomend looking into Django first, Here is a great tutorial from django-girls. However, if you want to go the mainly-JS route, here's some ideas:
Javascript First
Probably the best way is to first design your data structure(s), and make a REST api for your data. (e.g. /api/1.0/cars/rustbucket_94 which sends JSON data.)
Next, figure out your user-facing URL schema, and work out which REST endpoints need to be pulled in to get those pages. (How the URLs should look to the end users. e.g. /cars/rustbucket_94.html)
Design your React app as normal, using those REST / JSON endpoints, and the react router to show it correctly.
Once you have your whole react app working, you can now build a server-side rendered version of the whole thing, which you'll need to re-implement from scratch, and can either access the data through the JSON endpoints (slow) or by making the queries itself inside the pages.
The Python side
Which Python Framework to use on the backend?
I'd recommend Django to start with. It's very capable and can do ANYTHING you want it too. You probably need the Django REST framework too. There are other options available, but this will be the quickest way to get something secure and sane running.
Getting Django to work nicely with your webpack / react workflow is not too complex. There are projects like Django-Webpack-Loader and various tutorials about it online showing how to use it.
Good luck.
I'm writing a set of REST services for a Django project. I've been using django-rest-framework for a while. Because of its limited functionality I had to switch to django-piston which I quite enjoy.
However, django-rest-framework had one really nice feature - it was able to display an admin-like interface for testing the created services from the browser. It's just terrific for debugging purposes. It's very simple: one form is displayed for each HTTP method like "GET", "POST", etc. Along with that a drop-down list of available content types and a text field for putting in the data to be sent.
As I view it, this isn't really a feature in any way directly connected with a particular REST framework. It isn't even necessarily about Django. It could all be achieved just using HTML + JS, or an external website.
My question is: What do you use for manual testing / debugging web services? Could you point me to some HTML snippet or a Django app that would do the described thing?
This may seem obvious, but:
Why not just use Django's testing client (django.test.client.Client)? then instead of manually 'debugging' in your browser, you can write unit tests with expectations and get leverage out of those further down the track.
e.g.
from django.test.client import Client
client = Client()
resp = client.put('/employee/2/', data={'email': 'here#there.com'}, follow=True)
#... etc
As the author of django-rest-framework it'd be great to pick your brains about which bits of functionality could do with fleshing out. :) (obv i've got some thoughts of my own and areas I'm planning to work on, but be really good to get some user perspective)
Your absolutely right about the API browser not being limited to any particular framework. To me that's the big deal with DRF and I'd love to see more API frameworks take a similar approach. One of the supposed benefits of RESTful APIs is that they should be self-describing, and it seems counter-intuitive to me that so many of the Web APIs we build today are not Web browseable.
Oh, and totally agree with jsw re. testing Web APIs in django, I wouldn't use the framework's browsable API to replace automated tests.
I had the same problem and that was eaily solved by logging out of admin page in that project.
I'm developing an app (an API) in python and I would like to offer some of its functionality through a web interface (like web services do).
I've been looking at django, but I don't know if really fits well in my idea. I only want to create a web page that invokes to my API methods in order to acomplish the functionality that offers that web page. But, after followed the tutorial, I feel a little confused about the point of django. It seems to me to be more related with an ORM than a classic web application.
Is django a solution so heavy for such a simple development (as I mentioned, make calls to my API through the web)? Do I always have to use a database?
Thanks.
I love django but there is an lot of it to get your head around! If you don't want the database bit, focus on urls.py and views.py that will process your urls and return the info you want as an http response.
eg. urls.py
urlpatterns += patterns('myapp.views',
url(r'^getstuff/$', 'getstuff' ),
)
in views.py
def getstuff(request):
do whatever in python
return HttpResponse(stuff to return)
You don't need to use database in Django projects.
Basically django comes with some standardized architecture that follows MVC pattern (or MVT as sometimes described). This includes models, views, url dispatching, templates, etc.
Probably you need to do following things to accomplish your task:
create url definition in urls.py
to some django view
write
django view that call somehow your
api and displays result as a web
page
you don't need models and database at all but you need to get familliar with views, urls, templates. It might look like a big machinery for your simple case but if you have some time I encourage you to these django basics.
If you are looking for somethin much simpler I heard about webpy project. This might be better option if you need something really simple.
An important question is: Do you want the web services to be provided by a full-featured server like Apache, or are you just looking at the "web server" to be a thread (or equivalent) in your program?
If you want to run Apache, then I'd recommend something like Werkzeug, which will handle most of the WSGI stuff for you. For templating, I've heard good things about Jinja2.
If that is too much, and all you want is a lightweight, simple server (something that, say, just spits out some HTML or XML when asked, and doesn't need any fancy URL handling), you can use the SimpleHTTPServer or CGIHTTPServer modules that ship with Python.
Django is a full-featured, integrated package that provides almost everything you need to write database-backed web applications. While its various components can be used in isolation, if you're only using one thing (the template and view engines, in your case), it is probably overkill.
No need for a framework at all. Raw wsgi isn't hard but a little verbose. So I like to use WebOb
Here's Raw wsgi
def application(environ, start_response):
start_response("200 OK", [])
return ["<html><body><h1>Hello World</h1></body></html>"]
Here's the webob version
from webob.dec import wsgify
from webob import Request
#wsgify
def application(request):
return Response("<html><body><h1>Hello World</h1></body></html>")
That's enough to run under apache mod_wsgi, and there are plenty of libraries that you can use that expect/produce webob Request and Responses. Anything that Turbogears 2 or repoze.bfg uses is fair game at that point.
You definitely don't have to use a database with Django. Whether it fits your needs, only you can tell. There are other Python web frameworks that you can use.
For some quick background, I'm an XHTML/CSS guy with some basic PHP knowledge. I'm trying to dip my feet into the Python pool, and so far understand how to start simple_server and access a simple Hello World return in the same .py file. This is the extent of what I understand though, heh.
How do I integrate the simple_server and your basic XHTML/CSS files? I want to start the server and automagically call, for instance, index.py (does it need to be .py?). Obviously within the index file I would have my markup and stylesheet and I would operate it like a normal site at that point.
My eventual goal is to get a basic message board going (post, edit, delete, user sessions). I realize I'll need access to a database, and I know my way around MySQL enough to not have to worry about those portions.
Thanks for the help.
EDIT: Allow me to clarify my goal, as I have been told Python does a LOT more than PHP. My goal is to begin building simple web applications into my pre-existing static XHTML pages. Obviously with PHP, you simply make sure its installed on your server and you start writing the code. I'd like to know how different Python is in that sense, and what I have to do to, say, write a basic message board in Python.
The other answers give good recommendations for what you probably want to do towards your "eventual goal", but, if you first want to persist with wsgiref.simple_server for an instructive while, you can do that too. WSGI is the crucial "glue" between web servers (not just the simple one in wsgiref of course -- real ones, too, such as Apache or Nginx [both with respective modules called mod_wsgi] as well as, for example, Google App Engine -- that one offers WSGI, too, as its fundamental API) and web applications (and frameworks that make it easier to write such applications).
Everybody's recommending various frameworks to you, but understanding WSGI can't hurt (since it will underlie whatever framework you eventually choose). And for the purpose of such understanding wsgiref.simple_server will serve you for a while longer, if you wish.
Essentially, what you want to do is write a WSGI app -- a function or class that takes two parameters (an "enviroment" dictionary, and a "start response" callable that it must call back with status and headers before returning the response's body). Your "WSGI app" can open your index.py or whatever else it wants to prep the status, headers and body it returns.
There's much more to WSGI (the middleware concept is particularly powerful), though of course you don't have to understand it very deeply -- only as deeply as you care to! See wsgi.org for tutorials &c. Gardner's two-part article, I think, is especially interesting.
Once (and if that's your choice) you understand WSGI, you can better decide whether you want it all hidden in a higher level framework such as Django (so you can focus on application-level issues instead) or use a very light and modular toolbox of WSGI utilities such as Werkzeug -- or anything in-between!-)
I would recommend Django.
"Obviously with PHP, you simply make sure its installed on your server and you start writing the code."
Not true with Python. Python is just a language, not an Apache plug-in like PHP.
Generally, you can use something like mod_wsgi to create a Python plug-in for Apache. What you find is that web page processing involves a lot of steps, none of which are part of the Python language.
You must use either extension libraries or a framework to process web requests in Python. [At this point, some PHP folks ask why Python is so popular. And the reason is because you have choices of which library or framework to use.]
PHP parses the request and allows you to embed code in the resulting page.
Python frameworks -- generally -- do not work this way. Most Python frameworks break the operation down into several steps.
Parsing the URL and locating an appropriate piece of code.
Running the code to get a result data objects.
Interpolating the resulting data objects into HTML templates.
"My goal is to begin building simple web applications into my pre-existing static XHTML pages."
Let's look at how you'd do this in Django.
Create a Django project.
Create a Django app.
Transform your XTHML pages into Django templates. Pull out the dynamic content and put in {{ somevariable }} markers. Depending on what the dynamic content is, this can be simple or rather complex.
Define URL to View function mappings in your urls.py file.
Define view functions in your views.py file. These view functions create the dynamic content that goes in the template, and which template to render.
At that point, you should be able to start the server, start a browser, pick a URL and see your template rendered.
"write a basic message board in Python."
Let's look at how you'd do this in Django.
Create a Django project.
Create a Django app.
Define your data model in models.py
Write unit tests in tests.py. Test your model's methods to be sure they all work properly.
Play with the built-in admin pages.
Create Django templates.
Define URL to View function mappings in your urls.py file.
Define view functions in your views.py file. These view functions create the dynamic content that goes in the template, and which template to render.
Take a look at CherryPy. It's a nice http framework.
It depends on what you want to achieve,
a) do you want to just write a web application without worrying too much abt what goes in the background, how request are being handled, or templates being rendered than go for a goo webframework, there are many choices simple http server is NOT one of them. e.g. use django, turbogears, webpy, cheerpy, pylons etc etc
see http://wiki.python.org/moin/WebFrameworks for full list
b) if you want to develope a simple web framework from start so that you understand internals and improve you knowledge of python, then I will suggest use simple http server
see
how can you create a URL scheme so that URLs are dispatched to correct python function,
see how can you render a html
template e.g. containing place
holder variables $title etc which
you can convert to string using
string.Template
b) would be difficult but interesting exercise to do, a) will get you started and you may be writing web apps in couple of days
As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.
Closed 10 years ago.
I've got a Django application that works nicely. I'm adding REST services. I'm looking for some additional input on my REST strategy.
Here are some examples of things I'm wringing my hands over.
Right now, I'm using the Django-REST API with a pile of patches.
I'm thinking of falling back to simply writing view functions in Django that return JSON results.
I can also see filtering the REST requests in Apache and routing them to a separate, non-Django server instance.
Please nominate one approach per answer so we can vote them up or down.
I'm thinking of falling back to simply
writing view functions in Django that
return JSON results.
Explicit
Portable to other frameworks
Doesn't require patching Django
Please note that REST does not just mean JSON results. REST essentially means exposing a resource-oriented API over native but full-fledged HTTP. I am not an expert on REST, but here are a few of the things Rails is doing.
URLs should be good, simple names for resources
Use the right HTTP methods
HEAD, GET, POST, PUT, and DELETE
Optionally with an override (form parameter '_method' will override HTTP request-method)
Support content-type negotiation via Accept request-header
Optionally with an override (filename extension in the URL will override MIME-type in the Accept request-header)
Available content types should include XML, XHTML, HTML, JSON, YAML, and many others as appropriate
For example, to get the native HTTP support going, the server should respond to
GET /account/profile HTTP/1.1
Host: example.com
Accept: application/json
as it would respond to
GET /account/profile.json HTTP/1.1
Host: example.com
And it should respond to
PUT /account/profile HTTP/1.1
Host: example.com
var=value
as it would respond to
POST /account/profile HTTP/1.1
Host: example.com
_method=PUT&var=value
For anyone else looking for a very decent, pluggable API application for Django, make sure you checkout jespern's django-piston which is used internally at BitBucket.
It's well maintained, has a great following and some cool forks which do things like add support for pagination and other authentication methods (OAuth is supported out of the box).
Updated to reflect that django-piston is no longer maintained.
Tastypie is also a newly emerging REST framework for Django.
It has the same mindset as pistons, and removes a lot of boilerplate coding.
My answer to the same question here: Framework for Implementing REST web service in Django
The short version is, have a look at https://github.com/jgorset/django-respite/ a REST framework in its early days, but we use it every day on client projects.
Scrap the Django REST api and come up with your own open source project that others can contribute to. I would be willing to contribute. I have some code that is based on the forms api to do REST.
I'm thinking of falling back to simply
writing view functions in Django that
return JSON results.
I would go with that ..
Ali A summed it pretty well.
The main point for me is beign explicit. I would avoid using a function that automatically converts an object into json, what if the object has a reference to a user and somehow the password (even if it's hashed) go into the json snippit?
I ended up going with my own REST API framework for Django (that I'd love to get rid of if I can find a workable alternative), with a few custom views thrown in for corner cases I didn't want to deal with. It's worked out ok.
So a combination of 1 and 2; without some form of framework you'll end up writing the same boilerplate for the common cases.
I've also done a few stand-alone APIs. I like having them as stand-alone services, but the very fact that they stand alone from the rest of the code leads to them getting neglected. No technical reason; simply out-of-sight, out-of-mind.
What I'd really like to see is an approach that unifies Django forms and REST APIs, as they often share a lot of logic. Conceptually if your app exposes something in HTML it likely wants to expose it programmatically as well.
You could take look at django-dynamicresponse, which is a lightweight framework for adding REST API with JSON to your Django applications.
It requires minimal changes to add API support to existing Django apps, and makes it straight-forward to build-in API from the start in new projects.
Basically, it includes middleware support for parsing JSON into request.POST, in addition to serializing the returned context to JSON or rendering a template/redirecting conditionally based on the request type.
you could try making a generic functions that process the data (like parand mentioned) which you can call from the views that generate the web pages, as well as those that generate the json/xml/whatever
TastyPie looks quite interesting and promising. It goes well with Django.