I have a question regarding the Sage Cell Server. It is a relatively general question, I don't have any code that I need fixed.
I am trying to set up a server but I am finding the world of servers confusing. I've worked plenty with Java and JavaScript, HTML and CSS on frontend stuff, but for the backend Sage server I decided to develop I figured Python was a good choice. Some basic Lynda tutorials and a book later, I have a pretty good idea how python works... But am still lost. My question is basically this:
-I am trying to design a very simple server running an instance of Sage. I want to be able to SEND Sage code to it (NOT PYTHON) and have it evaluated and the result sent back, as if I were running sage on my own computer. Example: Sending "integrate(x,x)" to the server would return 1/2*x^2
-What is the best way to go about doing this? Should I download the Sage Cell Server?
-Will I need to even write Python code for this? How long would this task take in hours?
-Would it be easier for me to design the server to receive and execute Python code instead of Sage code? How would I go about doing this?
If you are wondering why I want to do this, I want to design a website where people can type in their math problems and have them evaluated by sage. I know there are simpler ways to do this but I've wanted to do server-side work and get people talking with a server for a while now and this seems like a good introduction to it.
Any and all advice is appreciated, resources, help, etc.
There are certainly a number of people who have successfully set up their own Sage cell server, and as such things go I hear it is definitely doable. The previous "simple server" API is pretty much defunct and the Sage cell is a pretty reasonable way to do this. I don't see why you would need to have reinvent the wheel here, and asking on sage-support#googlegroups.com is a logical next step for you.
I am usually working in fields of machine learning and hence my background is mostly in stats/ML and no formal web background.
Usually for my project, I work on python which is connected to my local mysql db... to fetch data adn everything.
Now, my work is mostly complete.. everything is console based..
(like traditional programs).
How do I integrate it on the front end. I understand that this is more like a server side scripting.
So, lets take an example of google.
In the front end.. someone enters a search query.. and in the backend lets say there is a program in C++ which executes that query.
How did this interaction takes place.. if front end is written in lets say php..
I assume shell execution of program is a bad bad way to run programs.. ??
Any suggestion will be greatly appreciated.
Thanks
As suggested by Ignacio, you will first need to design API for your project. This is basically clearly laying out what (and how) queries will be supported for your project.
You don't need shell execution and neither need to learn PHP. Since your project is in Python, you can use Python Web-frameworks like Django, Web2Py.
The first thing to do is to develop an API for your library. From there you can develop multiple frontends that use the same API in order to present it multiple ways.
Decouple your user interface logic from your business logic, then reuse the business logic libraries in an application that accepts input over HTTP instead of the console. Django is a popular web framework that will take care of a lot of the front end concerns for you, or you can use something like CGI if you prefer to be a bit closer to the bare metal. The Python Wiki has a section on web programming that you might find useful.
I am working on developing two systems:
A system that will constantly retrieve economic data from a 3rd party data feed and push it into a MySQL DB (using sqlalchemy)
A server that will allow anyone to query the data in the db over a JSON AJAX API (similar to Yelp or Yahoo API for example)
I have two main questions:
Which Python framework should I use in 2)? Pyramid is my first choice, but if you strongly suggest against it or in favor of something else like Django or Pylons I am definitely wiling to consider it.
Should I develop the two system separately? Or should 1) be a part of 2), running within the framework (using crontab or celery for example)?
Depends on what stage you are at, I would suggest to develop 2 systems because the load to pull data from 3rd party and the load to handle the API would be different. You can scale them into a different types of nodes if you want.
Django-Tastypie (https://github.com/toastdriven/django-tastypie) is not bad, it supports all JSON, XML and YAML. Also you can add OAuth easily. Though, Django itself maybe a bit heavy for your needs at this time.
You might want to check out web2py's new functionality for easily generating RESTful API's, particularly its parse_as_rest and smart_query functions. You might also consider using web2py's database abstraction layer to handle #1.
If you need any help, ask on the mailing list.
I agree with Anthony, you should look at Web2Py. It is very easy to get started, very low learning cure and easy to deploy on many systems including Linux, Windows and Amazon.
So far I have found nothing that Web2Py can not do. But more importantly it does things how you would think they should be done, so if you are not sure, very often a guess is good enough and it just works. If you do get stuck, it has by far the best and most up to date documentation for any Python Web Framework.
Even with all it's great features, easy use and up to date documentation, you will also find that the web2py user group on Google, is like having a paid for help desk staffed 24 hours a day. Most questions are answered with a couple minutes and Massimo (The original creator of Web2Py) goes out of his way not only to help, but to implement new ideas, suggestions and bug fixes within days of them being raised in the group.
I'm new to python, django and google app engine. All great tools and have been enjoying working with them.
However, on my production site its taking 4 seconds to load a webpage, which I think is horrible and needs to be less than a second. I've also verified the long amount of time is in the request to the get the page, not downloading any media files.
First thought is yes, it still has the first start issues any gae app would, I'm not trying to fix those. I understand that the first time you hit your website after uploading a new version it needs to load up the code for the first time. Additionally, if your site isn't visited often then this happens alot. All of this I'm aware of and not trying get more info on.
My site is relatively simple and its not loading big data or displaying complicated designs. And on my localhost it runs extremely fast. I should also point out that I'm using Django nonrel, which is a great tool that allows me to develop quickly with django on gae: http://www.allbuttonspressed.com/projects/django-nonrel
The problem I'm having is that its taking way to long for pages to load in production and I need to get to the bottom of it. I'm sure I've coded something poorly, but I'm not familiar enough with python and gae to know the best debugging practices, especially if it only seems to have issues in production.
So for a newbie python / django / google app engine developer, how do I quickly and easily find what functions are taking so much time?
Use appstats.
I'd like to do some server-side scripting using Python. But I'm kind of lost with the number of ways to do that.
It starts with the do-it-yourself CGI approach and it seems to end with some pretty robust frameworks that would basically do all the job themselves. And a huge lot of stuff in between, like web.py, Pyroxide and Django.
What are the pros and cons of the frameworks or approaches that you've worked on?
What trade-offs are there?
For what kind of projects they do well and for what they don't?
Edit: I haven't got much experience with web programing yet.
I would like to avoid the basic and tedious things like parsing the URL for parameters, etc.
On the other hand, while the video of blog created in 15 minutes with Ruby on Rails left me impressed, I realized that there were hundreds of things hidden from me - which is cool if you need to write a working webapp in no time, but not that great for really understanding the magic - and that's what I seek now.
CGI is great for low-traffic websites, but it has some performance problems for anything else. This is because every time a request comes in, the server starts the CGI application in its own process. This is bad for two reasons: 1) Starting and stopping a process can take time and 2) you can't cache anything in memory. You can go with FastCGI, but I would argue that you'd be better off just writing a straight WSGI app if you're going to go that route (the way WSGI works really isn't a whole heck of a lot different from CGI).
Other than that, your choices are for the most part how much you want the framework to do. You can go with an all singing, all dancing framework like Django or Pylons. Or you can go with a mix-and-match approach (use something like CherryPy for the HTTP stuff, SQLAlchemy for the database stuff, paste for deployment, etc). I should also point out that most frameworks will also let you switch different components out for others, so these two approaches aren't necessarily mutually exclusive.
Personally, I dislike frameworks that do too much magic for me and prefer the mix-and-match technique, but I've been told that I'm also completely insane. :)
How much web programming experience do you have? If you're a beginner, I say go with Django. If you're more experienced, I say to play around with the different approaches and techniques until you find the right one.
The simplest web program is a CGI script, which is basically just a program whose standard output is redirected to the web browser making the request. In this approach, every page has its own executable file, which must be loaded and parsed on every request. This makes it really simple to get something up and running, but scales badly both in terms of performance and organization. So when I need a very dynamic page very quickly that won't grow into a larger system, I use a CGI script.
One step up from this is embedding your Python code in your HTML code, such as with PSP. I don't think many people use this nowadays, since modern template systems have made this pretty obsolete. I worked with PSP for awhile and found that it had basically the same organizational limits as CGI scripts (every page has its own file) plus some whitespace-related annoyances from trying to mix whitespace-ignorant HTML with whitespace-sensitive Python.
The next step up is very simple web frameworks such as web.py, which I've also used. Like CGI scripts, it's very simple to get something up and running, and you don't need any complex configuration or automatically generated code. Your own code will be pretty simple to understand, so you can see what's happening. However, it's not as feature-rich as other web frameworks; last time I used it, there was no session tracking, so I had to roll my own. It also has "too much magic behavior" to quote Guido ("upvars(), bah").
Finally, you have feature-rich web frameworks such as Django. These will require a bit of work to get simple Hello World programs working, but every major one has a great, well-written tutorial (especially Django) to walk you through it. I highly recommend using one of these web frameworks for any real project because of the convenience and features and documentation, etc.
Ultimately you'll have to decide what you prefer. For example, frameworks all use template languages (special code/tags) to generate HTML files. Some of them such as Cheetah templates let you write arbitrary Python code so that you can do anything in a template. Others such as Django templates are more restrictive and force you to separate your presentation code from your program logic. It's all about what you personally prefer.
Another example is URL handling; some frameworks such as Django have you define the URLs in your application through regular expressions. Others such as CherryPy automatically map your functions to urls by your function names. Again, this is a personal preference.
I personally use a mix of web frameworks by using CherryPy for my web server stuff (form parameters, session handling, url mapping, etc) and Django for my object-relational mapping and templates. My recommendation is to start with a high level web framework, work your way through its tutorial, then start on a small personal project. I've done this with all of the technologies I've mentioned and it's been really beneficial. Eventually you'll get a feel for what you prefer and become a better web programmer (and a better programmer in general) in the process.
If you decide to go with a framework that is WSGI-based (for instance TurboGears), I would recommend you go through the excellent article Another Do-It-Yourself Framework by Ian Bicking.
In the article, he builds a simple web application framework from scratch.
Also, check out the video Creating a web framework with WSGI by Kevin Dangoor. Dangoor is the founder of the TurboGears project.
If you want to go big, choose Django and you are set. But if you want just to learn, roll your own framework using already mentioned WebOb - this can be really fun and I am sure you'll learn much more (plus you can use components you like: template system, url dispatcher, database layer, sessions, et caetera).
In last 2 years I built few large sites using Django and all I can say, Django will fill 80% of your needs in 20% of time. Remaining 20% of work will take 80% of the time, no matter which framework you'd use.
It's always worth doing something the hard way - once - as a learning exercise. Once you understand how it works, pick a framework that suits your application, and use that. You don't need to reinvent the wheel once you understand angular velocity. :-)
It's also worth making sure that you have a fairly robust understanding of the programming language behind the framework before you jump in -- trying to learn both Django and Python at the same time (or Ruby and Rails, or X and Y), can lead to even more confusion. Write some code in the language first, then add the framework.
We learn to develop, not by using tools, but by solving problems. Run into a few walls, climb over, and find some higher walls!
If you've never done any CGI programming before I think it would be worth doing one project - perhaps just a sample play site just for yourself - using the DIY approach. You'll learn a lot more about how all the various parts work than you would by using a framework. This will help in you design and debug and so on all your future web applications however you write them.
Personally I now use Django. The real benefit is very fast application deployment. The object relational mapping gets things moving fast and the template library is a joy to use. Also the admin interface gives you basic CRUD screens for all your objects so you don't need to write any of the "boring" stuff.
The downside of using an ORM based solution is that if you do want to handcraft some SQL, say for performance reasons, it much harder than it would have been otherwise, although still very possible.
If you are using Python you should not start with CGI, instead start with WSGI (and you can use wsgiref.handlers.CGIHandler to run your WSGI script as a CGI script. The result is something that is basically as low-level as CGI (which might be useful in an educational sense, but will also be somewhat annoying), but without having to write to an entirely outdated interface (and binding your application to a single process model).
If you want a less annoying, but similarly low-level interface, using WebOb would provide that. You would be implementing all the logic, and there will be few dark corners that you won't understand, but you won't have to spend time figuring out how to parse HTTP dates (they are weird!) or parse POST bodies. I write applications this way (without any other framework) and it is entirely workable. As a beginner, I'd advise this if you were interested in understanding what frameworks do, because it is inevitable you will be writing your own mini framework. OTOH, a real framework will probably teach you good practices of application design and structure. To be a really good web programmer, I believe you need to try both seriously; you should understand everything a framework does and not be afraid of its internals, but you should also spend time in a thoughtful environment someone else designed (i.e., an existing framework) and understand how that structure helps you.
OK, rails is actually pretty good, but there is just a little bit too much magic going on in there (from the Ruby world I would much prefer merb to rails). I personally use Pylons, and am pretty darn happy. I'd say (compared to django), that pylons allows you to interchange ints internal parts easier than django does. The downside is that you will have to write more stuff all by youself (like the basic CRUD).
Pros of using a framework:
get stuff done quickly (and I mean lighning fast once you know the framework)
everything is compying to standards (which is probably not that easy to achieve when rolling your own)
easier to get something working (lots of tutorials) without reading gazillion articles and docs
Cons:
you learn less
harder to replace parts (not that much of an issue in pylons, more so with django)
harder to tweak some low-level stuff (like the above mentioned SQLs)
From that you can probably devise what they are good for :-) Since you get all the code it is possible to tweak it to fit even the most bizzare situations (pylons supposedly work on the Google app engine now...).
For smaller projects, rolling your own is fairly easy. Especially as you can simply import a templating engine like Genshi and get alot happening quite quickly and easily. Sometimes it's just quicker to use a screwdriver than to go looking for the power drill.
Full blown frameworks provide alot more power, but do have to be installed and setup first before you can leverage that power. For larger projects, this is a negligible concern, but for smaller projects this might wind up taking most of your time - especially if the framework is unfamiliar.