My intention is to run python code 24/7 over several months to collect data through API calls and alert me if certain conditions are met.
How can I do that without keeping the code running on my laptop 24/7? Is there a way of doing it in "the cloud"?
Preferably free but would consider paying. Simplicity also a plus.
There is a plenty of free possibilities. You can use heroku servers. They have a strong documentation for python and is free (but of course with limit on megabytes and time usage). Heroku is also good because they have a lot of extensions (databases, dashboards, loggers etc.). In order to deploy your app on heroku you need to have some experience of working with git (but not that much, don't worry).
Another possibility could be Google Cloud Platform, which should be also easy to use.
Related
So far when dealing with web scraping projects, I've used GAppsScript, meaning that I can easily trigger the script to be run once a day.
Is there an equivalent service when dealing with python scripts? I have a RaspberryPi, so I guess I can keep it on 24/7 and use cronjobs to trigger the script daily. But that seems rather wasteful, since I'm talking about a few small scripts that take only a few seconds to run.
Is there any service that allows me to trigger a python script once a day? (without a need to keep a local machine on 24/7) The simpler the solution the better, wouldn't want to overengineer such a basic use case if a ready-made system already exists.
The only service I've found so far to do this with is WayScript and here's a python example running in the cloud. The free tier that should be enough for most simple/hobby-tier usecases.
Let me explain what I'm trying to achieve. In the past while working on Java platform, I used to write Java codes(say, to push or pull data from MySQL database etc.) then create a war file which essentially bundles all the class files, supporting files etc and put it under a servlet container like Tomcat and this becomes a web service and can be invoked from any platform.
In my current scenario, I've majority of work being done in Java, however the Natural Language Processing(NLP)/Machine Learning(ML) part is being done in Python using the NLTK, Scipy, Numpy etc libraries. I'm trying to use the services of this Python engine in existing Java code. Integrating the Python code to Java through something like Jython is not that straight-forward(as Jython does not support calling any python module which has C based extensions, as far as I know), So I thought the next option would be to make it a web service, similar to what I had done with Java web services in the past. Now comes the actual crux of the question, how do I run the ML engine as a web service and call the same from any platform, in my current scenario this happens to be Java. I tried looking in the web, for various options to achieve this and found things like CherryPy, Werkzeug etc but not able to find the right approach or any sample code or anything that shows how to invoke a NLTK-Python script and serve the result through web, and eventually replicating the functionality Java web service provides. In the Python-NLTK code, the ML engine does a data-training on a large corpus(this takes 3-4 minutes) and we don't want the Python code to go through this step every time a method is invoked. If I make it a web service, the data-training will happen only once, when the service starts and then the service is ready to be invoked and use the already trained engine.
Now coming back to the problem, I'm pretty new to this web service things in Python and would appreciate any pointers on how to achieve this .Also, any pointers on achieving the goal of calling NLTK based python scripts from Java, without using web services approach and which can deployed on production servers to give good performance would also be helpful and appreciable. Thanks in advance.
Just for a note, I'm currently running all my code on a Linux machine with Python 2.6, JDK 1.6 installed on it.
One method is to build an XML-RPC server, but you may wish to fork a new process for each connection to prevent the server from seizing up. I have written a detailed tutorial on how to go about this: https://speakerdeck.com/timclicks/case-studies-of-python-in-parallel?slide=68.
NLTK based system tends to be slow at response per request, but good throughput can be achieved given enough RAM.
I am trying to make a web application to perform scientific and engineering calculations. I am new to web developing and I've been looking for a free framework (with free hosting), that's why I came to Google App Engine, but there is no way to get scipy working, so I decided to switch to Java instead of python (although I found PythonAnywhere and it has numpy, scipy, etcetera, it has no GUI-building support like PyQt, wx, Tkinter...).
I would like to know if there is a way to use COLT or so for Java in the Google App Engine, or if there is some other option. I would rather free options since I'm at college, but cheap-customizable-options are totally welcome :D (even if it means to use another language)
PD: I hope this was understandable since english is not my language.
EDIT:
I TRIED to use apache commons math, and it seems like it's not going to work. The short answer to my question is: NO.
I believe GAE is severely limiting in what it will allow you to run.
I doubt you will find a completely free Java hosting solution.
To clarify the statements in other posts, GAE is incredibly limiting with respect to Python packages with C extensions. Anything pure Python will work fine. Scipy makes heavy use of C extensions, so it falls into this category.
Google recently introduced Python2.7 support, and with it, the ability to use NumPy on App Engine. I'm not sure if this covers your need, but it might be worth checking out.
I only develop with Python for Google App Engine, so I'm afraid I can't comment on the state of Java external dependencies.
GAE will limit lot of things if not all in your case. You might want to try out Heroku, Amazon Web Services within their free quota.
I see no reason not to do this. You can run front-end instances which can use 800MHz of processor and 128MB of RAM - you can run one all the time for free but you need to be able to split your tasks into 10min sections (if you use tasks, or 30 second sections otherwise). A backend is going to be chargeable and you'd probably find it cheaper to run on another system.
I'm a complete novice in this area, so please excuse my ignorance.
I have three questions:
What's the best (fastest, easiest, headache-free) way of hosting a python program online?
I'm currently looking at Google App Engine and Web Frameworks for Python, but all the options are a bit overwhelming.
Which gui/viz libraries will transfer to a web app environment without problems?
I'm willing to sacrifice some performance for the sake of simplicity.
(Google App Engine can't do C libraries, so this is causing a dilemma.)
Where can I learn more about running a program locally vs. having a program continuously run on a server and taking requests from multiple users?
Currently I have a working Python program that only uses standard Python libraries. It currently uses around 2.7gb of ram, but as I increase my dataset, I'm predicting it will use closer to 6gb. I can run it on my personal machine, and everything is just peachy. I'd like to continue developing on the front end on my home machine and implement the web app later.
Here is a relevant, previous post of mine.
Depending on your knowledge with server administration, you should consider a dedicated server. I was doing running some custom Python modules with Numpy, Scipy, Pandas, etc. on some data on a shared server with Godaddy. One program I wrote took 120 seconds to complete. Recently we switched to a dedicated server and it now takes 2 seconds. The shared environment used CGI to run Python and I installed mod_python on the dedicated server.
Using a dedicated server allows COMPLETE control (including root access) to the server which allows the compilation and/or installation of anything. It is a bit pricy but if you're making money with your stuff it might be worth it.
Another option would be to use something like http://www.dyndns.com/ where you can host a domain on your own machine.
So with that said, perhaps some answers:
It depends on your requirements. ~4gb of RAM might require a dedicated server. What you are asking is not necessarily an easy task so don't be afraid to get your hands dirty.
Not sure what you mean here.
A server is just a computer that responds to requests. On the dedicated server (I keep mentioning) you are operating in a Unix (or Windows) environment just like you would locally. You use SOFTWARE (e.g. Apache web server) to serve client requests. My vote is mod_python.
It's a greater headache than a dedicated server, but it should be much closer to your needs to go with an Amazon EC2 instance.
http://aws.amazon.com/ec2/#instance
Their extra large instance should be more than large enough for what you need to do, and you only turn the instance on when you need it so you don't have the massive bill that you get with a dedicated server that's the same size.
There are some nice javascript based visualization toolkits out there, so you can model your application to return raw (json) data and render that on the client.
I can mention d3.js http://mbostock.github.com/d3/ and the JavaScript InfoVis Toolkit http://thejit.org/
Is it possible to eliminate the desktop and fully code and deploy a python/django application to the cloud from within a browser? I think Heroku makes it possible to do that with Ruby on Rails but I have not come across a Python/Django equivalent.
I think there are huge advantages of developing code in the cloud. No deployment overhead, no code versioning headaches (because the cloud will track all changes anyway), access your code from anywhere, even an iPad, if you want to make a small change to your production code. I think Heroku has already proven that this is the future of web development and browser based IDEs like Mozilla Bespin are already getting some traction.
I'm not really sure that Heroku does that, it just a cloud space for Rails.....like phpfog (php), dotcloud (many), appengine (java, python)....
I think http://kodingen.com/ is something more like what you ask, code in the web and deploy on it.....
Haven't tried Kodingen though....
As an updated answer to #jjchiw, Koding provides a you with Python2 and Python3, and gives you root access to a full Ubuntu VM, so everything will work just like you expect :)
I say updated, because Kodingen turned into Koding a year ago, and is under very active development.
Let me know if you have any questions :)