My web host does not have python and I am trying to build a machine learning application. I know that heroku lets you use python. I was wondering if I could use heroku as a python server? As in I would let heroku do all of the python processing for me and use my regular domain for everything else.
Yes, and it may be a pain at first but once it is set I would say Heroku is the easiest platform to continually deploy to. However, it is not intuitive - don't try and just 'take a stab' at it; follow a tutorial and try and understand why Heroku works the way it does.
Following the docs is a good bet; Heroku has great documentation for the most part.
Here's the generalized workflow for deploying to Heroku:
Locally, create your project and use virtualenv to install/manage
libraries.
Initialize a git repository in the base dir for your
Python project; create a heroku remote (heroku create)
Create a
procfile for Heroku to use when starting gunicorn (or see
the options for using waitress/etc); this is used by Heroku to start your process
cd to your base dir; freeze
your virtualenv (pip freeze > requirements.txt) and add/commit
requirements.txt. This tells Heroku what packages need to be installed, a requirement for your deployment to work. If you are trying to run a Python project and there are required packages missing, the app will be unable to start and Heroku will display an Internal Server Error.
Whenever changes are made, git commit your changes and git push heroku master to push all commits to Heroku. This will cause Heroku to restart the server application with your updated deployment. If there's a failure, you can use heroku rollback to just return to your last deployment.
In reality, it's not a pain in the ass, just particular. Knowing the rules of Heroku, you are able to manage your deployment with command-line git commands with ease.
One caveat - If deploying Django, Flask applications etc there are peculiarities to account for; specifically, non-project files (including assets) should NOT be stored on Heroku as Heroku periodically restarts your 'dyno' (server instance(s)), loading the whole project from the latest push to Heroku. With Django and Flask, this typically means serving assets/static/media files from an Amazon S3 bucket.
That being said, if you use virtualenv properly, provision your databases, and follow Heroku practices for serving files and commiting updates, it is (imho) the absolute best platform out there for ease of use, reliable uptime, and well-oiled rolling deployments.
One last tip - if you are creating a Django app, I'd suggest starting your project out of this boilerplate. I have a custom one I use for new projects and can start and publish a project in minutes.
Yes, you can use Heroku as a python server. I put a Python Flask server on Heroku but it was a pain: Heroku seemed to have some difficulties, and there were lots of conflicting advice on getting around those. I eventually got it working, can't remember what web page had the ultimate answer but you might look at this one: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xviii-deployment-on-the-heroku-cloud
Have you done your Python Server on Heroku by using twisted?
I don't know if this can help you.
I see the doc 'Getting Started on Heroku with Python' is about the Django.
It is sure that Heroku can use Twisted from docs
Pure Python applications, such as headless processes and evented web frameworks like Twisted, are fully supported.
django-twisted-server has twisted in django but it isn't on Heroku.
Related
I'm moving away from WordPress and into bespoke Python apps.
I've settled on Django as my Python framework, my only problems at the moment are concerning hosting. My current shared hosting environment is great for WordPress (WHM on CloudLinux), but serving Django on Apache/cPanel appears to be hit and miss, although I haven't tried it as yet with my new hosting company. - who have Python enabled in cPanel.
What is the easiest way for me to set up a VPS to run a hosting environment for say, twenty websites? I develop everything in a virtualenv, but I have no experience in running Django in a production environment as yet. I would assume that venv isn't secure enough or has scalability issues? I've read some things about people using Docker to set up separate Django instances on a VPS, but I'm not sure whether they wrote their own management system.
It's my understanding that each instance Python/Django needs uWSGI and Nginx residing within that virtual container? I'm looking for a simple and robust solution to host 20 Django sites on a VPS - is there an out of the box solution? I'm also happy to develop one and set up a VPS if I'm pointed in the right direction.
Any wisdom would be gratefully accepted.
Andy :)
Traditional approach
Virtualenv is good enough and perfectly ready for production use. You can have multiple virtualenv for multiple projects on the same VM.
If you have multiple database engines for multiple projects. Like, MySQL for one, PostgreSQL for another something like this then you just need to set up each individually.
Install Nginx and configure each according to project.
Install supervisor to manage(restart/start/stop) each project individually.
Anything that required by the project.
Here it has a huge drawback. Because you can't use different versions on your database engine for a different project in an easy way. So, containerization is highly recommended.
For simple and robust solution,
Use Docker(docker-compose) for local and production deployment.
Configure uWsgi with Nginx(Available on docker.)
Create a CI/CD pipeline with any tool like Jenkins.
Monitor your projects using any good tool like Raygun.
That's it.
I created a bash script that deploys as many websites as you want on your server. It automatically installs all dependencies on your server, creates a virtual environment, configure Gunicorn, Nginx, and a database for Django, etc. Check it out:
https://github.com/jdbit/django-auto-deploy
I'm trying to deploy a Node.js application with a child process that runs a machine learning algorithm. I can use this locally, but when I try to run at the Heroku server I recive some messages calling that are some libraries missing, like bellow:
ModuleNotFoundError: No module named 'pandas'
I tried to create manually the requirements.txt and put the necessary libraries there:
pandas
pymongo
dnspython
scikit-learn
scipy
selenium
webdriver-manager
textblob
But it doesn't work. Do I need to do some extra configuration?
Thank you so much for your help!
The way your Heroku dynos run your software is through something called a buildpack.
When you deploy an application to Heroku, it looks at your code and tries to figure out which programming language you are using, then based on that, will run your app using the corresponding buildpack.
For example, if you deploy an app to Heroku and the app has a package.json file in the root of your project directory, Heroku will assume your app is a JavaScript app and use the Node.js buildpack.
Buildpacks contain a number of pre-installed dependencies. For example, the Node.js buildpack contains node (so you can run your JavaScript code) as well as a number of Linux dependencies so that your app will be able to install common libraries/tools that might rely on them.
But... One downside of this buildpack strategy is that if you're deploying a Node.js app, for example, the default Node.js building will NOT come with Python and the various Python library dependencies installed. This is because Heroku supports a lot of different programming environments, and it would be slow/complex if there was just a single buildpack that had EVERYTHING installed. It'd be crazy!
So what you need to do, in your case, is use multiple buildpacks! Heroku has a way for you to enable multiple buildpacks for your app so that your app can have the Node.js dependencies as well as the Python dependencies, for example!
This article on Heroku's documentation site explains how to use multiple buildpacks for a given app.
Here are the specific instructions for simplicity's sake:
# This command will set your default buildpack to Node.js
$ heroku buildpacks:set heroku/nodejs
# This command will set it up so that the Heroku Python buildpack will run first
$ heroku buildpacks:add --index 1 heroku/python
By doing the above, you'll be able to have Heroku install your Python dependencies via a traditional requirements.txt file like you would with any normal Python application.
I wrote a little bottle web application, played around with setup.py, and I'm ready to deploy.
I had two questions:
How do I deploy this new web app in production? Do I need a a webserver or reverse proxy? Can my main.py be run as a service? How is this typically done?
My web app uses a mongodb backend. Is there a way to package that with my application or to require it / look for it / alert the user when the installer runs? How is this typically done?
Pythonanywhere allows you for free to test/deploy a Bottle app with Pymongo driver. This is one of the webhostings which is python-deployers friendly... If you need something for a serious trafic, then you have to consider a paid programm or your own (virtual/dedicated) hosted server.
I ran my Bottle applications on OpenShift. There's a quickstart template on GitHub for getting a Bottle application online. You could just copy their directory structure and files, add your own, and send it all to the git repository that is created for you. OpenShift uses git when it comes to development, so if you ever make any changes, you just need to commit and push them to the server again.
MongoDB and RockMongo (a GUI for managing your MongoDB database, similar to phpMyAdmin for MySQL) are built in. You get three 'gears' or instances to work with, so the Bottle application will be one, with MongoDB and RockMongo included. You just have to enable them.
I would like to be able to log the number of words in certain files in a Github repo whenever there is a new push to the repo. I have set up a hook on Github to hit a Django Heroku app url after each push, but I don't know how to run a git pull in python from a Django app running on Heroku. Is it possible to write to the local file system in Heroku?
Check out github repo from Heroku?
from the command line you can pull from heroku easily: git pull heroku master
have set up a hook on Github to hit a Django Heroku app url after each push, but I don't know how to run a git pull in python from a Django app running on Heroku?
Is it a different heroku App (from the one that was deployed) that will be doing the pull?
Yes? then you are going to have issues. Because the pull app needs permission (heroku login) to pull... and it wont have it. Also, b/c of the ephemeral filesystem, even if you login (via heroku run bash or the like) to it, the pull app will eventually lose its logged in session (see why below)
No? then don't pull. just use the os filesystem libraries to look into the application directory...
Is it even possible to write to the local file system in Heroku?
Yes and No. You can write to the local filesystem, but its going to get nuked. See: https://devcenter.heroku.com/articles/dynos#ephemeral-filesystem
Also, with the EFS, each dyno is going to have a different EFS - so each web process is in a way sandboxed.
How should the project be deployed and run. There are loads of tools in this space. Which should be used and why?
Supervisor
Gunocorn
Ngnix
Fabric
Boto
Pip
Virtualenv
Load balancers
It depends on your configuration. We are using the following stack for our environment on Rackspace, but you can setup the same thing on AWS with EC2 instances.
Ubuntu 11.04
Varnish (in memory cache) to avoid disk seeks
NginX to server static content
Apache to server dynamic content (MOD-WSGI)
Python 2.7.2 with Django
Jenkins for our continuous builds
GIT for version control
Fabric for the deployment.
So the way it works is that a GIT push to the origin repository is being polled by Jenkins. Jenkins then pulls the changes down from the origin. Builds a Python Egg, runs Unit tests, uses Fabric to deploy this egg to the environments necessary and reloads the Apache config to make sure the forked Apache processes are picking up the new Python egg.
Hope this helps.
As Michael Klockel already stated depends on your configuration, I have:
Ubuntu 10.04 LTS
Nginx
Uwsgi
git version control
python virtualenv and pip
You can check the deployment settings here:
Django, Virtualenv, nginx + uwsgi import module wsgi error
and why I use nginx and uwsgi here:
http://nichol.as/benchmark-of-python-web-servers
Also I use fabric for the deployment of the app, and chef solo http://ericholscher.com/blog/2010/nov/8/building-django-app-server-chef/
johny cache for sql queries and raven and sentry to keep a log of whats going on on the app.
I'd use uWSGI+Nginx from a performance perspective (I think the comparison has already been linked in another answer), pip and virtualenv for deployment as this keeps things self-contained, and facilitates clean deployment using fabric or similar. Use git for version control. Jenkins can handle continuous integration. I'd use the AWS load balancer (ELB) in front of your EC2 instances for balancing - does the job without you having to fret too much about it. django-storages for uploading your static files to s3, that saves you the effort of having another server to hand out static files.
However, it depends a little on your admin overheads. If you're looking for something clean and simple for deployment and scaling, I'd scrap the whole AWS EC2 stack, use Heroku as a front end, and s3 for your static files. This saves all the admin time of maintaining the boxes, and allows you to concentrate on the dev.