We Keep Coding

Can I use Heroku as a Python server?

My web host does not have python and I am trying to build a machine learning application. I know that heroku lets you use python. I was wondering if I could use heroku as a python server? As in I would let heroku do all of the python processing for me and use my regular domain for everything else.

Yes, and it may be a pain at first but once it is set I would say Heroku is the easiest platform to continually deploy to. However, it is not intuitive - don't try and just 'take a stab' at it; follow a tutorial and try and understand why Heroku works the way it does.
Following the docs is a good bet; Heroku has great documentation for the most part.
Here's the generalized workflow for deploying to Heroku:
Locally, create your project and use virtualenv to install/manage
libraries.
Initialize a git repository in the base dir for your
Python project; create a heroku remote (heroku create)
Create a
procfile for Heroku to use when starting gunicorn (or see
the options for using waitress/etc); this is used by Heroku to start your process
cd to your base dir; freeze
your virtualenv (pip freeze > requirements.txt) and add/commit
requirements.txt. This tells Heroku what packages need to be installed, a requirement for your deployment to work. If you are trying to run a Python project and there are required packages missing, the app will be unable to start and Heroku will display an Internal Server Error.
Whenever changes are made, git commit your changes and git push heroku master to push all commits to Heroku. This will cause Heroku to restart the server application with your updated deployment. If there's a failure, you can use heroku rollback to just return to your last deployment.
In reality, it's not a pain in the ass, just particular. Knowing the rules of Heroku, you are able to manage your deployment with command-line git commands with ease.
One caveat - If deploying Django, Flask applications etc there are peculiarities to account for; specifically, non-project files (including assets) should NOT be stored on Heroku as Heroku periodically restarts your 'dyno' (server instance(s)), loading the whole project from the latest push to Heroku. With Django and Flask, this typically means serving assets/static/media files from an Amazon S3 bucket.
That being said, if you use virtualenv properly, provision your databases, and follow Heroku practices for serving files and commiting updates, it is (imho) the absolute best platform out there for ease of use, reliable uptime, and well-oiled rolling deployments.
One last tip - if you are creating a Django app, I'd suggest starting your project out of this boilerplate. I have a custom one I use for new projects and can start and publish a project in minutes.

Yes, you can use Heroku as a python server. I put a Python Flask server on Heroku but it was a pain: Heroku seemed to have some difficulties, and there were lots of conflicting advice on getting around those. I eventually got it working, can't remember what web page had the ultimate answer but you might look at this one: http://blog.miguelgrinberg.com/post/the-flask-mega-tutorial-part-xviii-deployment-on-the-heroku-cloud

Have you done your Python Server on Heroku by using twisted?
I don't know if this can help you.
I see the doc 'Getting Started on Heroku with Python' is about the Django.
It is sure that Heroku can use Twisted from docs
Pure Python applications, such as headless processes and evented web frameworks like Twisted, are fully supported.
django-twisted-server has twisted in django but it isn't on Heroku.

Aldryn - DjangoCMS install addons not present in "Manage Addons"

I am quite a Django n00b, and figured using Aldryn for my first real django site would be a good idea!
I have successfully installed and implementer Aldryn News & Blog.
Now I would like to install Aldryn Search that is not accessible from the "Manage Addons" under the Aldryn control panel.
I very confused on how to install an addon like Aldryn Search that is not accessible from within "Manage Addons". Should I somehow use the "Add custom Addon" and register the package as a new custom addon.
Or should I create a local development environment and somehow install the addon and upload it? (does it exist a tutorial for this?)
Thank you!

There are various ways in which to install arbitrary Django packages into an Aldryn project.
The quick, easy way
The easiest, quickest way is simply to place the module(s) you need into the project directory, thus placing them on the Python path. You need then to make sure that your settings.py, urls.py and so on are appropriately configured. Then you can push these changes to Aldryn itself. This is described in Adding a new application to your Aldryn project - the quick and easy way.
The create-an-Addon way
A more involved way to do it, that has benefits for long-term use and re-use, is to turn the package into a private or public Aldryn Addon. This is described in Developing an Addon application for Aldryn.
A middle way
Another way is somewhere between the two. Add the package to the project's requirements.in - you can do this in various ways, for example:
# standard install from PyPI
some-package==1.2.3
# install from an archive
https://example.com/some-package1.2.3.tar.gz#egg=some-package==1.2.3
# install from a GitHub repository
git+http://git#github.com/some-org/some-package.git#egg=some-package==1.2.3
You will need to make sure that your settings.py, urls.py and so on are appropriately configured.
Run aldryn project update. This in effect redeploys your project locally, except for:
docker-compose run --rm web python manage.py migrate - you need to run any migrations manually, unlike on Aldryn.
Finally, git add, commit and push your changes to your project, and redeploy it on Aldryn.
This method isn't yet documented in the Aldryn support system, but will be soon.

That's a very valid question in my opinion since add-ons are wrapped into an additional directory that makes the django app inside invisible to Django's INSTALLED_APPS.
If you add them to addons-dev they are ignored by git.
A possible solution (even if maybe not the cleanest) would be to unignore addons-dev by adding !/addons-dev to the .gitignore in the project's root directory and then add -e /app/addons-dev/aldryn-package-name to requirements.in (outside the section generated/overwritten by Aldryn). That's what aldryn project develop aldryn-package-name does (for the local environment).
Similarly, if you have a git repository that contains the code (like aldryn-search) you would use -e git+https://github.com/aldryn/aldryn-search.git in requirements.in
In case you need to apply changes to the addon code best practise would be forking the original repository and then checking out your fork as per above instructions.

Heroku: Python dependencies in private repos without storing my password

The Problem
My problem is exactly like How do I install in-house requirements for Python Heroku projects? and How to customize pip's requirements.txt in Heroku on deployment?. Namely, I have a private repo from which I need a Python dependency installed into my Heroku app. The canonical answer, given by Heroku's own Kenneth Reitz, is to put something like
-e git+https://username:password#github.com/kennethreitz/requests.git#v0.10.0#egg=requests
in your requirements.txt file.
My security needs prevent my storing my password in a repo. (I also do not want to put the dependency inside my app's repo; they're separate pieces of software and need to be in separate repos.) The only place I can give my password (or, preferably, a GitHub OAuth token or deployment key) to Heroku, is in an environment variable like
heroku config:add GITHUB_OAUTH_TOKEN=12312312312313
Attempted Solutions
I could use a custom .profile in my app's repo, but then I'd be downloading and installing my dependency each time a process (web, worker, etc) restarts.
This leaves using a custom buildpack and the Heroku Labs addon that exposes my heroku config environment before the buildpack compiles. I tried building one on top of Buildpack Multi. The idea is Buildpack Multi is the primary buildpack, and using the .buildpacks file in my app's repo, it first downloads the normal Heroku Python buildpack, then my custom one.
The trouble is even after Buildpack Multi successfully runs the Python buildpack, the Python binary and Pip package are not visible to my buildpack once Buildpack Multi runs. So the custom buildpack just fails outright. (In my tests, the GITHUB_OAUTH_TOKEN environment variable was correctly exposed to the buildpacks.)
The only other thing I can think to try is to make my own fork of the Python buildpack that installs my dependency when it installs everything from requirements.txt, or even rewrites requirements.txt directly. Both of these seem like really heavy solutions to what I would think is a very common problem.
Update: Current Workaround
My custom buildpack (linked above) now downloads and saves my closed-source dependency ("foo") into the vendor directory that the geos buildpack uses. I committed into my app the dependencies that foo itself has into my app's requirements.txt. Thus Pip installs foo's dependencies through my app's requirements.txt and the buildpack adds the vendored copy of foo to my app's environment's PYTHONPATH (so foo's setup.py install never runs).
The biggest problem with this approach is coupling my (admittedly badly written) buildpack with my app. The second problem is that my app's requirements.txt should just list foo as a dependency and leave foo's dependencies to foo to determine. Lastly, there isn't a good way to give myself in six months from now when I forget how I did all this an error message if I forget to set my GITHUB_OAUTH_TOKEN environment variable (or, producing even less useful error feedback would be if the token expires and the environment variable still exists but is no longer valid).
Cry for Help
What (likely obvious) thing am I missing? How have you solved this problem in your apps? Any suggestions on getting my build pack to work, or hopefully an even simpler solution?

I created a buildpack to solve this problem using a custom ssh key stored as an environment variable. As the buildpack is technology agnostic, it can be used to download dependencies using any tool like composer for php, bundler for ruby, npm for javascript, etc: https://github.com/simon0191/custom-ssh-key-buildpack
Add the buildpack to your app:
$ heroku buildpacks:add --index 1 https://github.com/simon0191/custom-ssh-key-buildpack
Generate a new SSH key (lets say you named it deploy_key)
Add the public key to your private repository account. For example:
Github: https://help.github.com/articles/adding-a-new-ssh-key-to-your-github-account/
Bitbucket: https://confluence.atlassian.com/bitbucket/add-an-ssh-key-to-an-account-302811853.html
Encode the private key as a base64 string and add it as the CUSTOM_SSH_KEY environment variable of the heroku app.
Make a comma separated list of the hosts for which the ssh key should be used and add it as the CUSTOM_SSH_KEY_HOSTS environment variable of the heroku app.
# MacOS
$ heroku config:set CUSTOM_SSH_KEY=$(base64 --input ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
# Ubuntu
$ heroku config:set CUSTOM_SSH_KEY=$(base64 ~/.ssh/deploy_key) CUSTOM_SSH_KEY_HOSTS=bitbucket.org,github.com
Deploy your app and enjoy :)

I faced the same problem. Like you, I am amazed how difficult it is to find good documentation on how to install private dependency (whatever the language and the service used).
Because this is not a main concern of service providers, I now try a systematic approach relying as few as possible on idiosyncratic features. I try to find the easier solution for each of these steps:
Pass the credentials to the build environment using a secure channel. For python, use an environment variable containing a SSH key as a base64 string. For js, same with the npm token.
Configure the build process to use these credentials. In the best case it involves configuring ssh to use a deploy key. Otherwise it can as basic as cloning the dependency for later use. For your specific case with python and heroku, you can use the hook 'pre_compile'.
I detailed the process for my future self here: https://gist.github.com/michelbl/a6163522d95540cf0c8b6667bd35d5f5
I need to give access to a private dependency. It can happen for continuous integration or deployment.
Here we use python and github, using the services CircleCI and Heroku. However, the principles applies everywhere.
What is a deploy key?
See https://developer.github.com/v3/guides/managing-deploy-keys/
There are 4 ways of granting access to a private dependency, but deploy keys are a good compromise in term of security and ease of use for projects that do not require too many dependencies (in that case, prefer a machine user). In any case, do not use username/password of a developer account or oauth token as they do not provide privilege limitation.
Create a deploy key:
ssh-keygen -t rsa -b 4096 -C "myself#my_company.com"
Give the public part to gihub.
Give the private part to the service needing access. See below.
General strategy
Whatever the service or the technology that I use, the goal is to access the git repo using ssh, using the deploy key.
Obviously, I do not want to put the deploy key in the repo. But most services (CI, deployment) provide a way to set protected environment variables that can be used at build time. The key can be encoded using base64:
cat deploy-key | base64
cat deploy-key.pub | base64
Most services also provide a way to tailor the build procedure. This is needed to configure ssh to use the deploy key.
CircleCI
Set the deploy key using env variables, encode with base64.
In config.yml, add a step:
echo $DEPLOY_KEY_PRIVATE | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
echo $DEPLOY_KEY_PUBLIC | base64 --decode > ~/.ssh/deploy-key.pub
ssh-add ~/.ssh/deploy-key
# Run this to check which private key is used. If the checkout key is used,
# github replies "Hi my_org/my_package". If the deploy key is used as wished,
# github replies "Hi my_org/my_dependency".
#ssh -i ~/.ssh/deploy-key -T git#github.com || true
# Now pip connects to git+ssh using the deploy key
export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"
pip install -r requirements.txt
requirements.txt can be something like:
# The purpose of this file is to install the private dependency *before*
# setup.py is run.
# Be sure ssh is configured to use a ssh key with read permission to the repo.
git+ssh://git#github.com/my_org/my_dependency#1.0.10
# Run setup.py. The private dependency is already installed with the good
# version so pip doesn't try to fetch it from PyPI.
--editable .
and setup.py does not care about the dependency beeing private:
from distutils.core import setup
setup(
name='my_package',
version='1.0',
packages=[
'my_package',
],
install_requires=[
# Beware, the following package is a private dependency.
# Python provides several way to install private dependencies, none
# are really satisfactory.
# 1. Use dependency_links / --process-dependency-links. Good luck with
# that!
# 2. Maintain a private package repository. Good luck with that!
# 3. Install the private dependency separately before setup.py is run.
# This is now the prefered way. Be sure that ssh is properly
# configured to use a ssh key with read permission to the github repo
# of the private dependency, then run:
# `pip install -r requirements.txt`
'my_dependency==1.0.10',
... # my normal dependencies
'unidecode==1.0.22',
'uwsgi==2.0.15',
'nose==1.3.7', # tests
'flake8==3.5.0', # style
],
)
Heroku
For python, there is no need to write a custom buildpack. First, set the deploy key using env variables, encode with base64.
Then add the hook bin/pre_compile:
# This script configures ssh on Heroku to use the deploy key.
# This is needed to install private dependencies.
#
# Note that this does not work with Heroku review apps. Indeed review apps can
# inherits env variables from their parents, but they access their values after
# the build. You would need a way to pass the ssh key to this script another
# way.
#
# See also
# * https://stackoverflow.com/questions/21297755/heroku-python-dependencies-in-private-repos-without-storing-my-password#
# * https://github.com/bjeanes/ssh-private-key-buildpack
# Ensure we have an ssh folder
if [ ! -d ~/.ssh ]; then
mkdir -p ~/.ssh
chmod 700 ~/.ssh
fi
# Create the key files
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key
chmod 400 ~/.ssh/deploy-key
cat $ENV_DIR/DEPLOY_KEY | base64 --decode > ~/.ssh/deploy-key.pub
#ssh-add ~/.ssh/deploy-key
# If you want to disable host verification, you could use that.
#ssh -oStrictHostKeyChecking=no -T git#github.com 2>&1
# Run that if you want to check that ssh uses the correct key.
#ssh -i ~/.ssh/deploy-key -T git#github.com || true
# Configure ssh to use the correct deploy key when connecting to github.
# Disables host verification.
echo -e "Host github.com\n"\
" IdentityFile ~/.ssh/deploy-key\n"\
" IdentitiesOnly yes\n"\
" UserKnownHostsFile=/dev/null\n"\
" StrictHostKeyChecking no"\
>> ~/.ssh/config
# Unfortunately this does not seem to work.
#export GIT_SSH_COMMAND="ssh -i ~/.ssh/deploy-key"
# The vanilla python buildpack can now install all the dependencies in
# requirement.txt

Create a private PyPI server
If you create your own PyPI server, you can simply list your packages in your requirements.txt file and then store the url for your server (including username and password) in the config variable, PIP_EXTRA_INDEX_URL.
For example:
heroku config:set PIP_EXTRA_INDEX_URL='https://username:password#privateserveraddress.com/simple'
Note that this is the same as using the pip install command line option, --extra-index-url. (See https://pip.pypa.io/en/stable/user_guide/#environment-variables)
The primary index url will still be the default (https://pypi.org/simple). This means that pip will first attempt to resolve package names in your requirements file at the default PyPI server, and then try your private server second.
If you need packages in your private server that have the same name as packages in PyPI, then you need the primary index url to be your server and the --extra-index-url option to be the default server's url. You would need to do this if you want to host your own version of an existing package without changing the package name. I haven't tried this, but it currently looks like you would need to to create a fork of heroku's official python buildpack and make a small change to the bin/steps/pip-install file.
The reason pip has access to the PIP_EXTRA_INDEX_URL is because of this block in that file:
# Set Pip env vars
# This reads certain environment variables set on the Heroku app config
# and makes them accessible to the pip install process.
#
# PIP_EXTRA_INDEX_URL allows for an alternate pypi URL to be used.
if [[ -r "$ENV_DIR/PIP_EXTRA_INDEX_URL" ]]; then
PIP_EXTRA_INDEX_URL="$(cat "$ENV_DIR/PIP_EXTRA_INDEX_URL")"
export PIP_EXTRA_INDEX_URL
mcount "buildvar.PIP_EXTRA_INDEX_URL"
fi
Code like this is necessary to read config variables in buildpacks (see https://devcenter.heroku.com/articles/buildpack-api#buildpack-api), but you should be able to simply duplicate this codeblock, replacing PIP_EXTRA_INDEX_URL with PIP_INDEX_URL. Then set PIP_INDEX_URL to your private server's url and PIP_EXTRA_INDEX_URL to the default PyPI url.
If you are using another source instead of a private PyPI server, such as github, and simply need a way to avoid hardcoding a username and password in your requirements.txt file, then also note that you can use environment variables in requirements.txt (see https://pip.pypa.io/en/stable/reference/pip_install/#using-environment-variables). You would just have to export them in bin/steps/pip-install as you would for PIP_INDEX_URL.

You could use a pre-compile step as described here to run something like M4 to do substitutions on your requirements.txt to file in the password from the environment variable.

how to make finished django app a website

So I finished making a simple django app. All it is, is a website which has a /admin page, where someone can sign into and create posts which update to the site. Basically like a blog. I could've done it with WordPress but I prefer django a lot more. My last question is, how do I know host the django app as a website? I don't want a step by step guide, I know this site isn't meant for it but can someone give me a textbook or a link to something which shows how to do it? Because really I tried looking and couldn't find anything. The app isn't for me, it's for an organization and I don't know if their site is going to be .com, .net etc. And I don't know what they will be hosting the site on but I really don't want to end up having to redo everything with WordPress in order to just host the site.

You could host your app on Amazon EC2. If you Google Django + EC2, you will see lots of tutorials.

you can host use OPENSHIFT.
Create an account at http://openshift.redhat.com/
Install the RHC client tools if you have not already done so:
sudo gem install rhc
Create a python-2.6 application
rhc app create -a django -t python-2.6
Add this upstream repo
cd django
git remote add upstream -m master git://github.com/openshift/django-example.git
git pull -s recursive -X theirs upstream master
Then push the repo upstream
git push
Here, the admin user name and password will be displayed, so pay
special attention.
That's it. You can now checkout your application at:
http://django-$yournamespace.rhcloud.com
now the difficult part
you want to edit the files in folder django/wsgi/openshift
mostly you want to edit the settings.py and copy your project folder to this folder.
now check it in localhost ie python manage.py runserver. if it worked successfully you can call git push that is it.

https://www.heroku.com/‎ is awesome web-hosting with django support and totally simplified deployment process. Tutorial included.

App Engine Bulkloader

I am trying to use Bulkloader of google app engine but unfortunately could not understand what to do from documentation. It says add this part to app.yaml
builtins:
- remote_api: on
ok i have added. Then says that i have to execute this command
appcfg.py update
but i don't have any appcfg.py file. And also what is the command which executes this line?
Please somebody tell me what i am missing I use AppEngineLauncher to upload my project to server.. I have naver used a command to update or upload it.
Thanks in advance..

If you're using App Engine Launcher, simply click the Deploy button; it runs appcfg.py update. appcfg is installed as part of the SDK.

In order to call it globally you must set the python path to include google appengine library, or call it from its excact location
~/google_appengine/appcfg.py update ~/project/path_to_your_yaml_file/
where ~ stands for you home folder.

For those using the the Java GAE SDK, it comes wtih appcfg.sh, but doesn't support full features of apppcfg.py. You need to also install the Python GAELauncher. On Mac, it installs appcfg.py in /usr/local/bin.
Working only with my local datastore, I did not have an app.yaml, nor did I need to update it as per the uploading data GAE documentation. Just needed to update the web.xml with the RemoteApiServlet.
Main gotcha is the Authentication error while connecting to the local data store. It may be specific to Mac and the Charles proxy work around mentioned in that thread likewise worked for me. I did have to modify the default proxy port from 8888 to avoid conflict with my app. The workaround is tedious and the trial version of Charles comes with limitations such as shutdown after 30 minutes.

To use the main database for development, first download it from app engine:
bulkloader.py --dump --url http://<app name>.appspot.com/_ah/remote_api --filename dump.bin
It will ask for an email and password (use your google login details).
To upload it to the development environment, make sure remote_api is
enabled in app.yaml
https://developers.google.com/appengine/docs/python/tools/uploadingdata#Setting_Up_remote_api
Then run:
bulkloader.py --restore --url http://localhost:8080/_ah/remote_api --filename dump.bin --application dev~<app name>
This time for the email, just leave it blank and hit enter.