We Keep Coding

Where is my python-flask app source stored on ec2 instance deployed with elastic beanstalk?

I just deployed a flask-python app with elastic beanstalk on AWS but cannot locate my app source files like application.py or templates/index.html etc
I've looked at looked at /var/../.. or /opt/../.. etc but nowhere to be found.
Is there an ebs command like $ eb find 'filename.py' etc?

/opt/python – Root of where you application will end up.
/opt/python/current/app – The current application that is hosted in the environment.
/opt/python/on-deck/app – The app is initially put in on-deck and then, after all the deployment is complete, it will be moved to current. If you are getting failures in yourcontainer_commands, check out out the on-deck folder and not the current folder.
/opt/python/current/env – All the env variables that eb will set up for you. If you are trying to reproduce an error, you may first need to source /opt/python/current/env to get things set up as they would be when eb deploy is running.
opt/python/run/venv – The virtual env used by your application; you will also need to run source /opt/python/run/venv/bin/activate if you are trying to reproduce an error

As of today, using the default AWS Linux option when creating eb (Python 3.7 running on 64bit Amazon Linux 2/3.1.1), I found the application files in:
/var/app/current
If that doesn't work you can search for a file that you know to be unique in your app, e.g
sudo find / -name hello.html
In my case the above returns
/var/app/current/templates/hello.html

Is there a better way to set a gcloud project in a directory?

I work on multiple appengine projects in any given week. i.e. assume multiple clients. Earlier I could set application in app.yaml. So whenever I did appcfg.py update.... it would ensure deployment to the right project.
When deploying, the application variable throws an error with gcloud deploy. I had to use
gcloud app deploy --project [YOUR_PROJECT_ID]. So what used to be a directory level setting for a project, is now going into our build tooling. And missing out that simple detail can push a project code to the wrong customer.
i.e. if I did gcloud config set project proj1 and then somehow did a gcloud app deploy in proj2, it would deploy to proj1. Production deployments are done after detailed verification on the build tools and hence it is less of an issue there because we still use the --project flag.
But its hard to do similar stuff on the development environment. dev_appserver.py doesn't have a --project flag.
When starting dev_appserver.py I've to do gcloud config set project <project-id> before I start the server. This is important when I using stuff like PubSub or GCS (in dev topics or dev buckets).
Unfortunately, missing out a simple configuration like setting a project ID in a dev environment can result into uploading blobs/messages/etc into the wrong dev gcs bucket or wrong dev pubsub topic (not using emulators). And this has happened quite a few times especially when starting new projects.
I find the above solutions as hackish-workarounds. Is there a good way to ensure that we do not deploy or develop in a wrong project when working from a certain directory?

TL;DR - Not supported based on the current working directory, but there are workarounds.
Available workarounds
gcloud does not directly let you set up a configuration per working directory. Instead, you could use one of these 3 options to achieve something similar:
Specify --project, --region, --zone or the config of interest per command. This is painful but gets the job done.
Specify a different gcloud configuration directory per command (gcloud uses ~/.config/gcloud on *nix by default):
CLOUDSDK_CONFIG=/path/to/config/dir1 gcloud COMMAND
CLOUDSDK_CONFIG=/path/to/config/dir2 gcloud COMMAND
Create multiple configurations and switch between them as needed.
gcloud config configurations activate config-1 && gcloud COMMAND
Shell helpers
As all of the above options are ways to customize on the command line, aliases and/or functions in your favorite shell will also help make things easier.
For example in bash, option 2 can be implemented as follows:
function gcloud_proj1() {
CLOUDSDK_CONFIG=CLOUDSDK_CONFIG=/path/to/config/dir1 $#
}
function gcloud_proj2() {
CLOUDSDK_CONFIG=CLOUDSDK_CONFIG=/path/to/config/dir2 $#
}
gcloud_proj1 COMMAND
gcloud_proj2 COMMAND

There's a very nice way I've been using with PyCharm, I suspect you can do so with other IDEs.
You can declare the default env variables for the IDE Terminal, so when you open a new terminal gcloud recognises these env variables and sets the project and account.
No need to switch configurations between projects manually (gcloud config configurations activate ). Terminals open in other projects will inherit it's own GCP project and config from the ENV variables.

I've had this problem for years and I believe I found a decent compromise.
Create a simple script called contextual-gcloud. Note the \gcloud, fundamental for future aliasing.
🐧$ cat > contextual-gcloud
#!/bin/bash
if [ -d .gcloudconfig/ ]; then
echo "[$0] .gcloudconfig/ directory detected: using that dir for configs instead of default."
CLOUDSDK_CONFIG=./.gcloudconfig/ \gcloud "$#"
else
\gcloud "$#"
fi
Add to your .bashrc and reload / start new bash. This will fix autocompletion.
alias gcloud=contextual-gcloud
That's it! If you have a directory called that way the system will use that instead, which means you can load your configuration into source control etc.. only remember to git ignore stuff like logs, and private stuff (keys, certificates, ..).
Note: auto-completion is fixed by the alias ;)
Code: https://github.com/palladius/sakura/blob/master/bin/contextual-gcloud

These are exactly the reasons for which I highly dislike gcloud. Making command line argument mandatory and dropping configuration files support, much too error prone for my taste.
So far I'm still able to use the GAE SDK instead of Google Cloud SDK (see What is the relationship between Google's App Engine SDK and Cloud SDK?), which could be one option - basically keep doing stuff "the old way". Please note that it's no longer the recommended method.
You can find the still compatible GAE SDKs here.
For whenever the above will no longer be an option and I'll be forced to switch to the Cloud SDK my plan is to have version-controlled cheat-sheet text files in each app directory containing the exact cmds to use for running the devserver, deploy, etc for that particular project which I can just copy-paste into the terminal without fear of making mistakes. You carefully set these up once and then you just copy-paste them. As a bonus you can have different branch versions for different environments (staging/production, for example).
Actually I'm using this approach even for the GAE SDK - to prevent accidental deployment of the app-level config files to the wrong GAE app (such deployments must use cmdline arguments to specify the app in multi-service apps).
Or do the same but with environment config files and wrapper scripts instead of cheat-sheet files, if that's your preference.

Modify deployment process on elasticbeanstalk ami

I've grown tired of trying to get elastic beanstalk to run python 3.5. Instead, I want to create a custom ami which establishes a separate virtualenv for the application (with python 3.5) and knows enough to launch the application using that virtualenv.
The problem is that once I ssh into the ec2 instance in order to create my custom ami, I am left wondering where the scripts are which govern the elastic beanstalk deployment behavior.
For example, when deploying via travis to elastic beanstalk, EB knows enough to look in a specific folder for the file application.py and to execute the file using a specific virtualenv (or maybe even the shudder root python installation of the machine). It even knows to execute a pip install -r requirements. Can anyone point me to where the script(s) are which govern this behavior?
UPDATE
Please see Elastic beanstalk require python 3.5 for those referencing the .ebextensions option. So far, it has not proved able to handle this problem due to the interdependency between the EB image operating system and the python environment used to run the application.

All of the EB files can be found in /opt/elasticbeanstalk - /opt/elasticbeanstalk/hooks is probably most relevant for what you're looking for.

You can use the ebextensions to run scripts you want when starting your ami.

Pushing to an existing AWS Elastic Beanstalk application from the command line

I've used the web dashboard of Elastic Beanstalk to make an application and an environment. I know I can update that using the dashboard and uploading a zip file of my application, but I would rather use the command line to upload my application.
Apparently the correct tool for this is eb, the CLI for Elastic Beanstalk. I've installed this and attempted to use it, following the Amazon "Deploying a Flask Application to AWS Elastic Beanstalk" tutorial. However, this seems to create a completely different application to the one visible on the EB dashboard - changes made to it don't appear on the dashboard, and the application even has a different URL.
How can I use the command line to access an existing application on AWS Elastic Beanstalk?

To begin using git aws.push for your application you will have to initialize your git repository with AWS Beanstalk metadata. I'm assuming you are using git for version control (if you are not, you will have to initialize your project with git init first).
$ cd angrywhopper
$ git init #optional
$ eb init
...
$ git aws.push
Walk through wizard steps, commit your code and push the app.
Elastic Beanstalk container can be further customized by either rerunning eb init or with configuration file inside .ebextensions directory.
If eb does not support something you would like to use, have a look at AWS Elastic Beanstalk API Command Line Interface, which is more feature-rich.
More details on the configuration can be found in the following guides:
Customizing and Configuring AWS Elastic Beanstalk Environments
Customizing and Configuring a Python Container
Make sure that service region in eb wizard is the same as region you pick in dashboard dropdown.
NB: I would suggest to use temporary name in the beginning to make sure your app works as expected with the new workflow and then rename it to the original by rerunning eb init. Don't forget to terminate the temporary environment as soon as you done with the migration to avoid any unnecessary fees.

Here are the steps to use "git aws.push" with your existing ElasticBeanstalk(EB) application. (These steps would be useful, specifically, for your question and also if you had setup EB using command line from another machine and are now setting up the tools on a new machine.)
--
Before you start
You should have git installed on your system and your code should have a git repository.
Download the latest "AWS Elastic Beanstalk Command Line Tool" and get it working. Find a link to download here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/command-reference-branch-environment.html
The git aws.push command won't work yet cause your .ebextensions isn't configured. (Basically the .ebextensions stores your AWS Keys and info on EB instance to deploy to etc.)
--
Steps
Run the eb --init command. (I do this from the root of my application code directory, and it automatically picks the name of the application. Maybe you can run the command from any other location as well and specify the name manually later.)
AWS-ElasticBeanstalk-CLI-2.6.0/eb/linux/python2.7/eb (on Linux) or
AWS-ElasticBeanstalk-CLI-2.6.0/eb/windows/eb.exe (on Windows)
Enter you AWS Access Key ID and Secret Access Key
Select the environment you configured your application with (The choices are AMI Linux 64bit, Ubuntu 32bit etc.). Basically select the options that you selected while creating your first EB instance.
For Create RDS instance? [y/n]: Say n (You already have a DB instance or don't need one).
Choose "Create a default instance profile".
This would be the last step under eb --init and the script will exit.
You can find more information on the above steps here: http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python.html
--
Environment ready to use
The above steps will result in an .ebextensions directory (in ~/ I guess.)
From now on just git commit the changes in your code and run git aws.push and the application will be deployed to AWS. It's quite cool once you have it all configured.
Hope this helps. I jotted this down quickly. Let me know if you find the steps confusing, and I'll try to write it better.

Created Application in aws.amazon.com->elastic beanstalk and trying to access the application on the eb CLI:
a. When you provide eb init on console, CLI will prompt you to choose the region.
b. Make sure to choose the same region as the one you choose on the webpage.
(Note: if you don't choose the same region, it is going to take you into creating a whole new application. This was the mistake I did)
Creating application using eb CLI first locally and then trying to access the same application on the webpage.
a. $> eb console (from the app root directory, provided you did $> eb init initially)
b. You can directly login into the website and make sure to choose the same region (eg: US - N California) where you configured the app locally and you should be able to see the application you deployed.

python | heroku | how to access packages over ssh

Hi heroku python people,
I want my heroku app to access shared private libraries in my github account.
So I would like to have a requirements.txt file that looks like this ...
# requirements.txt
requests==1.2.2
-e git+ssh://git#github.com/jtushman/dict_digger.git#egg=dict_digger
And I would like it to use a ssh key that I upload with heroku keys:add or have some mechanism to get a private key from the heroku cli.
Right now I get the following error (which is I guess expected):
Host key verification failed.
It does work if I do (per #kenneth_reitz's https://stackoverflow.com/a/9136665/192791):
-e git+https://username:password#github.com/jtushman/dict_digger.git#egg=dict_digger
But it is really unworkable for me to put credentials in my requirements.txt file
Has anyone come up with a nice solution for this?
I have also posted an issue on the heroku python buildpack project here

Kenneth, the maintainer of heroku's python buildpack said the following (and I am cutting and pasting here)
I would currently recommend the way mentioned (git over https)
Using the key you have registered with heroku would be cool, but
unfortunately, you would have to provide your private key for this to
work. Quite undesirable.
However, you could also write your keys into a .ssh folder in your app
or use .profile scripts to facilitate this.
Can see the full thread here: https://github.com/heroku/heroku-buildpack-python/issues/97

I had the same issue before I wanted to use django-avatar and the version in PyPI is old and doesn't support Django 1.5 Custom User .
The simple solution is to download the package and use it as a regular app as if it was part of your project then just git add . and push it and it works !
It might not be the best idea but it just works .