We Keep Coding

How to declare extra modules (static files) in python dash app hosted on AWS Elastic Beanstalk

I am trying to run a python dash app on AWS Elastic Beanstalk service but it is missing the css, and the folder in which I have csv's as data to give results.
I could only declare one static folders but my directory looks like this.
My folder structure:

You can use a configuration file to configure static file paths and directory locations using configuration options. You can add a configuration file to your application's source bundle and deploy it during environment creation or a later deployment.
If your environment uses a platform branch based on Amazon Linux 2, use the aws:elasticbeanstalk:environment:proxy:staticfiles namespace.
The following example configuration file tells the proxy server to serve files in the statichtml folder at the path /html, and files in the staticimages folder at the path /images.
Example .ebextensions/static-files.config
option_settings:
aws:elasticbeanstalk:environment:proxy:staticfiles:
/html: statichtml
/images: staticimages
More information can be found here: Elastic BeanStalk - Serving static files.

How to deploy Visual Studio Flask app to Elastic Beanstalk

I am using the Visual Studio 2019 Flask Web Project template. It runs in my local Python environment and is a good start (very much like the standard Asp.net) template. I created a Python Elastic Beanstalk Application to host this. I am attempting to deploy this on AWS Elastic Beanstalk. I created a repository at: https://github.com/jlongo62/Flask-Web-Project
I think one of these files (or a missing file) needs to be named
application.py(it may also need some special content). I included a
directory listing.
It appears that I need to zip this and upload it
through the portal/cli. I am not sure what zip should look like, but
I suspect requirements.txt needs to be at the root.). If AWS Toolkit
Extension cannot handle this job, it should be easy to script.
Is there a better template or sample project in GitHub ?
Is the fix something simple ?
\FlaskWebProject1\FlaskWebProject1
\FlaskWebProject1\FlaskWebProject1.pyproj
\FlaskWebProject1\FlaskWebProject1.pyproj.user
\FlaskWebProject1\requirements.txt
\FlaskWebProject1\runserver.py
\FlaskWebProject1\FlaskWebProject1\static
\FlaskWebProject1\FlaskWebProject1\templates
\FlaskWebProject1\FlaskWebProject1\views.py
\FlaskWebProject1\FlaskWebProject1\__init__.py
\FlaskWebProject1\FlaskWebProject1\__pycache__
\FlaskWebProject1\FlaskWebProject1\static\content
\FlaskWebProject1\FlaskWebProject1\static\fonts
\FlaskWebProject1\FlaskWebProject1\static\scripts
\FlaskWebProject1\FlaskWebProject1\static\content\bootstrap.css
\FlaskWebProject1\FlaskWebProject1\static\content\bootstrap.min.css
\FlaskWebProject1\FlaskWebProject1\static\content\site.css
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.eot
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.svg
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.ttf
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.woff
\FlaskWebProject1\FlaskWebProject1\static\scripts\bootstrap.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\bootstrap.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.intellisense.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.min.map
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate-vsdoc.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.unobtrusive.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.unobtrusive.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\modernizr-2.6.2.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\respond.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\respond.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\_references.js
\FlaskWebProject1\FlaskWebProject1\templates\about.html
\FlaskWebProject1\FlaskWebProject1\templates\contact.html
\FlaskWebProject1\FlaskWebProject1\templates\index.html
\FlaskWebProject1\FlaskWebProject1\templates\layout.html
\FlaskWebProject1\FlaskWebProject1\__pycache__\views.cpython-37.pyc
\FlaskWebProject1\FlaskWebProject1\__pycache__\__init__.cpython-37.pyc

Your application works on Python 3.7 running on 64bit Amazon Linux 2/3.0.3 EB environment.
Just change runserver.py into application.py. Also you can slightly modify (port and name) its content, unless you want to customize EB environment to match your application' settings.
application.py
"""
This script runs the FlaskWebProject1 application using a development server.
"""
from os import environ
from FlaskWebProject1 import app as application
if __name__ == '__main__':
HOST = environ.get('SERVER_HOST', 'localhost')
try:
PORT = int(environ.get('SERVER_PORT', '8000'))
except ValueError:
PORT = 8000
application.run(HOST, PORT)
Zip it into app.zip (example name) with content:
application.py
FlaskWebProject1
requirements.txt
Deploy the app.zip.

With thanks to Marcin.
AWS Toolkit for Visual Studio will not support deployment or environment creation. It only applies to .Net environments.
I have updated https://github.com/jlongo62/Flask-Web-Project to reflect these notes
Create AWS Elastic Beanstalk Environment. Use the portal or CLI.
Environments > Create environment (Web server environment) > Select
Settings:
Platform: Python
Platform Branch: Python 3.7
Platform Version: 3.0.3
Add application.py with content (the port 8000 seems to be a key ingredient):
"""
This script runs the FlaskWebProject1 application using a development server.
"""
from os import environ
from FlaskWebProject1 import app as application
if __name__ == '__main__':
HOST = environ.get('SERVER_HOST', 'localhost')
try:
PORT = int(environ.get('SERVER_PORT', '8000'))
except ValueError:
PORT = 8000
application.run(HOST, PORT)
Create a zip with the following root directory. Zip file name does not matter:
application.py
FlaskWebProject1
requirements.txt
Deploy zip using the portal (or cli)
Elastic Beanstalk > Environments > Flaskwebproject1-env
click Upload and deploy

Not able to install nltk data on django app on elastic beanstalk

I am using nltk_tokenize in an django app . To do the same I need to do nltk data download so that I can use it for stemming . I am deploying the django app on cloud through Elastic beanstalk .
Right now I have included
nltk.download('punkt')
in the my views so that the required data gets downloaded . But I am getting following error
[Errno 2] No such file or directory: '/home/wsgi/nltk_data'
What is the correct way to do so?

I am not sure what nltk_tokenize is really, but your app is running on Elastic Beanstalk by wsgi user. This is a user that doesn't have a home directory. You need to specify the path to somewhere that exists like /opt/python/current/app (your app's directory in Elastic beanstalk), or /tmp/ or what now would make sense.
Edit: Corrected directory after comment.

I achieved it by adding nltk_data files to my s3 bucket and then copying them from my s3 bucket to server by creating an eb extensions file with following command:
commands:
01_copy_nltk_data:
command: aws s3 cp s3://my_s3_bucket/nltk_data /usr/local/share/nltk_data --recursive
After that, I added NLTK_DATA environment variable in my python script and pointed it to the location of nltk_data on server.
os.environ['NLTK_DATA'] = "/usr/local/share/nltk_data"

You can use this block code:
import nltk
try:
nltk.download('punkt', download_dir='/opt/python/current/app')
except:
nltk.download('punkt')
from nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA
What we are trying to do here is import nltk and then we are trying to download 'punkt' in app directory of Elastic beanstalk app since this is a wsgi app, so wsgi user doesn't have access to home directory.

Print to file Beanstalk Worker Tier (Python)

I asked something similar to this question and I haven't gotten any responses that help. So, I have decided to simplify things as much as I can with the following:
I have developed a python flask application and deployed to a beanstalk worker tier python environment. The issue is I can't figure out how to print or log or write anything anywhere. I need to debug this application and the only way I know how to do that is by printing to either the console or a log file to see exactly what is going on. When I run the application locally I can print to the console, write to files, and log with zero problems, it is just when I deploy it to the beanstalk environment that nothing happens. I have SSHed into the ec2 instance where I have application deployed and searched practically every file and I find that nothing was written by my python script anywhere.
This question probably seems absolutely stupid but can someone please provide me with an example of a python flask application that will run on a beanstalk worker environment that just prints "Hello World" to some file that I can find on the ec2 instance? Please include what should be written the requirements.txt file and any *.config files in the .ebextensions folder.
Thank You

Here is another simple python app that you can try. The one in the blog will work as well but this shows a minimal example of an app that prints messages received from SQS to a file on the EC2 instance.
Your app source folder should have the following files:
application.py
import os
import time
import flask
import json
application = flask.Flask(__name__)
start_time = time.time()
counter_file = '/tmp/worker_role.tmp'
#application.route('/', methods=['GET', 'POST'])
def hello_world():
if flask.request.method == 'POST':
with open(counter_file, 'a') as f:
f.write(flask.request.data + "\n")
return flask.Response(status=200)
if __name__ == '__main__':
application.run(host='0.0.0.0', debug=True)
requirements.txt
Flask==0.9
Werkzeug==0.8.3
.ebextensions/01-login.config
option_settings:
- namespace: aws:autoscaling:launchconfiguration
option_name: EC2KeyName
value: your-key-name
Launch a worker tier 1.1 environment with a Python 2.7 solution stack. I tested with (64bit Amazon Linux 2014.03 v1.0.4 running Python 2.7).
Wait for the environment to go green. After it goes green click on the queue URL as visible in the console. This will take you to the SQS console page. Right click on the queue and click on "Send a message". Then type the following message: {"hello" : "world"}.
SSH to the EC2 instance and open the file /tmp/worker_role.tmp. You should be able to see your message in this file.
Make sure you have IAM policies correctly configured for using Worker Role environments.
For more information on IAM policies refer this answer: https://stackoverflow.com/a/23942498/161628

There is a python+flask on beanstalk example on AWS Application Management Blog:
http://blogs.aws.amazon.com/application-management/post/Tx1Y8QSQRL1KQZC/Elastic-Beanstalk-Video-Tutorial-Worker-Tier
http://blogs.aws.amazon.com/application-management/post/Tx36JL4GPZR4G98/A-Sample-App-For-Startups
For the logging issues, i'd suggest:
Check your /var/log/eb-cfn-init.log (and other log files in this directory), if a .config command is failing you will see which and why there.
In your .config commands, output messages to a different log file so you see exactly where your bootstrap failed in your own file.
Add you application log file to EB Log Snapshots (/opt/elasticbeanstalk/tasks/taillogs.d/) and EB S3 log rotation (/opt/elasticbeanstalk/tasks/publishlogs.d/). See other files in these directories for examples.

How to deploy structured Flask app on AWS elastic beanstalk

After successfully deploying a test app using the steps outlined here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html
I tried to deploy my actual flask application which has the following structure:
myApp/
runServer.py
requirements.txt
myApp/
__init__.py
helpers.py
clean.sh
static/
myApp.css
handlers/
__init__.py
views.py
templates/
layout.html
viewOne.html
viewTwo.html
Where views.py contains my url mappings.
I have tried initializing the eb instance in the root directory as well as within the myApp module and git aws.push but I get the following error on the AWS dashboard:
ERROR Your WSGIPath refers to a file that does not exist. and the application does not work (404 for any path).
How can I deploy the above Flask application to elastic beanstalk?

I encountered a similar problem deploying a Flask application to EB, with a similar directory structure, and had to do 2 things:
Update my manage.py to create an object of name application, not app
import os
from application import create_app, db
from flask.ext.script import Manager, Shell
application = create_app(os.getenv('FLASK_CONFIG') or 'default')
manager = Manager(application)
Create .ebextensions/myapp.config, and define the following block to point to manage.py
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: manage.py
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "application/static/"
This let Elastic Beanstalk find the application callable correctly.
This is described briefly at the official docs, and is described in more detail in this blog post
EDIT - see project structure below
ProjectRoot
.ebextensions
application.config
application
main
forms.py
views.py
static
templates
tests
manage.py
requirements.txt
config.py
etc, etc

Add the following to .ebextensions/<env-name>.config:
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: myApp/handlers/views.py
Update:
If you don't have .ebextensions directory, please create one for the project. You can find more information of what can be done regarding the container configuration in Customizing and Configuring AWS Elastic Beanstalk Environments guide.

Your WSGIPath refers to a file that does not exist.
This error appears because Beanstalk, by default, looks for application.py. Check at Beanstalk web UI, Configuration > Software Configuration, WSGIPath is mapped to application.py
Update the WSGIPath as shown in the previous replies or rename to application.py file.

As of awsebcli 3.0, you can actually edit your configuration settings to represent your WSGI path via eb config. The config command will then pull (and open it in your default command line text editor, i.e nano) an editable config based on your current configuration settings. You'll then search for WSGI and update it's path that way. After saving the file and exiting, your WSGI path will be updated automatically.

WSGI configuration was painful for me. I did changed WSCI settings using eb config command but it did not work. Below you can fix this in 5 easy steps.
1- Moved app.py function to the root of the directory (where I runned eb init command.
2- Also renamed app.py as application.py and in that initilized application as application = Flask(__name__) not app = Flask(__name__)
3- eb deploy did not worked after this (in the same project) I tried to fix config by using eb config but it was too hairy to sort it out. Delete all .extensions, .gitignore etc from your project.
4- re initialize your project on EB with eb init and follow the prompts. when deployment is done, eb open would launch your webapp (hopefully!)

When I encountered this problem it was because I was using the GUI to upload a zip of my project files. Initially I was zipping the project level directory and uploading that zip to EB.
Then I switched to simply uploading a zip of the project files themselves-ie select all files and send those to a zip-and then the GUI upload utility was able to find my application.py file without a problem because the application.py file was not in a subfolder.

Well, In my case I followed the entire process and conventions but was still getting 404. The problem was my virtual environment. I was ignoring all environment config related folders/files in my .gitignore but not in .ebignore. After creating .ebignore and ignoring all the folders/files which were not related to project code, fixed the issue.