I am using nltk_tokenize in an django app . To do the same I need to do nltk data download so that I can use it for stemming . I am deploying the django app on cloud through Elastic beanstalk .
Right now I have included
nltk.download('punkt')
in the my views so that the required data gets downloaded . But I am getting following error
[Errno 2] No such file or directory: '/home/wsgi/nltk_data'
What is the correct way to do so?
I am not sure what nltk_tokenize is really, but your app is running on Elastic Beanstalk by wsgi user. This is a user that doesn't have a home directory. You need to specify the path to somewhere that exists like /opt/python/current/app (your app's directory in Elastic beanstalk), or /tmp/ or what now would make sense.
Edit: Corrected directory after comment.
I achieved it by adding nltk_data files to my s3 bucket and then copying them from my s3 bucket to server by creating an eb extensions file with following command:
commands:
01_copy_nltk_data:
command: aws s3 cp s3://my_s3_bucket/nltk_data /usr/local/share/nltk_data --recursive
After that, I added NLTK_DATA environment variable in my python script and pointed it to the location of nltk_data on server.
os.environ['NLTK_DATA'] = "/usr/local/share/nltk_data"
You can use this block code:
import nltk
try:
nltk.download('punkt', download_dir='/opt/python/current/app')
except:
nltk.download('punkt')
from nltk.sentiment.vader import SentimentIntensityAnalyzer as SIA
What we are trying to do here is import nltk and then we are trying to download 'punkt' in app directory of Elastic beanstalk app since this is a wsgi app, so wsgi user doesn't have access to home directory.
Related
I am trying to run a python dash app on AWS Elastic Beanstalk service but it is missing the css, and the folder in which I have csv's as data to give results.
I could only declare one static folders but my directory looks like this.
My folder structure:
You can use a configuration file to configure static file paths and directory locations using configuration options. You can add a configuration file to your application's source bundle and deploy it during environment creation or a later deployment.
If your environment uses a platform branch based on Amazon Linux 2, use the aws:elasticbeanstalk:environment:proxy:staticfiles namespace.
The following example configuration file tells the proxy server to serve files in the statichtml folder at the path /html, and files in the staticimages folder at the path /images.
Example .ebextensions/static-files.config
option_settings:
aws:elasticbeanstalk:environment:proxy:staticfiles:
/html: statichtml
/images: staticimages
More information can be found here: Elastic BeanStalk - Serving static files.
I am using the Visual Studio 2019 Flask Web Project template. It runs in my local Python environment and is a good start (very much like the standard Asp.net) template. I created a Python Elastic Beanstalk Application to host this. I am attempting to deploy this on AWS Elastic Beanstalk. I created a repository at: https://github.com/jlongo62/Flask-Web-Project
I think one of these files (or a missing file) needs to be named
application.py(it may also need some special content). I included a
directory listing.
It appears that I need to zip this and upload it
through the portal/cli. I am not sure what zip should look like, but
I suspect requirements.txt needs to be at the root.). If AWS Toolkit
Extension cannot handle this job, it should be easy to script.
Is there a better template or sample project in GitHub ?
Is the fix something simple ?
\FlaskWebProject1\FlaskWebProject1
\FlaskWebProject1\FlaskWebProject1.pyproj
\FlaskWebProject1\FlaskWebProject1.pyproj.user
\FlaskWebProject1\requirements.txt
\FlaskWebProject1\runserver.py
\FlaskWebProject1\FlaskWebProject1\static
\FlaskWebProject1\FlaskWebProject1\templates
\FlaskWebProject1\FlaskWebProject1\views.py
\FlaskWebProject1\FlaskWebProject1\__init__.py
\FlaskWebProject1\FlaskWebProject1\__pycache__
\FlaskWebProject1\FlaskWebProject1\static\content
\FlaskWebProject1\FlaskWebProject1\static\fonts
\FlaskWebProject1\FlaskWebProject1\static\scripts
\FlaskWebProject1\FlaskWebProject1\static\content\bootstrap.css
\FlaskWebProject1\FlaskWebProject1\static\content\bootstrap.min.css
\FlaskWebProject1\FlaskWebProject1\static\content\site.css
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.eot
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.svg
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.ttf
\FlaskWebProject1\FlaskWebProject1\static\fonts\glyphicons-halflings-regular.woff
\FlaskWebProject1\FlaskWebProject1\static\scripts\bootstrap.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\bootstrap.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.intellisense.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery-1.10.2.min.map
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate-vsdoc.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.unobtrusive.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\jquery.validate.unobtrusive.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\modernizr-2.6.2.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\respond.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\respond.min.js
\FlaskWebProject1\FlaskWebProject1\static\scripts\_references.js
\FlaskWebProject1\FlaskWebProject1\templates\about.html
\FlaskWebProject1\FlaskWebProject1\templates\contact.html
\FlaskWebProject1\FlaskWebProject1\templates\index.html
\FlaskWebProject1\FlaskWebProject1\templates\layout.html
\FlaskWebProject1\FlaskWebProject1\__pycache__\views.cpython-37.pyc
\FlaskWebProject1\FlaskWebProject1\__pycache__\__init__.cpython-37.pyc
Your application works on Python 3.7 running on 64bit Amazon Linux 2/3.0.3 EB environment.
Just change runserver.py into application.py. Also you can slightly modify (port and name) its content, unless you want to customize EB environment to match your application' settings.
application.py
"""
This script runs the FlaskWebProject1 application using a development server.
"""
from os import environ
from FlaskWebProject1 import app as application
if __name__ == '__main__':
HOST = environ.get('SERVER_HOST', 'localhost')
try:
PORT = int(environ.get('SERVER_PORT', '8000'))
except ValueError:
PORT = 8000
application.run(HOST, PORT)
Zip it into app.zip (example name) with content:
application.py
FlaskWebProject1
requirements.txt
Deploy the app.zip.
With thanks to Marcin.
AWS Toolkit for Visual Studio will not support deployment or environment creation. It only applies to .Net environments.
I have updated https://github.com/jlongo62/Flask-Web-Project to reflect these notes
Create AWS Elastic Beanstalk Environment. Use the portal or CLI.
Environments > Create environment (Web server environment) > Select
Settings:
Platform: Python
Platform Branch: Python 3.7
Platform Version: 3.0.3
Add application.py with content (the port 8000 seems to be a key ingredient):
"""
This script runs the FlaskWebProject1 application using a development server.
"""
from os import environ
from FlaskWebProject1 import app as application
if __name__ == '__main__':
HOST = environ.get('SERVER_HOST', 'localhost')
try:
PORT = int(environ.get('SERVER_PORT', '8000'))
except ValueError:
PORT = 8000
application.run(HOST, PORT)
Create a zip with the following root directory. Zip file name does not matter:
application.py
FlaskWebProject1
requirements.txt
Deploy zip using the portal (or cli)
Elastic Beanstalk > Environments > Flaskwebproject1-env
click Upload and deploy
I'm running Ubuntu 13.10. Just downloaded the Python SDK for Google App Engine. I type
python appcfg.py udaproject319
and it says
ValueError: GAE_SDK_ROOT '/home/username/app/new_project_template' does not refer to a valid SDK directory
I have no idea what this means. What makes a directory an SDK directory? Here are the files I have in the folder:
appcfg.py favicon.ico main.py main.pyc
index.yaml udaproject319.yaml
After you download and unzip the SDK, you have a folder google_appengine somewhere. The sdk directory is something like /home/username/google_appengine
when you want deploy or start dev_appserver you type:
python [GAE_SDK_ROOT]/appcfg.py update [YOUR_APP_ROOT]
for example:
python /home/username/google_appengine/appcfg.py update /home/username/udaproject319
After successfully deploying a test app using the steps outlined here:
http://docs.aws.amazon.com/elasticbeanstalk/latest/dg/create_deploy_Python_flask.html
I tried to deploy my actual flask application which has the following structure:
myApp/
runServer.py
requirements.txt
myApp/
__init__.py
helpers.py
clean.sh
static/
myApp.css
handlers/
__init__.py
views.py
templates/
layout.html
viewOne.html
viewTwo.html
Where views.py contains my url mappings.
I have tried initializing the eb instance in the root directory as well as within the myApp module and git aws.push but I get the following error on the AWS dashboard:
ERROR Your WSGIPath refers to a file that does not exist. and the application does not work (404 for any path).
How can I deploy the above Flask application to elastic beanstalk?
I encountered a similar problem deploying a Flask application to EB, with a similar directory structure, and had to do 2 things:
Update my manage.py to create an object of name application, not app
import os
from application import create_app, db
from flask.ext.script import Manager, Shell
application = create_app(os.getenv('FLASK_CONFIG') or 'default')
manager = Manager(application)
Create .ebextensions/myapp.config, and define the following block to point to manage.py
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: manage.py
"aws:elasticbeanstalk:container:python:staticfiles":
"/static/": "application/static/"
This let Elastic Beanstalk find the application callable correctly.
This is described briefly at the official docs, and is described in more detail in this blog post
EDIT - see project structure below
ProjectRoot
.ebextensions
application.config
application
main
forms.py
views.py
static
templates
tests
manage.py
requirements.txt
config.py
etc, etc
Add the following to .ebextensions/<env-name>.config:
option_settings:
"aws:elasticbeanstalk:container:python":
WSGIPath: myApp/handlers/views.py
Update:
If you don't have .ebextensions directory, please create one for the project. You can find more information of what can be done regarding the container configuration in Customizing and Configuring AWS Elastic Beanstalk Environments guide.
Your WSGIPath refers to a file that does not exist.
This error appears because Beanstalk, by default, looks for application.py. Check at Beanstalk web UI, Configuration > Software Configuration, WSGIPath is mapped to application.py
Update the WSGIPath as shown in the previous replies or rename to application.py file.
As of awsebcli 3.0, you can actually edit your configuration settings to represent your WSGI path via eb config. The config command will then pull (and open it in your default command line text editor, i.e nano) an editable config based on your current configuration settings. You'll then search for WSGI and update it's path that way. After saving the file and exiting, your WSGI path will be updated automatically.
WSGI configuration was painful for me. I did changed WSCI settings using eb config command but it did not work. Below you can fix this in 5 easy steps.
1- Moved app.py function to the root of the directory (where I runned eb init command.
2- Also renamed app.py as application.py and in that initilized application as application = Flask(__name__) not app = Flask(__name__)
3- eb deploy did not worked after this (in the same project) I tried to fix config by using eb config but it was too hairy to sort it out. Delete all .extensions, .gitignore etc from your project.
4- re initialize your project on EB with eb init and follow the prompts. when deployment is done, eb open would launch your webapp (hopefully!)
When I encountered this problem it was because I was using the GUI to upload a zip of my project files. Initially I was zipping the project level directory and uploading that zip to EB.
Then I switched to simply uploading a zip of the project files themselves-ie select all files and send those to a zip-and then the GUI upload utility was able to find my application.py file without a problem because the application.py file was not in a subfolder.
Well, In my case I followed the entire process and conventions but was still getting 404. The problem was my virtual environment. I was ignoring all environment config related folders/files in my .gitignore but not in .ebignore. After creating .ebignore and ignoring all the folders/files which were not related to project code, fixed the issue.
I'm developing python application which works on aws beanstalk environment. For error handling and debugging proposes I write logs to custom lof file on the directory /var/logs/.
What should I do in order to have ability snapshot logs from Elastic beanstalk management console?
Expanding on Vadim911 (and my own comment), I solved the problem using a config file in .ebextensions. Here is the python code:
import logging
logging.basicConfig(filename='/opt/python/log/my.log', level=logging.DEBUG)
Here is the .ebextensions config file code:
files:
"/opt/python/log/my.log" :
mode: "000666"
owner: ec2-user
group: ec2-user
content: |
# Askbot log file
The contents of this file (along with other log files) is available using the Logs snapshot function on the AWS elastic beanstalk console.
If you need have ability to snapshot log files from Beanstalk management console, you should just write you log files to "/opt/python/log/" folder. Elastic beanstalk scripts use this folder for log tailing.