We Keep Coding

How do I add python libraries to an AWS lambda function for Alexa?

I was following the tutorial to create an Alexa app using Python:
Python Alexa Tutorial
I was able to successfully follow all the steps and get the app to work.I now want to modify the python code and use external libraries such as import requests
or any other libraries that I install using pip. How would I setup my lambda function to include any pip packages that I install locally on my machine?

As it is described in the Amazon official documentation link here It is as simple as just creating a zip of all the folder contents after installing the required packages in your folder where you have your python lambda code.
As Vineeth pointed above in his comment, The very first step in moving from an inline code editor to a zip file upload approach is to change your lambda function handler name under configuration settings to include the python script file name that holds the lambda handler.
lambda_handler => {your-python-script-file-name}.lambda_handler.
Other solutions like python-lambda and lambda-uploader help with simplifying the process of uploading and the most importantly LOCAL TESTING. These will save a lot of time in development.

The official documentation is pretty good. In a nutshell, you need to create a zip file of a directory containing both the code of your lambda function and all external libraries you use at the top level.
You can simulate that by deactivating your virtualenv, copying all your required libraries into the working directory (which is always in sys.path if you invoke a script on the command line), and checking whether your script still works.

You may want to look into using frameworks such as zappa which will handle packaging up and deploying the lambda function for you.
You can use that in conjunction with flask-ask to have an easier time making Alexa skills. There's even a video tutorial of this (from the zappa readme) here

To solve this particular problem we're using a library called juniper. In a nutshell, all you need to do is create a very simple manifest file that looks like:
functions:
# Name the zip file you want juni to create
router:
# Where are your dependencies located?
requirements: ./src/requirements.txt.
# Your source code.
include:
- ./src/lambda_function.py
From this manifest file, calling juni build will create the zip file artifact for you. The file will include all the dependencies you specify in the requirements.txt.
The command will create this file ./dist/router.zip. We're using that file in conjunction with a sam template. However, you can then use that zip and upload it to the console, or through the awscli.

Echoing #d3ming's answer, a framework is a good way to go at this point. Creating the deployment package manually isn't impossible, but you'll need to be uploading your packages' compiled code, and if you're compiling that code on a non-linux system, the chance of running into issues with differences between your system and the Lambda function's deployed environment are high.
You can then work around that by compiling your code on a linux machine or Docker container.. but between all that complexity you can get much more from adopting a framework.
Serverless is well adopted and has support for custom python packages. It even integrates with Docker to compile your python dependencies and build the deployment package for you.
If you're looking for a full tutorial on this, I wrote one for Python Lambda functions here.

Amazon created a repository that deals with your situation:
https://github.com/awsdocs/aws-lambda-developer-guide/tree/master/sample-apps/blank-python
The blank app is an example on how to push a lambda function that depends on requirements, with the bonus that being made by Amazon.
Everything you need to do is to follow the step by step, and update the repository based on your needs.

For some lambda POCs and fast lambda prototyping you can include and use the following function _install_packages, you can place a call to it before lambda handling function (for lambda init time package installation, if your deps need less than 10 seconds to install) or place the call at the beginning of the lambda handler (this will call the function exactly once at the first lambda event). Given pip install options included, packages to be installed must provide binary installable versions for manylinux.
_installed = False
def _install_packages(*packages):
global _installed
if not _installed:
import os
import sys
import time
_started = time.time()
os.system("mkdir -p /tmp/packages")
_packages = " ".join(f"'{p}'" for p in packages)
print("INSTALLED:")
os.system(
f"{sys.executable} -m pip freeze --no-cache-dir")
print("INSTALLING:")
os.system(
f"{sys.executable} -m pip install "
f"--no-cache-dir --target /tmp/packages "
f"--only-binary :all: --no-color "
f"--no-warn-script-location {_packages}")
sys.path.insert(0, "/tmp/packages")
_installed = True
_ended = time.time()
print(f"package installation took: {_ended - _started:.2f} sec")
# usage example before lambda handler
_install_packages("pymssql", "requests", "pillow")
def lambda_handler(event, context):
pass # lambda code
# usage example from within the lambda handler
def lambda_handler(event, context):
_install_packages("pymssql", "requests", "pillow")
pass # lambda code
Given examples install python packages: pymssql, requests and pillow.
An example lambda that installs requests and then calls ifconfig.me to obtain it's egress IP address.
import json
_installed = False
def _install_packages(*packages):
global _installed
if not _installed:
import os
import sys
import time
_started = time.time()
os.system("mkdir -p /tmp/packages")
_packages = " ".join(f"'{p}'" for p in packages)
print("INSTALLED:")
os.system(
f"{sys.executable} -m pip freeze --no-cache-dir")
print("INSTALLING:")
os.system(
f"{sys.executable} -m pip install "
f"--no-cache-dir --target /tmp/packages "
f"--only-binary :all: --no-color "
f"--no-warn-script-location {_packages}")
sys.path.insert(0, "/tmp/packages")
_installed = True
_ended = time.time()
print(f"package installation took: {_ended - _started:.2f} sec")
# usage example before lambda handler
_install_packages("requests")
def lambda_handler(event, context):
import requests
return {
'statusCode': 200,
'body': json.dumps(requests.get('http://ifconfig.me').content.decode())
}
Given single quote escaping is considered when building pip's command line, you can specify a version in a package spec like this pillow<9, the former will install most recent 8.X.X version of pillow.

I too struggled for a while with this. The after deep diving into aws resources I got to know the lambda function on aws runs locally on a a linux. And it's very important to have the the python package version which matches with the linux version.
You may find more information on this on :
https://aws.amazon.com/lambda/faqs/
Follow the steps to download the version.
1. Find the .whl image of the package from pypi and download it on you local.
2. Zip the packages and add them as layers in aws lambda
3. Add the layer to the lambda function.
Note: Please make sure that version you're trying to install python package matches the linux os on which the aws lambda performs computes tasks.
References :
https://pypi.org/project/Pandas3/#files

A lot of python libraries can be imported via Layers here: https://github.com/keithrozario/Klayers, or your can use a framework like serverless that has plugins to package packages directly into your artifact.

how to control the import paths in Python?

I am new to Python. I am at a company where they built a large system in Python. They use a proprietary system to manage the paths when the system is running, but now I have been asked to build a standalone script that interacts with some of the code in their system. Sadly, my standalone script won't be running under the path-manager they use, so I need to figure out the paths on my own.
So, for instance, I have this line:
from hark.tasks import REPLY_LINE
This is actually copied from some of their older code. In this case, the script can find hark, but hark has an __init__.py file, and that is where the problems start. So I get this:
meg/src/python2/hark/hark/__init__.py in <module>()
5 from flask import jsonify, render_template, request
6 import jinja2
----> 7 import logbook, logbook.compat
8
9 from healthhark.context import Ghost, g
The project that they built actually includes logbook 3 times. If I do:
find . -name "*logbook*"
I see:
meg/zurge/opt/python2.7/lib/python2.7/site-packages/logbook
meg/zurge/opt/python2.7-hark/lib/python2.7/site-packages/logbook
meg/zurge/opt/python3.4/lib/python3.4/site-packages/logbook
Like I said, they have a proprietary path manager that usually tells each piece of code where it can find the packages that it should include, but I am building a standalone app.
I don't know much about Python, but I am wondering if their is an idiomatic and Pythonic way of including packages that are in such distant directories?
And, before anyone suggests pip install, we don't rely on global installs at all.

The best solution would probably be virtualenv or virtualenvwrapper. This would allow you to define an environment which contains all of the libraries that your script requires. This would not be global.
This can be done as follows:
Create a requirements.txt file defining the required libraries
Install pip and virtualenvwrapper
source /usr/local/bin/virtualenvwrapper.sh
workon hark_task_script_env || ( mkvirtualenv hark_task_script_env && pip install -r requirements.txt )
python your-script.py
The virtualenvwrapper installs the environment into a folder in your home directory. The plain virtualenv library installs the environment into a folder in the project. Other than that they are equivalent.
I would really recommend using them over a proprietry package manager. If you must use the proprietry package manager then it is reasonable to have access to the package loader for it!
If this really isn't satisfactory then you can hack a package loader as follows (assuming you use a *nix system). This is a shell script written in zsh. Hopefully it is clear enough for you to rewrite it in a supported shell if your system does not have that available:
#!/bin/zsh
setopt extended_glob
function find_package_init_files () {
locate __init__.py
}
# Get the containing folder of a file or a folder
function file_or_folder_to_parent_folder () {
while read file_or_folder
do
echo ${file_or_folder:h}
done
}
# Exclude folders where the parent folder also has an __init__.py file in it
function exclude_inner_packages () {
while read init_file_folder
do
init_file_parent_folder=${init_file_folder:h}
if [ ! -e ${init_file_parent_folder}/__init__.py ]
then
echo ${init_file_folder}
fi
done
}
# This produces an array of all folders
# that contain at least one python package
function get_distinct_python_package_folders () {
find_package_init_files |
file_or_folder_to_parent_folder |
exclude_inner_packages |
file_or_folder_to_parent_folder |
sort |
uniq
}
PYTHONPATH=${(j/:/)$(get_distinct_python_package_folders)} YOUR_SCRIPT_HERE
You may well need to update this script to put the default python path first, and remember that this is an incredibly clumsy approach. If there are multiple versions of libraries installed on a system then the one that you may end up using will be ill defined.
Using a proper package manager, even if that is the proprietry one, is the best way.

What is the cleanest way to add a directory of third-party packages to the beginning of the Python path?

My context is appengine_config.py, but this is really a general Python question.
Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:
dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)
For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):
sys.path.insert(1, dirpath)
However, that won't work if any of the packages in that directory are namespace packages.
To support namespace packages we can instead use:
site.addsitedir(dirpath)
But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.
The solution I have so far is this bit of code which I'd really like to simplify:
sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)
Is there a cleaner or more Pythonic way of accomplishing this?

For this answer I assume you know how to use setuptools and setup.py.
Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:
import os
import sys
if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
# If we are unittesting, fake the non-existence of appengine_config.
# The error message of the import error is handled by gae and must
# exactly match the proper string.
raise ImportError('No module named appengine_config')
# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
for lib in os.listdir(lib_dir):
if lib.endswith('.egg'):
lib = os.path.join(lib_dir, lib)
# Insert to override default libraries such as webob 1.1.1.
sys.path.insert(0, lib)
And this piece of code in setup.cfg:
[develop]
install-dir = libs
always-copy = true
If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.
We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.

You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.
I do want to address something else that you said that I think is misleading:
My context is appengine_config.py, but this is really a general Python
question.
The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.
The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:
Create a virtualenv
Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
Install your dependencies into the virtualenv with pip
Install your project, itself, into the virtualenv with pip and the -e/--editable flag.
Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.
If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.

Code coverage for a python google-app-engine site?

I used to be able to get code coverage for unit testing a Google App Engine test via a commandline like:
coverage run --omit=/Applications --source=../mycode --branch /usr/local/bin/dev_appserver.py ...
[This uses Ned Batchelder's coverage.py] But, after recently updating to the latest SDK (after a long spell of not working on the code), I find that this does not work any more. The server process must run the application code in a subprocess or somesuch.
I tried following this http://nedbatchelder.com/code/coverage/subprocess.html#subprocess
I see another semi-recent question about this with a comment that suggests that coverage.py just won't work. Getting coverage with dev_appserver.py excludes my project files
I've spent a few frustrating hours googling around and trying some things with no luck. So...is this still impossible? Has anyone gotten code coverage to work in any manner? Is there some other tool that can figure out code coverage?

A short term fix might be to run the old dev_appserver.py
https://developers.google.com/appengine/docs/python/tools/old_devserver#Running_the_Old_Development_Web_Server

dragonx's suggestion to use old_dev_appserver.py worked well for me. More specifically, here's what I did using App Engine 1.9.6, coverage 3.7.1, and Python 2.7 on MacOS X 10.9.3:
MyAppDir is the directory containing app.yaml.
--omit is optional. You may well not need it. I had already moved my test code out of MyAppDir because I did not want appcfg.py to upload it.
--branch is optional but useful.
old_dev_appserver.py ships (for now) with App Engine. There is no need to download or install a copy.
# One time:
sudo pip install coverage
# Start the server:
APP=MyAppDir
coverage run \
--source=$APP \
--omit='$APP/exclude/*' \
--branch \
/usr/local/bin/old_dev_appserver.py \
$APP
# Run your tests in a separate tab. In my case I use this command:
webdriver/system_tests.py
# Kill the server with Control-C once the tests are finished.
# Display a quick text summary:
coverage report -m
# Generate and open an HTML report linking to line by line coverage:
coverage html
open htmlcov/index.html
My relatively straightforward app (email, full text search, ndb, urlfetch, webapp2) did not need any changes to work with old_dev_appserver. I did remove the flags I passed to dev_appserver. I was able to live without them. --port is supported if you need it, as are a few others.
If you'd like to see code coverage support in future versions of dev_appserver.py please vote up Add support for code coverage tests and some documentation, formerly https://code.google.com/p/googleappengine/issues/detail?id=4936.

How to bundle Python dependancies in IronWorker?

I'm writing a simple IronWorker in Python to do some work with the AWS API.
To do so I want to use the boto library which is distributed via PyPi repository. The boto library is not installed by default in the IronWorker runtime environment.
How can I bundle the boto library dependancy with my IronWorker code?
Ideally I'm hoping I can use something like the gem dependancy bundling available for Ruby IronWorkers - i.e in myRuby.worker specify
gemfile '../Gemfile', 'common', 'worker' # merges gems from common and worker groups
In the Python Loggly sample, I see that the hoover library is used:
#here we have to include hoover library with worker.
hoover_dir = os.path.dirname(hoover.__file__)
shutil.copytree(hoover_dir, worker_dir + '/loggly') #copy it to worker directory
However, I can't see where/how you specify which hoover library version you want, or where to download it from.
What is the official/correct way to use 3rd party libraries in Python IronWorkers?

Newer iron_worker version has native support of pip command.
So, you need:
runtime "python"
exec "something.py"
pip "boto"
pip "someotherpip"
full_remote_build true

[edit]We've worked on our toolset a bit since this answer was written and accepted. The answer from my colleague below is the recommended course moving forward.[/edit]
I wrote the Python client library for IronWorker. I'm also employed by Iron.io.
If you're using the Python client library, the easiest (and recommended) way to do this is to just copy over the library's installed folder, and include it when uploading the package. That's what the Python Loggly sample is doing above. As you said, that doesn't specify a version or where to download the library from, because it doesn't care. It just takes the one installed on your system and uses it. Whatever you get when you enter "import boto" on your local machine is what would be uploaded.
The other option is using our CLI to upload your worker, with a .worker file.
To do this, here's what you'd need to do:
Create a botoworker.worker file:
runtime "binary"
build 'pip install --install-option="--prefix=`pwd`/pips" boto'
file 'botoworker.py'
exec "botoworker.sh"
That second line is the pip command that will be run to install the dependency. You can modify it like you would any pip command run from the command line. It's going to execute that command on the worker during the "build" phase, so it's only executed once instead of every time you run a task.
The third line should be changed to the Python file you want to run--it's your Python worker file. Here's the one we used to test this:
import boto
If you save that as botoworker.py, the above should work without any modification. :)
The fourth line is a shell script that's going to actually run your worker. I've included the one we used below. Just save it as botoworker.sh, and you won't have to worry about modifying the .worker file above.
PYTHONPATH="$HOME/pips/lib/python2.7/site-packages:$PYTHONPATH" python botoworker.py "$#"
You'll notice it refers to your Python file--if you don't name your Python file botoworker.py, remember to change it here, too. All this does is set your PYTHONPATH to include the installed library, and then runs your Python file.
To upload this, just make sure you have the CLI installed (gem install iron_worker_ng, making sure your Ruby version is 1.9.3 or higher) and then run "iron_worker upload botoworker" in your shell, from the same directory your botoworker.worker file is in.
Hope this helps!