We Keep Coding

Automatically create zip file with dependencies for AWS Lambda - Python

I am building a script which automatically builds AWS Lambda function. I follow this github repo as an inspiration.
However, the lambda_handler that I want to deploy is having extra dependencies, such as numpy, pandas, or even lgbm. Simple example below:
import numpy as np
def lambda_handler(event, context):
result = np.power(event['data'], 2)
response = {'result': result}
return response
example of an event and its response:
event = {'data' = [1,2,3]}
lambda_handler(event)
> {"result": [1,4,9]}
I would like to automatically add the needed layer, while creating the AWS Lambda. For that I think I need to change create_lambda_deployment_package function that is in the lambda_basic.py in the repo. What I was thinking of doing is a following:
import zipfile
import glob
import io
def create_full_lambda_deployment_package(function_file_name):
buffer = io.BytesIO()
with zipfile.ZipFile(buffer, 'w') as zipped:
# Adding all the files around the lambda_handler directory
for i in glob.glob(function_file_name):
zipped.write(i)
# Adding the numpy directory
for i in glob.glob('./venv/lib/python3.7/site-packages/numpy/*'):
zipped.write(i, f'numpy/{i[41:]}')
buffer.seek(0)
return buffer.read()
Despite the fact that lambda is created and 'numpy' folder appears my lambda environment, unfortunately this doesn't work (error cannot import name 'integer' from partially initialized module 'numpy' (most likely due to a circular import) (/var/task/numpy/__init__.py)").
How could I fix this issue? Or is there maybe another way to solve my problem?

Import data from Zoho Analytics to Python

I'm trying to connect Zoho Analytics and Python via Zoho client library here: https://www.zoho.com/analytics/api/#python-library
I downloaded the client library file but now have no idea how to use it. What I want to do is importing data from Zoho Analytics to Python and the suggested code on Zoho is:
from __future__ import with_statement
from ReportClient import ReportClient
import sys
from __future__ import with_statement
from ReportClient import ReportClient
import sys
class Sample:
LOGINEMAILID="abc#zoho.com"
AUTHTOKEN="************"
DATABASENAME="Workspace Name"
TABLENAME="Table Name"
rc = None
rc = ReportClient(self.AUTHTOKEN)
def importData(self,rc):
uri = rc.getURI(self.LOGINEMAILID,self.DATABASENAME,self.TABLENAME)
try:
with open('StoreSales.csv', 'r') as f:
importContent = f.read()
except Exception,e:
print "Error Check if file StoreSales.csv exists in
the current directory"
print "(" + str(e) + ")"
return
impResult = rc.importData(uri,"APPEND",importContent,None)
print "Added Rows :" +str(impResult.successRowCount) + " and Columns :"
+ str(impResult.selectedColCount)
obj = Sample()
obj.importData(obj.rc)
How do I make from ReportClient import ReportClient work?
Also, how does rc = ReportClient(self.AUTHTOKEN) work if self wasn't predefined?

On the site you linked, you can download a zip file containing the file Zoho/ZohoReportPythonClient/com/adventnet/zoho/client/report/python/ReportClient.py. I'm not sure why it's so deeply nested, or why most of the folders contain an __init__.py file which only has #$Id$ in it.
You'll need to extract that file, and place it somewhere where your Python interpreter can find it. For more information about where Python will look for the module (ReportClient.py), see this question: How does python find a module file if the import statement only contains the filename?
Please note that the file is Python 2 code. You'll need to use a Python 2 interpreter, or convert it to Python 3 code. Once you've got it importing properly, you can use their API reference to start writing code with it: https://css.zohostatic.com/db/api/v7_m2/docs/python/

Google Cloud Function Python installing dependency on each request

requirements.txt:
pymystem3==0.2.0
main.py:
from pymystem3 import Mystem
import json
def hello_world(request):
request_json = request.get_json()
if request_json and 'text' in request_json:
text = request_json['text']
m = Mystem()
lemmas = m.lemmatize(text)
return json.dumps(m.analyze(text), ensure_ascii=False, encoding='utf8', sort_keys=True, indent=2)
else:
return 'No text provided'
Logs in google cloud:
hello_world
032ljsdfawo
Function execution started
hello_world
032ljsdfawo
Installing mystem to /tmp/.local/bin/mystem from http://download.cdn.yandex.net/mystem/mystem-3.1-linux-64bit.tar.gz
Expand all | Collapse all {
insertId: "000000-a9eaa07b-d936-45d8-a186-927cc94246d6"
labels: {…}
logName: "projects/project_name/logs/cloudfunctions.googleapis.com%2Fcloud-functions"
receiveTimestamp: "2018-09-26T09:41:31.070927654Z"
resource: {…}
severity: "ERROR"
textPayload: "Installing mystem to /tmp/.local/bin/mystem from http://download.cdn.yandex.net/mystem/mystem-3.1-linux-64bit.tar.gz"
timestamp: "2018-09-26T09:41:19.595Z"
}
I'm new to python, so it's probably a very simple mistake. Maybe dependencies suppose to be installed before deploying the function somehow and compiled with main.py?

This is due to unfortunate behavior by the pymsystem library: instead of including everything it needs in it's Python package, it's attempting to download additional things at runtime, specifically every time you import it.
You can see that the message in your logs is coming from the library itself.
It seems like the library is attempting to determine if it has already been installed or not, but this might not be working correctly. I'd recommend filing an issue on the project's issue tracker.

python lambda can't detect packaged modules

I'm trying to create a lambda function by uploading a zip file with a single .py file at the root and 2 folders which contain the requests lib downloaded via pip.
Running the code local works file. When I zip and upload the code I very often get this error:
Unable to import module 'main': No module named requests
Sometimes I do manage to fix this, but its inconsistent and I'm not sure how I'm doing it. I'm using the following command:
in root dir zip -r upload.zip *
This is how I'm importing requests:
import requests
FYI:
1. I have attempted a number of different import methods using the exact path which have failed so I wonder if thats the problem?
2. Every time this has failed and I've been able to make it work in lambda, its involved a lot of fiddling with the zip command as I thought the problem was I was zipping the contents incorrect and hiding them behind an extra parent folder.
Looking forward to seeing the silly mistake i've been making!
Adding code snippet:
import json ##Built In
import requests ##Packaged with
import sys ##Built In
def lambda_function(event, context):
alias = event['alias']
message = event['message']
input_type = event['input_type']
if input_type == "username":
username = alias
elif input_type == "email":
username = alias.split('#',1)[0]
elif input_type is None:
print "input_type 'username' or 'email' required. Closing..."
sys.exit()
payload = {
"text": message,
"channel": "#" + username,
"icon_emoji": "<an emoji>",
"username": "<an alias>"
}
r = requests.post("<slackurl>",json=payload)
print(r.status_code, r.reason)

I got some help outside the stackoverflow loop and this seems to consistently work.
zip -r upload.zip main.py requests requests-2.9.1.dist-info

Import error on django models.py

I wrote this funcion on a utils.py located on the app direcroty:
from bm.bmApp.models import Client
def get_client(user):
try:
client = Client.objects.get(username=user.username)
except Client.DoesNotExist:
print "User Does not Exist"
return None
else:
return client
def to_safe_uppercase(string):
if string is None:
return ''
return string.upper()
Then when i use the function to_safe_uppercase on my models.py file, by importing it in this way:
from bm.bmApp.utils import to_safe_uppercase
I got the python error:
from bm.bmApp.utils import to_safe_uppercase
ImportError: cannot import name to_safe_uppercase
I got the solution for this problem when i change the import statement for:
from bm.bmApp.utils import *
But i can't understand why is this, why when i import the specific function i got the error?

You are doing what is known as a Circular import.
models.py:
from bm.bmApp.utils import to_safe_uppercase
utils.py:
from bm.bmApp.models import Client
Now when you do import bm.bmApp.models The interpreter does the following:
models.py - Line 1: try to import bm.bmApp.utils
utils.py - Line 1: try to import bm.bmApp.models
models.py - Line 1: try to import bm.bmApp.utils
utils.py - Line 1: try to import bm.bmApp.models
...
The easiest solution is to move the import inside the function:
utils.py:
def get_client(user):
from bm.bmApp.models import Client
try:
client = Client.objects.get(username=user.username)
except Client.DoesNotExist:
print "User Does not Exist"
return None
else:
return client
def to_safe_uppercase(string):
if string is None:
return ''
return string.upper()

You are creating a circular import.
utils.py
from bm.bmApp.models import Client
# Rest of the file...
models.py
from bm.bmApp.utils import to_safe_uppercase
# Rest of the file...
I would suggest your refactor your code so that you don't have a circular dependency (i.e. utils should not need to import models.py or vice versa).

I'm not sure I can explain the Import error, but I have three ideas. First, your function needs tweaking. You've used a reserved word 'string' as an argument. Consider renaming.
Second, what happens if you invoke ./manage.py shell, and do the import by hand. Does it give you any different?
Third, try deleting your pyc files to force django to recompile python code (this one is a very long shot...but worth eliminating)