i work on a google cloud environment where i don't have internet access. I'm trying to launch a dataflow job passing it the sdk like this:
python wordcount.py --no_use_public_ip --sdk_location "<basepath>/dist/package-import-0.0.2.tar.gz"
I generated package-import-0.0.2.tar.gz with this setup.py
import setuptools
setuptools.setup(
name='package-import',
version='0.0.2',
install_requires=[
'apache-beam==2.43.0',
'cachetools==4.2.4',
'certifi==2022.12.7',
'charset-normalizer==2.1.1',
'cloudpickle==2.2.0',
'crcmod==1.7',
'dill==0.3.1.1',
'docopt==0.6.2',
'fastavro==1.7.0',
'fasteners==0.18',
'google-api-core==2.11.0',
'google-apitools==0.5.31',
'google-auth==2.15.0',
'google-auth-httplib2==0.1.0',
'google-cloud-bigquery==3.4.1',
'google-cloud-bigquery-storage==2.13.2',
'google-cloud-bigtable==1.7.3',
'google-cloud-core==2.3.2',
'google-cloud-datastore==1.15.5',
'google-cloud-dlp==3.10.0',
'google-cloud-language==1.3.2',
'google-cloud-pubsub==2.13.11',
'google-cloud-pubsublite==1.6.0',
'google-cloud-recommendations-ai==0.7.1',
'google-cloud-spanner==3.26.0',
'google-cloud-videointelligence==1.16.3',
'google-cloud-vision==1.0.2',
'google-crc32c==1.5.0',
'google-resumable-media==2.4.0',
'googleapis-common-protos==1.57.1',
'grpc-google-iam-v1==0.12.4',
'grpcio==1.51.1',
'grpcio-status==1.51.1',
'hdfs==2.7.0',
'httplib2==0.20.4',
'idna==3.4',
'numpy==1.22.4',
'oauth2client==4.1.3',
'objsize==0.5.2',
'orjson==3.8.3',
'overrides==6.5.0',
'packaging==22.0',
'proto-plus==1.22.1',
'protobuf==3.20.3',
'pyarrow==9.0.0',
'pyasn1==0.4.8',
'pyasn1-modules==0.2.8',
'pydot==1.4.2',
'pymongo==3.13.0',
'pyparsing==3.0.9',
'python-dateutil==2.8.2',
'pytz==2022.7',
'regex==2022.10.31',
'requests==2.28.1',
'rsa==4.9',
'six==1.16.0',
'sqlparse==0.4.3',
'typing-extensions==4.4.0',
'urllib3==1.26.13',
'zstandard==0.19.0'
],
packages=setuptools.find_packages(),
)
but in dataflow log worker i have this error: Could not install Apache Beam SDK from a wheel: could not find a Beam SDK wheel among staged files, proceeding to install SDK from source tarball.
And then he tries to download it but since he doesn't have internet he can't
my biggest problem is that the google cloud environment doesn't access the internet so dataflow can't download what it needs. Do you know of a way to pass it an sdk_location?
If you don't have access to internet from your environement, I thought on a solution based on a Docker image.
- Workers
Dataflow Python can use a Docker image in the execution phase while creating the workers.
In this image Docker you can install all the needed packages in the container and publish it to Container Registry, example :
FROM apache/beam_python3.8_sdk:2.44.0
# Pre-built python dependencies
RUN pip install lxml
# Pre-built other dependencies
RUN apt-get update \
&& apt-get dist-upgrade \
&& apt-get install -y --no-install-recommends ffmpeg
# Set the entrypoint to the Apache Beam SDK launcher.
ENTRYPOINT ["/opt/apache/beam/boot"]
In the Dataflow job, you have to specify 2 program arguments to use the image :
experiments
sdk_container_image
python -m apache_beam.examples.wordcount \
--input=INPUT_FILE \
--output=OUTPUT_FILE \
--project=PROJECT_ID \
--region=REGION \
--temp_location=TEMP_LOCATION \
--runner=DataflowRunner \
--disk_size_gb=DISK_SIZE_GB \
--experiments=use_runner_v2 \
--sdk_container_image=$IMAGE_URI
- Runner from your google environment
Your Google environment executing the job needs also to have the packages installed in order to be able to instantiate the job.
You need to find a way to install the packages in the machines and your environment. If you can use the same Docker image used for Dataflow workers and execution phase, it would be perfect.
I solved using an internal proxy that allowed me to access the internet. In the command added this --no_use_public_ip and i set no_proxy="metadata.google.internal,www.googleapis.com,dataflow.googleapis.com,bigquery.googleapis.com" thanks
I've been trying to configure an Amplify project with a Python based Lambda backend API.
I have followed the tutorials by creating an API through the AWS CLI and installing all the dependencies through pipenv.
When I cd into the function's directory, my Pipfile looks like this:
name = "pypi"
url = "https://pypi.python.org/simple"
verify_ssl = true
[dev-packages]
[packages]
src = {editable = true, path = "./src"}
flask = "*"
flask-cors = "*"
aws-wsgi = "*"
boto3 = "*"
[requires]
python_version = "3.8"
And when I run amplify push everything works and the Lambda Function gets created successfully.
Also, when I run the deploy pipeline from the Amplify Console, I see in the build logs that my virtual env is created and my dependencies are downloaded.
Something else that was done based on github issues (otherwise build would definitely fail) was adding the following to amplify.yml:
backend:
phases:
build:
commands:
- ln -fs /usr/local/bin/pip3.8 /usr/bin/pip3
- ln -fs /usr/local/bin/python3.8 /usr/bin/python3
- pip3 install --user pipenv
- amplifyPush --simple
Unfortunately, from the Lambda's logs (both dev and prod), I see that it fails importing every dependency that was installed through Pipenv. I added the following in index.py:
import os
os.system('pip list')
And saw that NONE of my dependencies were listed so I was wondering if the Lambda was running through the virtual env that was created, or was just using the default Python.
How can I make sure that my Lambda is running the virtualenv as defined in the Pipfile?
Lambda functions do not run in a virtualenv. Amplify uses pipenv to create a virtualenv and download the dependencies. Then Amplify packages those dependencies, along with the lambda code, into a zip file which it uploads to AWS Lambda.
Your problem is either that the dependencies are not packaged with your function or that they are packaged with a bad directory structure. You can download the function code to see exactly how the packaging went.
I'm trying to use serverless framework with a python project.
I created a hello world example that I run in offline mode. It works well but when I try to import a python package I get ModuleNotFoundError.
Here is my serverless.yaml file:
service: my-test
frameworkVersion: "3"
provider:
name: aws
runtime: python3.8
functions:
hello:
handler: lambdas.hello.hello
events:
- http:
path: /hello
method: get
plugins:
- serverless-python-requirements
- serverless-offline
In lambdas.hello.py:
import json
import pandas
def hello(event, context):
body = {
"message": 'hello world',
}
response = {"statusCode": 200, "body": json.dumps(body)}
return response
In my Pipfile:
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true
[dev-packages]
[packages]
pandas = "*"
[requires]
python_version = "3.8"
To run it, I use the command $ sls offline start
Then When I query on postman http://localhost:3000/dev/hello I get the error ModuleNotFoundError.
If I remove the line import pandasin hello.py file, it works.
I don't understand why I get this error as serverless-python-requirements is supposed to check the pipfile and pandas is in my pipfile.
How can I use pandas (or any other python package) in my lambdas with serverless framework in offline mode ?
The serverless-python-requirements plugin is used to bundle your dependencies and package them for deployment. This only comes to effect when you run sls deploy.
From the plugin page -
The plugin will now bundle your python dependencies specified in your requirements.txt or Pipfile when you run sls deploy
Read more about python packaging here - https://www.serverless.com/blog/serverless-python-packaging
Since you are running your service locally, this plugin will not be used.
Your dependencies need to be installed locally.
perform the below steps to make it work -
Create a virtual environment in you serverless directory.
install the plugin serverless plugin install -n serverless-offline
install pandas using pip
run sls offline start
Your lambda function don't have the panda module installed
You need to use the serverless-python-requirements plugin : https://www.serverless.com/plugins/serverless-python-requirements. To use it you need docker on your machine and to create a requirement.txt file in your service with the packages you need in your lambda
I am trying to deploy the python serverless application on AWS.
I follow pretty straightforward tutorial and I do following steps:
Install serverless
npm install -g serverless
Generate template project sls create --template aws-python3 --name sls-demo --path sls-demo
My handler.py file looks as follows:
import json
def hello(event, context):
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"input": event
}
response = {
"statusCode": 200,
"body": json.dumps(body)
}
return response
And my serverless.yml configuration file looks as follows:
service: sls-demo
frameworkVersion: '2'
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
region: eu-central-1
functions:
hello:
handler: handler.hello
I have already installed aws cli on my machine, configured it with aws credentials and when I run the deployment command sls deploy it finishes successfully.
I test the lambda function with following command sls invoke --function hello and the result returns successfully:
{
"statusCode": 200,
"body": "{\"message\": \"Go Serverless v1.0! Your function executed successfully!\", \"input\": {}}"
}
Now I want to introduce some extra dependencies in my lambda function, dockerize it and deploy using the serverless-python-requirements plugin.
For this I do following steps:
Create virtual environment python -m venv ./venv
Activate virtual environment source venv/bin/activate
Install numpy dependency pip install numpy
Freeze python dependencies pip freeze > requirements.txt
Install serverless-python-requirements plugin sls plugin install -n serverless-python-requirements
My updated handler.py file looks as follows:
import json
import numpy
def hello(event, context):
array = numpy.arange(15).reshape(3, 5)
body = {
"message": "Go Serverless v1.0! Your function executed successfully!",
"input": event,
"array": array
}
response = {
"statusCode": 200,
"body": json.dumps(body)
}
return response
And my updated serverless.yml configuration file looks as follows:
service: sls-demo
frameworkVersion: '2'
provider:
name: aws
runtime: python3.8
lambdaHashingVersion: 20201221
region: eu-central-1
functions:
hello:
handler: handler.hello
plugins:
- serverless-python-requirements
custom:
pythonRequirements:
dockerizePip: non-linux
After these changes, when I run the deployment command sls deploy it fails with the following error:
Serverless: Recoverable error occurred (Inaccessible host: `sls-demo-dev-serverlessdeploymentbucket-xyz.s3.eu-central-1.amazonaws.com'. This service may not be available in the `eu-central-1' region.), sleeping for ~5 seconds. Try 1 of 4
I enabled debug logs for the serverless by exporting this flag export SLS_DEBUG=* and the exception stack trace looks as follows:
Serverless: Recoverable error occurred (UnknownEndpoint: Inaccessible host: `sls-demo-dev-serverlessdeploymentbucket-xyz.s3.eu-central-1.amazonaws.com'. This service may not be available in the `eu-central-1' region.
at Request.ENOTFOUND_ERROR (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:507:46)
at Request.callListeners (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
at Request.emit (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
at Request.emit (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/request.js:688:14)
at ClientRequest.error (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/event_listeners.js:339:22)
at ClientRequest.<anonymous> (/Users/macbook/.nvm/versions/node/v14.16.0/lib/node_modules/serverless/node_modules/aws-sdk/lib/http/node.js:96:19)
at ClientRequest.emit (events.js:315:20)
at ClientRequest.EventEmitter.emit (domain.js:467:12)
at TLSSocket.socketErrorListener (_http_client.js:469:9)
at TLSSocket.emit (events.js:315:20)
at TLSSocket.EventEmitter.emit (domain.js:467:12)
at emitErrorNT (internal/streams/destroy.js:106:8)
at emitErrorCloseNT (internal/streams/destroy.js:74:3)
at processTicksAndRejections (internal/process/task_queues.js:80:21)
----------------------------------------------------------------------------------------------------), sleeping for ~7 seconds. Try 1 of 4
This error happens when serverless is trying to upload the build artifact to the s3 bucket:
Serverless: Uploading service sls-demo.zip file to S3 (33.14 MB)...
I suspect that this error happens due to the large size of the file, but I don't know how to resolve this problem, or if I miss something in the configuration.
I tried to upload the large file in the mentioned s3 bucket using the aws cli and it works without any problem:
aws s3 cp large_file.zip s3://sls-demo-dev-serverlessdeploymentbucket-xyz/large_file.zip
I don't know how to troubleshoot this problem, and I was unable to find any answer regarding this error in the internet. Any help appreciated.
I created a simple Flask web app with CRUD operations and deployed in beanstalk with the below requirements.txt file
Flask==1.1.1
Flask-MySQLdb==0.2.0
Jinja2==2.11.1
mysql==0.0.2
mysqlclient==1.4.6
SQLAlchemy==1.3.15
Werkzeug==1.0.0
Flask-Cors==3.0.8
Flask-Mail==0.9.1
Flask-SocketIO==4.3.0
It worked fine, and then I wrote a below function
import tensorflow as tf
import keras
from keras.models import load_model
import cv2
import os
def face_shape_model():
classifier = load_model('face_shape_recog_model.h5')
image = cv2.imread('')
res = str(classifier.predict_classes(image, 1, verbose=0)[0])
return {"prediction": res}
with including below packages in to requirments.txt file
keras==2.3.1
tensorflow==1.14.0
opencv-python==4.2.0.32
whole flask application working fine in my local environment so I zipped and deploy into AWS elasticbeanstalk after deployment it logged below error
Unsuccessful command execution on instance id(s) 'i-0a2a8a4c5b3e56b81'. Aborting the operation.
Your requirements.txt is invalid. Snapshot your logs for details.
as mentioned above I checked my log and it shows below error
distutils.errors.CompileError: command 'gcc' failed with exit status 1
so I searched about the above error find below solution according to that and I created yml file and added it into .ebextension file as below
packages:
yum:
gcc-c++: []
but I still get the same error. how can I solve this or is there any wrong steps above
Thank you.
Finally solved with docker container, I created docker environment In AWS ElasticBeanstalk and deployed it, and now it works fine, below shows my config file and Dockerfile
Dockerfile
FROM python:3.6.8
RUN mkdir -p /usr/src/flask_app/
COPY src/requirements.txt /usr/src/flask_app/
WORKDIR /usr/src/flask_app/
RUN pip install -r requirements.txt
COPY . /usr/src/flask_app
ENTRYPOINT ["python", "src/app.py"]
EXPOSE 5000
Dockerrun.aws.json
{
"AWSEBDockerrunVersion": "1",
"Ports": [
{
"ContainerPort": "5000",
"HostPort": "80"
}
]
}