I got an AWS python Lambda function which contains few python files and also several dependencies.
The app is build using Chalice so by that the function will be mapped like any REST function.
Before the deployment in prod env, I want to test it locally, so I need to pack all this project (python files and dependencies), I tried to look over the web for the desired solution but I couldn't find it.
I managed to figrue how to deploy one python file, but a whole project did not succeed.
Take a look to the Atlassian's Localstack: https://github.com/atlassian/localstack
It's a full copy of the AWS cloud stack, locally.
I use Travis : I hooked it to my master branch in git, so that when I push on this branch, Travis tests my lambda, with a script that uses pytest, after having installed all its dependencies with pip install. If all the tests passed, it then deploy the lambda in AWS in my prod-env.
Related
I want to use this repo and I have created and activated a virtualenv and installed the required dependencies.
I get an error when I run pytest.
And under the file binance_cdk/app.py it describes the following tasks:
App (PSVM method) entry point of the program.
Note:
Steps tp setup CDK:
install npm
cdk -init (creates an empty project)
Add in your infrastructure code.
Run CDK synth
CDK bootstrap <aws_account>/
Run CDK deploy ---> This creates a cloudformation .yml file and the aws resources will be created as per the mentioned stack.
I'm stuck on step 3, what do I add in this infrastructure code, and if I want to use this on amazon sagemaker which I am not familiar with, do I even bother doing this on my local terminal, or do I do the whole process regardless on sagemaker?
Thank you in advance for your time and answers !
The infrastructure code is the Python code that you want to write for the resources you want to provision with SageMaker. In the example you provided for example the infra code they have is creating a Lambda function. You can do this locally on your machine, the question is what do you want to achieve with SageMaker? If you want to create an endpoint then following the CDK Python docs with SageMaker to identify the steps for creating an endpoint. Here's two guides, the first is an introduction to the AWS CDK and getting started. The second is an example of using the CDK with SageMaker to create an endpoint for inference.
CDK Python Starter: https://towardsdatascience.com/build-your-first-aws-cdk-project-18b1fee2ed2d
CDK SageMaker Example: https://github.com/philschmid/cdk-samples/tree/master/sagemaker-serverless-huggingface-endpoint
Is it possible to run the AWS CDK lifecycle directly in python (or any other language), without using the cli? e.g.
app.py
app = cdk.App()
Stack(app, ..)
app.synth()
app.deploy()
run python
python app.py //instead of cdk deploy...
this could be useful to prepare temporary test scenarios.
What's possible as of today:
Programatically synth templates without cdk synth?
Yes. CDK's testing constructs work this way. Tests assert against a programatically synth-ed template.
template = Template.from_stack(processor_stack)
template.resource_count_is("AWS::SNS::Subscription", 1)
Programatically deploy apps without cdk deploy?
Not in the way the OP describes, but you can get close with the CDK's CI/CD constructs. The CDK Pipelines construct synths and deploys CDK apps programatically. Build a pipeline that builds and tears-down a testing environment on each push to your remote repository. Alternatively, a standalone CodeBuild Project can be used with commands to cdk deploy an app and perform tests when triggered by an event.
Also worth mentioning is cdk watch for rapid local development iterations. It automatically deploys incremental changes as you code.
I would like to use a serverless lambda that will execute commands from a tool called WSO2 API CTL as I would on linux cli. I am not sure of how to mimic the downloading and calling of the commands as if I were on a linux machine using either Nodejs or Python via the lambda?
I am okay with creating and setting up the lambda and even getting it in the right VPC so that the commands will reach an application on an EC2 instance but I am stuck at how to actually execute the linux commands using either Nodejs or Python and which one would be better, if any.
After adding the following I get an error trying to download:
os.system("curl -O https://apim.docs.wso2.com/en/latest/assets/attachments/learn/api-controller/apictl-3.2.1-linux-x64.tar.gz")
Warning: Failed to create the file apictl-3.2.1-linux-x64.tar.gz: Read-only
It looks like there is no specific reason to download apictl during the initialisation of your Lambda. Therefore, I would propose to bundle it with your deployment package.
The advantage of this approach are:
Quicker initialisation
Less code in your Lambda
You could extend your CI/CD pipeline to download the application during build and then add it to your ZIP archive that you deploy.
Just a conceptual question here.
I'm a newbie to aws. I have a node app and a python file that is currently on a Flask server. The node app sends data to the Py server and gets data back. This takes approx 3.2 secs to happen. I am not sure how I can apply this to AWS. I tried sagemaker but it was really costly for me. Is there anyway I can create a Python server with an endpoint in AWS within the free tier?
Thanks
Rushi
You do not need to use sagemaker to deploy your flask application to AWS. AWS has a nice documentation to deploy a Flask Application to an AWS Elastic Beanstalk environment.
Other than that you can also deploy the application using two methods.
via EC2
via Lambda
EC2 Instances
You can launch the ec2 instance with public IP with SSH enabled from your IP address. Then SSH into the instance and install the python, it's libraries and your application.
Lambda
AWS lambda is the perfect solution. It scales automatically, depends upon the requests your application will receive.
As lambda needs your dependencies be available in the package, so you need to install them using --target parameter, zip the python code along with the installed packages and then upload to the Lambda.
pip install --target ./package Flask
cd package
zip -r9 function.zip . # Create a ZIP archive of the dependencies.
cd .. && zip -g function.zip lambda_function.py # Add your function code to the archive.
For more detailed instructions you can read these documentations
Lambda
I wonder is it able to create an HTTP triggered python function on Azure Function without doing any local coding? I want to do everything on Azure cloud. My python function codes are in a Github/Azure repos repository, but I do not have all the extra files of an Azure function project (for example, a init.py script file that is the HTTP trigger function of the Azure Function App). Is it possible to generate those files from Azure (without generating any Azure Function related files on my local computer)? I noticed that we cannot do in-portal editing for Python function Apps.
As far as I know, we can just deploy the python function from local to Azure cloud, but can not deploy it as you expected. And I think it will not be too difficult to us to deploy the python function from local to azure cloud.
Since you have had the main python function code "init.py", you just need to sign in to Azure in you VS code and create python function by following this tutorial. And then use your init.py code to replace the new python function code. After that, run the command below in "TERMINAL" window to generate the "requirements.txt":
pip freeze > requirements.txt
The "requirements.txt" includes all of the modules which imported in your "init.py" and when the function deployed to azure, azure will install these modules by this "requirements.txt". I saw you mentioned you don't have all the extra files of this function project, if these modules are what you care about, the "requirements.txt" will solve your problem.
Then use this command to deploy it to Azure:
func azure functionapp publish hurypyfunapp --build remote