I'm using Amazon S3 in a Flask Python application, however, I don't want to hardcode my access keys as Amazon has problems with making the keys publicly available. Is there a way to get the keys into the application without exposing them. I saw a suggestion about using environment variables and another about IAM User roles but the documentation isn't helping.
Edit: I forgot to mention that I'm deploying this application on Docker and want to allow it so that if another user pulls the image from docker, my access keys won't be compromised. I'm not using AWS EC2
Following could be of help:
IAM Roles (If your application is already running in AWS, Instance Roles can help you fetch temporary tokens to access resources like S3. AWS CLIs OR official SDKs are already built with these capabilities and you need not implement any custom code. For this to work, you assign a Role 'X' to the EC2 instance, Role 'X' then needs to have a policy mapping, where you define the permissions)
A sample policy could be something like:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "AllowAccessToObjects",
"Action": [
"s3:PutObject",
"s3:PutObjectAcl",
"s3:GetObject",
"s3:GetObjectAcl",
"s3:GetObjectVersion",
"s3:GetObjectVersionAcl",
"s3:ListMultipartUploadParts"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<Bucket_Name>/*"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"w.x.y.z/32"
]
}
}
},
{
"Sid": "AllowAccessToBucket",
"Action": [
"s3:PutObject",
"s3:ListBucket",
"s3:GetBucketVersioning",
"s3:ListBucketVersions"
],
"Effect": "Allow",
"Resource": [
"arn:aws:s3:::<Bucket_Name>"
],
"Condition": {
"IpAddress": {
"aws:SourceIp": [
"a.b.c.d/32"
]
}
}
}
]
}
Usually the best approach is to never have any statically provisioned credentials at all. In case implementing something like that is not possible at all, then :
Other options could be storing the secrets in an external secret store like Vault etc., and when the container starts, the secrets can be fetched and injected before bootstrapping the application. These are then available as ENVs
The Python AWS SDK looks at several possible locations for credentials. The most pertinent here would be environment variables:
Boto3 will check these environment variables for credentials:
AWS_ACCESS_KEY_ID
The access key for your AWS account.
AWS_SECRET_ACCESS_KEY
The secret key for your AWS account.
You should create separate IAM users with appropriate permissions for each user or team you're going to distribute your docker image to, and they can set those environment variables via docker when running your image.
Related
I'm trying to upload an image to AWS S3. This code previously worked fine (and still working for another project). This is a brand new project with a new AWS S3 bucket. I noticed they again changed a lot and maybe it's a problem.
This is the code:
s3_client.upload_fileobj(
uploaded_file,
files_bucket_name,
key_name,
ExtraArgs={
'ContentType': uploaded_file.content_type
}
)
This is the permission policy for the bucket:
{
"Version": "2012-10-17",
"Id": "Policy1204",
"Statement": [
{
"Sid": "Stmt15612",
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject",
"s3:PutObject"
],
"Resource": "arn:aws:s3:::bucket-name/*"
}
]
}
The upload did not work until I added the "PutObject" here but it was working for another project. I don't like about this policy that PutObject is now public available.
How to make:
all images are public available
but only owner can upload files?
This are screenshots from AWS permissions for this bucket:
The problem has gone as soon as I created an IAM user and granted it full access to S3. Not sure if this solution is good or not but at least it's working now.
It appears that your requirement is:
Allow everyone to see the files
Only allow an owner to upload them
There is a difference between "seeing the files" -- ListObjects allows listing of the objects in a bucket while GetObject allows downloading of an object.
If you want to make all objects available for download assuming that the user knows the name of the object, then you could use a policy like this:
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject"
],
"Resource": "arn:aws:s3:::bucket-name/*"
}
]
}
Note that this policy will not permit viewing the contents of the bucket.
If you wish to allow a specific IAM User permission to upload files to the bucket, then put this policy on the IAM User (not on the Bucket):
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::bucket-name/*"
}
]
}
I have a python script in our K8s cluster that is run as a k8s Cronjob every few minutes. The script checks the nodes in the cluster and if a node is unhealthy for more than 5 minutes, it terminates the node.
To connect to AWS I use Boto3.
requirement.txt
boto3==1.16.11
botocore==1.19.11
and the permissions are passed as pod annotations.
Annotations: iam.amazonaws.com/role: arn:aws:iam::123456789:role/k8s-nodes-monitoring-role
The IAM role has arn:aws:iam::aws:policy/AmazonEC2FullAccess policy and a valid trust policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"Service": "ec2.amazonaws.com"
},
"Action": "sts:AssumeRole"
},
{
"Sid": "",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::123456789:role/nodes.my-domain.com"
},
"Action": "sts:AssumeRole"
}
]
}
The problem that I facing is that on some occasions the script throws NoCredentialsError('Unable to locate credentials') error. This behaviour is not consistent as on most occasions the script has successfully terminates the unhealthy node and I can cross-check it against AWS CloudTrail events.
I can see in kub2iam logs that the Get request receives 200 but the Put request receives 403.
ime="2020-12-21T12:50:16Z" level=info msg="GET /latest/meta-data/iam/security-
credentials/k8s-nodes-monitoring-role (200) took 47918.000000 ns" req.method=GET
req.path=/latest/meta-data/iam/security-credentials/k8s-nodes-monitoring-role
req.remote=100.116.203.13 res.duration=47918 res.status=200
time="2020-12-21T12:52:16Z" level=info msg="PUT /latest/api/token (403) took 19352999.000000
ns" req.method=PUT req.path=/latest/api/token req.remote=100.116.203.14
res.duration=1.9352999e+07 res.status=40
Any help or idea about how to debug this will be highly appreciated.
I dont know kube2iam in detail, but maybe you should switch to a AWS native way called IRSA (IAM Roles for Service Accounts). You can find all necessary information in this blog post: https://aws.amazon.com/de/blogs/opensource/introducing-fine-grained-iam-roles-service-accounts/
I have a lambda stack that is deployed in account A, and a stepfunction stack deployer in account B. Now How do I invoke this stepfunction from the lambda using python cdk? specifically what permissions do i need to give them?
The lambda (Account A) has an IAM role(RoleA) assigned. The Step function (Account B) has an IAM role (RoleB) assigned.
Permissions
The lambda's IAM role should have permission to assume the role from (Account B)
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "sts:AssumeRole",
"Resource": "*"
}
]
}
The Step functions IAM role(Role B) should have a trusted policy that allows the Lambda's IAM role to assume it. in the following trust policy 123456789012 is the account number of Account A
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::123456789012:root"
]
},
"Action": "sts:AssumeRole"
}
]
}
Inside the lambda
The lambda should have code that assumes the role(Role B) from Account B and get temporary credentials
Using those credentials the lambda should invoke the step function.
How to assume an IAM role in a different account from lambda
If you want to trigger something when a cdk deployment happens:
Seems a bizarre use-case but I think the solution here is to define a custom resource.
Both cdk and cloudformation support that:
https://docs.aws.amazon.com/cdk/api/latest/docs/custom-resources-readme.html
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/template-custom-resources.html
Because you are using a custom resource you would need to further handle Create, Update, Delete events by yourself. Similarly send completion responses, I would advice you to use cfnresponsemodule to send back completion responses otherwise cdk will never be able to tell when your custom resource function completed:
https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/cfn-lambda-function-code-cfnresponsemodule.html
I was provisioned some AWS keys. These keys give me access to certain directories in a s3 bucket. I want to use boto3 to interact with the directories that were exposed to me, however it seems that I can't actually do anything with the bucket at all, since I don't have access to the entire bucket.
This works for me from my terminal:
aws s3 ls s3://the_bucket/and/this/specific/path/
but if I do:
aws s3 ls s3://the_bucket/
I get:
An error occurred (AccessDenied) when calling the ListObjects
operation: Access Denied
which also happens when I try to access the directory via boto3.
session = boto3.Session(profile_name=my_creds)
client=session.client('s3')
list_of_objects = client.list_objects(Bucket='the_bucket', Prefix='and/this/specific/path', Delimiter='/')
Do I need to request access to the entire bucket for boto3 to be usable?
You need to set this Bucket Policy:
{
"Sid": "<SID>",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::<account>:user/<user_name>"
},
"Action": [
"s3:GetBucketLocation",
"s3:ListBucket"
],
"Resource": "arn:aws:s3:::<bucket_name>"
}
For more information about Specifying Permissions in a Policy
So I have created a lambda function (the purpose of which doesn't matter anymore) and have tested that it works when run from my laptop. However the bigger problem is that I cannot get it to run off of a test event or on a schedule in AWS.
When I try to run it from AWS I get a 300s timeout error.
The following is included for your consideration:
Function
Logs
Trigger Event
Policy
VPC Related Configuration
If anyone can tell me what the issue might be, I would appreciate it as I have been searching for the solution for about 3 days.
FUNCTION:
from __future__ import print_function
def lambda_handler(event, context):
if event["account"] == "123456789012":
print("Passed!!")
return event['time']
import boto3
import datetime
def find_auto_scaling_instances():
"""
Find Auto-scaling Instances
"""
client=boto3.client("autoscaling")
auto_scaling=client.describe_auto_scaling_groups()
dict={}
for group in auto_scaling["AutoScalingGroups"]:
print("hello")
auto_scaling_group_name=group["AutoScalingGroupName"]
number_of_instances=len(group["Instances"])
if number_of_instances == 0:
continue
else:
for i in range(number_of_instances):
if group["Instances"][i]["LifecycleState"] == "InService":
instance_id=group["Instances"][i]["InstanceId"]
dict[auto_scaling_group_name]=instance_id
break
return dict
def find_staging_instances():
"""
Find Static Instances
"""
stg_list=[]
tag_list=["Y-qunie-stepsrv-1a","S-StepSvr"]
for i in range(num_of_instances):
print("hello2")
target=instances[i]["Instances"][0]
number_of_tags=len(target["Tags"])
for tag in range(number_of_tags):
if target["Tags"][tag]["Value"] in tag_list:
stg_list+=[target["InstanceId"]]
return stg_list
def volumes_per_instance():
"""
Find the EBS associated with the Instances
"""
instance_disk={}
for i in range(num_of_instances):
print("hello3")
target=instances[i]["Instances"][0]
if target["InstanceId"] in instance_list:
instance_disk[target["InstanceId"]]=[]
for disk in range(len(target["BlockDeviceMappings"])):
print("hello4")
instance_disk[target["InstanceId"]]+=\
target["BlockDeviceMappings"][disk]["Ebs"]["VolumeId"]]
return instance_disk
#Group instances together and prepare to process
instance_in_asgroup_dict=find_auto_scaling_instances()
as_instance_list=[]
for group in instance_in_asgroup_dict:
print("hello5")
as_instance_list+=[instance_in_asgroup_dict[group]]
client=boto3.client("ec2")
instances=client.describe_instances()["Reservations"]
num_of_instances=len(instances)
staging_instances=find_staging_instances()
instance_list=[]
instance_list+=as_instance_list
instance_list+=staging_instances
#Gather Disk Information
inst_disk_set=volumes_per_instance()
date=str(datetime.datetime.now()).replace(" \
","_").replace(":","").replace(".","")
#Create Disk Images
as_image={}
image=[]
for instance in instance_list:
print("hello6")
if instance in as_instance_list:
as_image[instance]=client.create_image(
InstanceId=instance,
Name=instance+"-"+date+"-AMI",
Description="AMI for instance "+instance+".",
NoReboot=True
)
else:
image+=[client.create_image(
InstanceId=instance,
Name=instance+"-"+date+"-AMI",
Description="AMI for instance "+instance+".",
NoReboot=True
)]
LOGS:
18:03:30
START RequestId: 0ca9e0a3-7f11-11e6-be11-6974d9213102 Version: $LATEST
18:08:30
END RequestId: 0ca9e0a3-7f11-11e6-be11-6974d9213102
18:08:30
REPORT RequestId: 0ca9e0a3-7f11-11e6-be11-6974d9213102 Duration: 300001.99 ms Billed Duration: 300000 ms Memory Size: 128 MB Max Memory Used: 24 MB
18:08:30
2016-09-20T09:08:30.544Z 0ca9e0a3-7f11-11e6-be11-6974d9213102 Task timed out after 300.00 seconds
TRIGGER_EVENT:
{
"account": "123456789012",
"region": "us-east-1",
"detail": {},
"detail-type": "Scheduled Event",
"source": "aws.events",
"time": "1970-01-01T00:00:00Z",
"id": "cdc73f9d-aea9-11e3-9d5a-835b769c0d9c",
"resources": [
"arn:aws:events:us-east-1:123456789012:rule/my-schedule"
]
}
EDIT-1
IAM POLICY:
From my understanding all I need to allow VPC Access to my function is to add the following privilages to the lambda function's assigned policy.
ec2:CreateNetworkInterface
ec2:DeleteNetworkInterface
ec2:DescribeNetworkInterfaces
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents",
"ec2:CreateNetworkInterface",
"ec2:DescribeNetworkInterfaces",
"ec2:DeleteNetworkInterface"
],
"Resource": "*"
},
{
"Action": [
"ec2:DescribeInstances",
"ec2:CreateImage",
"autoscaling:DescribeAutoScalingGroups"
],
"Effect": "Allow",
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:PutLogEvents"
],
"Resource": "arn:aws:logs:*:*:*"
}
] }
EDIT 2
CONFIGURATION:
The subnets, security groups and VPC attached to the function.
EDIT 3 (CONCLUSION)
Mark Gave an excellent answer, informing me that I had set my function up to run inside a VPC yet I was not accessing resources within the VPC. Rather, I was accessing the Amazon API endpoint which required that I have access to the internet or the transaction would timeout.
As such, there were two options available to fix this situation.
Remove my VPC settings
Create a NAT Gateway inside my VPC
I chose the one that costs the least money.
You enabled VPC access for your Lambda function. So now it only has access to resources inside your VPC. Note that the AWS API exists outside your VPC. Any attempt to access something outside your VPC is going to result in a network timeout. That's why you are seeing the timeout issues.
To fix this, you can move the Lambda function outside the VPC, or you can add a NAT Gateway to your VPC. I'm not seeing anything in your code that is accessing anything inside your VPC, so it's probably cheapest and easiest to just remove the VPC settings from the Lambda function.