I am working on a project that requires that an AWS Glue Python script access the AWS Secrets Manager.
I tried giving Glue permissions to do this via IAM, but I don't see how; I can see the permissions strings showing that Lambda has access but I don't see a way to edit them.
I tried creating a new role that had the right permissions but when I went to attach it seemed to have disappeared ...
My fallback workaround is to grab the secret via a tiny Lambda and xfer it via S3 to Glue ... but this should be doable directly. Any help you can give is very welcome!
You may need to add SecretsManagerReadWrite policy to the IAM Role that is associated with the AWS Glue. Please check, we are using secrets manager in our AWS Glue.
After adding the policy to the AWS Glue associated IAM Role, please add the following code snippet to read the credentials from the secret manager:
# Getting DB credentials from Secrets Manager
client = boto3.client("secretsmanager", region_name="us-west-2")
get_secret_value_response = client.get_secret_value(
SecretId="mysecrets-info" <--name as configured in secrets manager
)
secret = get_secret_value_response['SecretString']
secret = json.loads(secret)
uname = secret.get('username')
pwd = secret.get('password')
url = secret.get('host')
By the way you need to be an AWS admin user, to modify the IAM role. If you are a power user, please reach out the admin team for adding the policy to the IAM.
The SecretsManagerReadWrite policy does not give permissions only to Lambda. I think you may be looking at the second statement which grants the Role permissions to create Lambdas (used to create Lambdas to rotate secrets).
Looking at the Glue blog post for setting up ETL jobs, they also say you should only need to add the SecretsManagerReadWrite policy to the Glue role. However, they also say that this is just for testing and you should use a policy that grants only the necessary permissions needed (e.g. use an inline policy that grants secretsmanager:GetSecretValue with Resource being the secret in question).
You don't actually say what error message you are seeing. That might be helpful in figuring out what is going wrong.
Related
I am currently working on the development of a flask application that will be deployed in ECS using boto3 and python.
Currently, for the development of it, I have a python env where I program the different functionalities and interact with the boto3 API assuming roles using the profiles in the .aws / credentials file:
iam = boto3.Session (profile_name = "account_alias", region_name = 'eu-west-1'). client ('iam')
However, to deploy it in the ECS I have to change all these sessions to something like this so that it is assumed by the role of the task:
try:
sts_client = boto3.client ('sts')
assumed_role_object = sts_client.assume_role (
RoleArn = "arn: aws: iam ::" + str (Account) + ": role /" + str (Role)
except Exception as e:
logging.exception ("Could NOT assume role in account:% s", Account)
credentials = assumed_role_object ['Credentials']
boto3_session = boto3.session.Session (
aws_access_key_id = credentials ['AccessKeyId'],
aws_secret_access_key = credentials ['SecretAccessKey'],
aws_session_token = credentials ['SessionToken'],
)
This makes it unable to test it locally (impossible to assume) and leads to numerous errors when deploying it (creating the ECR without testing it).
Is there a way to assume roles without profiles locally in the same way that the task will in the ECS?
Thank you.
I recommend you make a consistently-named role that will exist in all the target accounts. Make the trust policy of that role such that your own credentials can assume it (access key and secret that you are using for your default profile). Then you can use the same logic to assume a role during your local development and not have to change anything. Essentially, instead of using local profiles with different credentials, you would use a single credential that has permission to assume a role in every target account. I've worked in an environment with the same problem, and doing this allows me to have the same local code that I deploy.
There's no reason your local code can't also assume a role, and if there are concerns about making a role that you can assume in all those target accounts just consider that by using different profiles (with, I assume, different credentials) you must already have access to do the things you would be doing by assuming the role. If the role has more permissions than you should have, then make a new role so you assume different roles locally and in AWS. But at least that will allow you to use the same code--just with a different role name variable passed in.
You'll either need to make ECR usage use profiles or make localhost usage assume roles if you want the code to be the same, and I believe it is easier to make localhost usage assume roles than to get your ECR tasks set up to use profiles.
What I understand is that, in order to access AWS applications such as redshift, the way to do it is
client = boto3.client("redshift", region_name="someRegion", aws_access_key_id="foo", aws_secret_access_key="bar")
response = client.describe_clusters(ClusterIdentifier="mycluster")
print(response)
This code runs fine for both locally through pycharm, as well as on AWS lambda.
However, am I correct that this aws_access_key_id and aws_secret_access_key are both from me? IE: my IAM user security access keys. Is this supposed to be the case? Or am I suppose to create a different user / role in order to access redshift via boto3?
The more important question is, how do I properly store & retrieve aws_access_key_id and aws_secret_access_key? I understand that this could potentially be done via secrets manager, but I am still faced with the problem that, if I run the below code, I get an error saying that it is unable to locate credentials.
client = boto3.client("secretsmanager", region_name="someRegion")
# Met with the problem that it is unable to locate my credentials.
The proper way to do this would be for you to create an IAM role which allows the desired redshift functionality, and then attaching that role to your lambda.
When you create the role, you have the flexibility to create a policy to fine-grain access permissions to certain actions and/or certain resources.
After you have attached the IAM role to your lambda, you will simply be able to do:
>>> client = boto3.client("redshift")
From the docs. The first & seconds options are not secured since you mix the credentials with the code.
If the code runs on AWS EC2 the best way is using "assume role" where you grant the EC2 instance permissions. If the code run outside AWS you will have to select an option like using ~/.aws/credentials
Boto3 will look in several locations when searching for credentials. The mechanism in which Boto3 looks for credentials is to search through a list of possible locations and stop as soon as it finds credentials. The order in which Boto3 searches for credentials is:
Passing credentials as parameters in the boto.client() method
Passing credentials as parameters when creating a Session object
Environment variables
Shared credential file (~/.aws/credentials)
AWS config file (~/.aws/config)
Assume Role provider
Boto2 config file (/etc/boto.cfg and ~/.boto)
Instance metadata service on an Amazon EC2 instance that has an IAM role configured.
I want to access AWS comprehend api from python script. Not getting any leads of how do I remove this error. One thing I know that I have to get session security token.
try:
client = boto3.client(service_name='comprehend', region_name='us-east-1', aws_access_key_id='KEY ID', aws_secret_access_key= 'ACCESS KEY')
text = "It is raining today in Seattle"
print('Calling DetectEntities')
print(json.dumps(client.detect_entities(Text=text, LanguageCode='en'), sort_keys=True, indent=4))
print('End of DetectEntities\n')
except ClientError as e:
print (e)
Error : An error occurred (UnrecognizedClientException) when calling the DetectEntities operation: The security token included in the request is invalid.
This error suggesting that you have provided invalid credentials.
It is also worth nothing that you should never put credentials inside your source code. This can lead to potential security problems if other people obtain access to the source code.
There are several ways to provide valid credentials to an application that uses an AWS SDK (such as boto3).
If the application is running on an Amazon EC2 instance, assign an IAM Role to the instance. This will automatically provide credentials that can be retrieved by boto3.
If you are running the application on your own computer, store credentials in the .aws/credentials file. The easiest way to create this file is with the aws configure command.
See: Credentials — Boto 3 documentation
Create a profile using aws configure or updating ~/.aws/config. If you only have one profile to work with = default, you can omit profile_name parameter from Session() invocation (see example below). Then create AWS service specific client using the session object. Example;
import boto3
session = boto3.session.Session(profile_name="test")
ec2_client = session.client('ec2')
ec2_client.describe_instances()
ec2_resource = session.resource(‘ec2’)
One useful tool I use daily is this: https://github.com/atward/aws-profile/blob/master/aws-profile
This makes assuming role so much easier!
After you set up your access key in .aws/credentials and your .aws/config
you can do something like:
AWS_PROFILE=**you-profile** aws-profile [python x.py]
The part in [] can be substituted with anything that you want to use AWS credentials. e.g., terraform plan
Essentially, this utility simply put your AWS credentials into os environment variables. Then in your boto script, you don't need to worry about setting aws_access_key_id and etc..
I have a python app that uses several services from aws. I have one access key and secret for each service. For most of the services I use boto and don't need AWS_ACCESS_KEY_ID or AWS_SECRET_ACCESS_KEY in the environment. For dynamoDB I use pynamoDB and I have no idea how to set the credentials without these variables.
I want to standardize the credential in a settings file to avoid errors like clash of credentials.
Is this possible? If so, how is it done?
From the PynamoDB documentation:
PynamoDB uses botocore to interact with the DynamoDB API. Thus, any
method of configuration supported by botocore works with PynamoDB. For
local development the use of environment variables such as
AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY is probably preferable.
You can of course use IAM users, as recommended by AWS. In addition
EC2 roles will work as well and would be recommended when running on
EC2.
Note that if all the services you are interacting with are within the same AWS account, then the preferred way to supply credentials would be to create a single IAM account with all the necessary permissions attached, or an IAM role if the code is running on EC2 or Lambda.
I was searching for this online and came across this question, though this is old, I am sharing my solution so that it might be helpful someone.
When defining the Dynamo DB model all we need is to add one additional line of code which contains the IAM rolename. Below is a sample model.
If you change the model like the one below we don't need ~/.aws/credentials file on the container.
Note: Make sure you attach DynamoDBRead or write policy to the IAM role, I have attached AmazonDynamoDBFullAccess policy for my instances IAM role.
from pynamodb.models import Model
from pynamodb.attributes import (
UnicodeAttribute, NumberAttribute, UnicodeSetAttribute, UTCDateTimeAttribute
)
import urllib2
class TestClass(Model):
email = UnicodeAttribute(hash_key=True)
UUID = UnicodeAttribute(range_key=True)
class Meta:
region = 'eu-west-2'
# Refer: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/iam-roles-for-amazon-ec2.html
instanceProfileName = urllib2.urlopen(
'http://169.254.169.254/latest/meta-data/iam/security-credentials/').read()
table_name = 'dynamodb-tablename'
I'm trying to use AWS' recently announced "IAM roles for EC2" feature, which lets security credentials automatically get delivered to EC2 instances. (see http://aws.amazon.com/about-aws/whats-new/2012/06/11/Announcing-IAM-Roles-for-EC2-instances/).
I've set up an instance with an IAM role as described. I can also get (seemingly) proper access key / credentials with curl.
However, boto fails to do a simple call like "get_all_buckets", even though I've turned on ALL S3 permissions for the role.
The error I get is "The AWS Access Key Id you provided does not exist in our records"
However, the access key listed in the error matches the one I get from curl.
Here is the failing script, run on an EC2 instance with an IAM role attached that gives all S3 permissions:
import urllib2
import ast
from boto.s3.connection import S3Connection
resp=urllib2.urlopen('http://169.254.169.254/latest/meta-data/iam/security-credentials/DatabaseApp').read()
resp=ast.literal_eval(resp)
print "access:" + resp['AccessKeyId']
print "secret:" + resp['SecretAccessKey']
conn = S3Connection(resp['AccessKeyId'], resp['SecretAccessKey'])
rs= conn.get_all_buckets()
If you are using boto 2.5.1 or later it's actually much easier than this. Boto will automatically find the credentials in the instance metadata for you and use them as long as no other credentials are found in environment variables or in a boto config file. So, you should be able to simply do this on the EC2 instance:
>>> import boto
>>> c = boto.connect_s3()
>>> rs = c.get_all_buckets()
The reason that your manual approach is failing is that the credentials associated with the IAM Role is a temporary session credential and consists of an access_key, a secret_key and a security_token and you need to supply all three of those values to the S3Connection constructor.
I don't know if this answer will help anyone but I was getting the same error, I had to solve my problem a little differently.
First, my amazon instance did not have any IAM roles. I thought I could just use the access key and the secret key but I kept getting this error with only those two keys. I read I needed a security token as well, but I didn't have one because I didn't have any IAM roles. This is what I did to correct the issue:
Create an IAM role with AmazonS3FullAccess permissions.
Start a new instance and attach my newly created role.
Even after doing this it still didn't work. I had to also connect to the proper region with the code below:
import boto.s3.connection
conn = boto.s3.connect_to_region('your-region')
conn.get_all_buckets()