Authenticating to Azure Key Vault locally using DefaultAzureCredential - python

I am attempting to run this 'Retrieve a secret from the vault' example locally (Ubuntu 19.10) to retrieve a secret from an Azure Key Vault:
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
client = SecretClient(vault_url="https://<<vaultname>>.vault.azure.com",
credential=DefaultAzureCredential())
secret = client.get_secret("<<mysecret>>")
However I receive the following error:
azure.core.exceptions.ClientAuthenticationError:
No credential in this chain provided a token.
Attempted credentials:
EnvironmentCredential: Incomplete environment configuration. See https://aka.ms/python-sdk-identity#environment-variables for
expected environment variables
ImdsCredential: IMDS endpoint unavailable
Please visit the documentation at
https://aka.ms/python-sdk-identity#defaultazurecredential
to learn what options DefaultAzureCredential supports
The documentation on Service-to-Service authentication to Key Vault seems to suggest that I should be able to authenticate by the Azure CLI, and I've followed the steps to login via az login, select the appropriate subscription (which I've done just in case, despite only having one), and verify access via az account get-access-token --resource https://vault.azure.net which does return a token, however still receive the error above.
Am I wrong in assuming I should be able to authenticate after logging in via the cli?
And if so, and I need to manually set the environment variables described in the documentation link provided for EnvironmentCredential, what values do I need to supply for AZURE_CLIENT_ID and AZURE_CLIENT_SECRET?

Am I wrong in assuming I should be able to authenticate after logging in via the cli?
You're not wrong, it's possible with the current preview version of azure-identity, 1.4.0b2 as I write this. With that installed, your code should work once you've logged in to the CLI.
... what values do I need to supply for AZURE_CLIENT_ID and AZURE_CLIENT_SECRET?
These would be the client (or "application") ID of a service principal, and one of its secrets. The azure-keyvault-secrets documentation describes how to create a service principal and configure its access to a Key Vault, using the CLI.
Briefly restating that documentation here, you can create a service principal with this command:
az ad sp create-for-rbac --name http://my-application
From the output of that command, "appId" is the value of AZURE_CLIENT_ID and "password" is the value of AZURE_CLIENT_SECRET.
Then, to grant the service principal access to the Key Vault's secrets:
az keyvault set-policy --name <<vaultname>> --spn $AZURE_CLIENT_ID --secret-permissions get set list delete backup recover restore purge

Related

unable to locate credentials for boto3.client both locally, and on lambda

What I understand is that, in order to access AWS applications such as redshift, the way to do it is
client = boto3.client("redshift", region_name="someRegion", aws_access_key_id="foo", aws_secret_access_key="bar")
response = client.describe_clusters(ClusterIdentifier="mycluster")
print(response)
This code runs fine for both locally through pycharm, as well as on AWS lambda.
However, am I correct that this aws_access_key_id and aws_secret_access_key are both from me? IE: my IAM user security access keys. Is this supposed to be the case? Or am I suppose to create a different user / role in order to access redshift via boto3?
The more important question is, how do I properly store & retrieve aws_access_key_id and aws_secret_access_key? I understand that this could potentially be done via secrets manager, but I am still faced with the problem that, if I run the below code, I get an error saying that it is unable to locate credentials.
client = boto3.client("secretsmanager", region_name="someRegion")
# Met with the problem that it is unable to locate my credentials.
The proper way to do this would be for you to create an IAM role which allows the desired redshift functionality, and then attaching that role to your lambda.
When you create the role, you have the flexibility to create a policy to fine-grain access permissions to certain actions and/or certain resources.
After you have attached the IAM role to your lambda, you will simply be able to do:
>>> client = boto3.client("redshift")
From the docs. The first & seconds options are not secured since you mix the credentials with the code.
If the code runs on AWS EC2 the best way is using "assume role" where you grant the EC2 instance permissions. If the code run outside AWS you will have to select an option like using ~/.aws/credentials
Boto3 will look in several locations when searching for credentials. The mechanism in which Boto3 looks for credentials is to search through a list of possible locations and stop as soon as it finds credentials. The order in which Boto3 searches for credentials is:
Passing credentials as parameters in the boto.client() method
Passing credentials as parameters when creating a Session object
Environment variables
Shared credential file (~/.aws/credentials)
AWS config file (~/.aws/config)
Assume Role provider
Boto2 config file (/etc/boto.cfg and ~/.boto)
Instance metadata service on an Amazon EC2 instance that has an IAM role configured.

Accessing DynamoDB Local from boto3

I am doing AWS tutorial Python and DynamoDB. I downloaded and installed DynamoDB Local. I got the access key and secret access key. I installed boto3 for python. The only step I have left is setting up authentication credentials. I do not have AWS CLI downloaded, so where should I include access key and secret key and also the region?
Do I include it in my python code?
Do I make a file in my directory where I put this info? Then should I write anything in my python code so it can find it?
You can try passing the accesskey and secretkey in your code like this:
import boto3
session = boto3.Session(
aws_access_key_id=ACCESS_KEY,
aws_secret_access_key=SECRET_KEY,
)
client = session.client('dynamodb')
OR
dynamodb = session.resource('dynamodb')
From the AWS documentation:
Before you can access DynamoDB programmatically or through the AWS
Command Line Interface (AWS CLI), you must configure your credentials
to enable authorization for your applications. Downloadable DynamoDB
requires any credentials to work, as shown in the following example.
AWS Access Key ID: "fakeMyKeyId"
AWS Secret Access Key:"fakeSecretAccessKey"
You can use the aws configure command of the AWS
CLI to set up credentials. For more information, see Using the AWS
CLI.
So, you need to create an .aws folder in yr home directory.
There create the credentials and config files.
Here's how to do this:
https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-files.html
If you want to write portable code and keep in the spirit of developing 12-factor apps, consider using environment variables.
The advantage is that locally, both the CLI and the boto3 python library in your code (and pretty much all the other offical AWS SDK languages, PHP, Go, etc.) are designed to look for these values.
An example using the official Docker image to quickly start DynamoDB local:
# Start a local DynamoDB instance on port 8000
docker run -p 8000:8000 amazon/dynamodb-local
Then in a terminal, set some defaults that the CLI and SDKs like boto3 are looking for.
Note that these will be available until you close your terminal session.
# Region doesn't matter, CLI will complain if not provided
export AWS_DEFAULT_REGION=us-east-1
# Set some dummy credentials, dynamodb local doesn't care what these are
export AWS_ACCESS_KEY_ID=abc
export AWS_SECRET_ACCESS_KEY=abc
You should then be able to run the following (in the same terminal session) if you have the CLI installed. Note the --endpoint-url flag.
# Create a new table in DynamoDB Local
aws dynamodb create-table \
--endpoint-url http://127.0.0.1:8000 \
--table-name tmp \
--attribute-definitions AttributeName=id,AttributeType=S \
--key-schema AttributeName=id,KeyType=HASH \
--billing-mode PAY_PER_REQUEST
You should then able to list out the tables with:
aws dynamodb list-tables --endpoint-url http://127.0.0.1:8000
And get a result like:
{
"TableNames": [
"tmp"
]
}
So how do we get the endpoint-url that we've been specifying in the CLI to work in Python? Unfortunately, there isn't a default environment variable for the endpoint url in the boto3 codebase, so we'll need to pass it in when the code runs. The docs for .NET and Java are comprehensive but for Python, they are a bit more elusive. From the boto3 github repo and also see this great answer, we need to create a client or resource with the endpoint_url keyword. In the below, we're looking for a custom environment variable called AWS_DYNAMODB_ENDPOINT_URL. The point being that if specified, it will be used, otherwise will fall back to whatever the platform default is, making your code portable.
# Run in the same shell as before
export AWS_DYNAMODB_ENDPOINT_URL=http://127.0.0.1:8000
# file test.py
import os
import boto3
# Get environment variable if it's defined
# Make sure to set the environment variable before running
endpoint_url = os.environ.get('AWS_DYNAMODB_ENDPOINT_URL', None)
# Using (high level) resource, same keyword for boto3.client
resource = boto3.resource('dynamodb', endpoint_url=endpoint_url)
tables = resource.tables.all()
for table in tables:
print(table)
Finally, run this snippet with
# Run in the same shell as before
python3 test.py
# Should produce the following output:
# dynamodb.Table(name='tmp')

Google Storage not using service account even with environment variable properly set

I was trying to save two files to GCP Storage using the following commands in a Jupyter Notebook:
!gsutil cp ./dist/my_custom_code-0.1.tar.gz gs://$BUCKET_NAME/custom_prediction_routine_tutorial/my_custom_code-0.1.tar.gz
!gsutil cp model.h5 preprocessor.pkl gs://$BUCKET_NAME/custom_prediction_routine_tutorial/model/
The bucket has been created properly since I can see it in the bucket list on GCP. Also in Permissions for the bucket, I can see the service account created. Plus, I made sure the environment variable is set by running:
export GOOGLE_APPLICATION_CREDENTIALS="/home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json"
This can be verified by running this in Python:
import os
print('Credendtials from environ: {}'.format(os.environ.get('GOOGLE_APPLICATION_CREDENTIALS')))
which shows:
Credentials from environ: /home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json
And I do have the json file stored at the specified location. However, when I tried to save files using the commands shown at the top, I kept getting this error message:
AccessDeniedException: 403 george***#gmail.com does not have storage.objects.list access to the Google Cloud Storage bucket.
Copying file://model.h5 [Content-Type=application/octet-stream]...
AccessDeniedException: 403 george***#gmail.com does not have storage.objects.create access to the Google Cloud Storage object.
So the question is, how come Google Storage is not using my service account and keeps using my user account?
UPDATE
After activating the service account for the project as pointed out by #Hao Z, GCP is using my service account now. However, I do have the permissions set for this service account...
UPDATE 2
This seems to be a known issue: https://github.com/GoogleCloudPlatform/gsutil/issues/546
Check How to use Service Accounts with gsutil, for uploading to CS + BigQuery
Relevant bit:
Download service account key file, and put it in e.g. /etc/backup-account.json
gcloud auth activate-service-account --key-file /etc/backup-account.json
Or you can do gsutil -i to impersonate a service account. Use 'gsutil help creds' for more info. I guess the env variable is just used by the Python SDK and not by the CLI.
I was able to resolve this in the following steps:
First, Using the way suggested by #Hao Z above, I was able to activate the service account in Jupyter Notebook using:
!gcloud auth activate-service-account \
prediction-routine-new#prediction-routine-test.iam.gserviceaccount.com \
--key-file=/home/george/Documents/Credentials/prediction-routine-new-b7a445077e61.json \
--project=prediction-routine-test
Second, I changed the bucket name used after realizing that I was using the wrong name - it should be "prediction-routine" instead of "prediction-routine-bucket".
BUCKET_NAME="prediction-routine"
Third, I changed the role from "Storage Object Admmin" to "Storage Admin" for the service account's permissions.

How to I access Security token for Python SDK boto3

I want to access AWS comprehend api from python script. Not getting any leads of how do I remove this error. One thing I know that I have to get session security token.
try:
client = boto3.client(service_name='comprehend', region_name='us-east-1', aws_access_key_id='KEY ID', aws_secret_access_key= 'ACCESS KEY')
text = "It is raining today in Seattle"
print('Calling DetectEntities')
print(json.dumps(client.detect_entities(Text=text, LanguageCode='en'), sort_keys=True, indent=4))
print('End of DetectEntities\n')
except ClientError as e:
print (e)
Error : An error occurred (UnrecognizedClientException) when calling the DetectEntities operation: The security token included in the request is invalid.
This error suggesting that you have provided invalid credentials.
It is also worth nothing that you should never put credentials inside your source code. This can lead to potential security problems if other people obtain access to the source code.
There are several ways to provide valid credentials to an application that uses an AWS SDK (such as boto3).
If the application is running on an Amazon EC2 instance, assign an IAM Role to the instance. This will automatically provide credentials that can be retrieved by boto3.
If you are running the application on your own computer, store credentials in the .aws/credentials file. The easiest way to create this file is with the aws configure command.
See: Credentials — Boto 3 documentation
Create a profile using aws configure or updating ~/.aws/config. If you only have one profile to work with = default, you can omit profile_name parameter from Session() invocation (see example below). Then create AWS service specific client using the session object. Example;
import boto3
session = boto3.session.Session(profile_name="test")
ec2_client = session.client('ec2')
ec2_client.describe_instances()
ec2_resource = session.resource(‘ec2’)
One useful tool I use daily is this: https://github.com/atward/aws-profile/blob/master/aws-profile
This makes assuming role so much easier!
After you set up your access key in .aws/credentials and your .aws/config
you can do something like:
AWS_PROFILE=**you-profile** aws-profile [python x.py]
The part in [] can be substituted with anything that you want to use AWS credentials. e.g., terraform plan
Essentially, this utility simply put your AWS credentials into os environment variables. Then in your boto script, you don't need to worry about setting aws_access_key_id and etc..

Python gcloud clusters get-credentials returns permission error

I'm working on a Python(3.6) project in which I have implemented Google cloud apis, I have setup credentials for Google Cloud via service account.
Here's what I have tried:
Obtain Credentials:
def prepare_credentials(cred_file):
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file(cred_file)
return credentials
And then I have tried to run gcloud container clusters get-credentials $CLUSTER_NAME --zone $ZONE --project $PROJECT
as below:
Run gcloud command via Python's Subprocess:
print(subprocess.call(
'gcloud container clusters get-credentials ' + data['cluster_name'] + ' --zone '
+ data['zone_region']
+ ' --project ' + data['project_id'],
shell=True))
it returns this error:
Fetching cluster endpoint and auth data.
ERROR: (gcloud.container.clusters.get-credentials) ResponseError: code=403, message=Required "container.clusters.get" permission for "projects/brain-183103/zones/europe-west1-d/clusters/numpy".
1
W0629 04:12:00.776926 2222 factory_object_mapping.go:423] Failed to download OpenAPI (Get https://104.197.10.169/swagger-2.0.0.pb-v1: dial tcp 104.197.10.169:443: i/o timeout), falling back to swagger
Unable to connect to the server: dial tcp 104.197.10.169:443: i/o timeout
Service Account credentials are working well as I'm using these credentials for various other Google Cloud API calls and also I have set the owner permission to the project for this service_account.
Have I configured something wrong?
Help me, please!
Thank You,
Abdul
This approach won't work.
Python's subprocess provides a way to run e.g. gcloud commands and communicate between your Python process and e.g. gcloud using I/O (and error) pipes.
In your code, you show the creation of a Python credential object representing your service account and then your run gcloud in a sub-process. There's no credential sharing between these two processes and so gcloud is using whatever it has in context, probably the user's $HOME/.config/gcloud.
It's further complicated by the fact that gcloud container clusters get-credentials performs a bunch of magic with the user's gcloud credentials and the cluster specified, to configure $HOME/.kube/config so that subsequent kubectl commands authenticate using Open ID Connect.
Given that you're using a service account, I assume, because this requires distributing the service account's key to authenticate, that you don't intend to have many|dynamic copies of your code performing this authentication. If that's correct, then you may wish to consider performing the gcloud container cluster get-credentials outside of (and before) your Python code (either manually or via a shell script) and then (once you have a correctly configured $HOME/.kube/config for your cluster), use Kubernetes' Python client (!) to perform whatever work you'd like to do on the cluster.
Although not a solution to this problem specifically, I recommend you use Application Default Credentials (ADCs) in your code. ADCs result in one way to authenticate regardless of whether your code runs off-or-on Google Cloud Platform and regardless of whether you use user or service accounts.
Here's a Python sample using the Cloud Client Library and Application Default Credentials. The library does not (appear) to solve the challenge in effectively converting GCP service account credentials into credentials (via ~/.kube/config) that you may use to access the Kubernetes cluster. But, you can apply operations to clusters using this library:
requirements.txt:
google-cloud-container==0.1.1
python.py:
from google.cloud import container_v1
client = container_v1.ClusterManagerClient()
project_id = "[[YOUR-PROJECT-ID]]"
zone = "[[YOUR-ZONE]]"
cluster_id = "[[YOUR-CLUSTER-ID]]"
cluster = {
"name": cluster_id,
"initial_node_count": 1,
"master_auth":{
#"username": "admin",
"client_certificate_config": {
},
},
#logging_service = "logging.googleapis.com",
#monitoring_service = "monitoring.googleapis.com",
"initial_cluster_version": "1.10.5-gke.3",
}
response = client.create_cluster(project_id, zone, cluster)
response = client.list_clusters(project_id, zone)
response = client.delete_cluster(project_id, zone, cluster_id)
NB username is not present in the dict to disable username|password. client_certificate_config is empty ({}) to not issue a client cert. You may wish to change these. I think there's a bug in the JSON response of the create cluster too that I'll report.

Categories