We Keep Coding

Invalid scope for ComputeManagementClient for Azure US Government account?

I'm trying to create a simple script that lists out the virtual machines on my Azure US Government account. However, I am faced with this error:
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
VisualStudioCodeCredential: Azure Active Directory error '(invalid_scope) AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://management.azure.com/.default https://management.core.usgovcloudapi.net/.default is not valid. static scope limit exceeded.
This is the code I have used:
def get_access_to_virtual_machine():
subscription_id = key.SUBSCRIPTION_ID
credentials = DefaultAzureCredential(authority = AZURE_US_GOV_CLOUD.endpoints.active_directory,
tenant_id = key.TENANT_ID,
exclude_environment_credential = True,
exclude_managed_identity_credential = True,
exclude_shared_token_cache_credential = True)
compute_client = ComputeManagementClient(credential = credentials,
subscription_id = subscription_id,
base_url = AZURE_US_GOV_CLOUD.endpoints.resource_manager,
credential_scopes = [AZURE_US_GOV_CLOUD.endpoints.active_directory_resource_id + '.default'])
return compute_client
def get_azure_vm(resource_group_name, virtual_machine_name):
compute_client = get_access_to_virtual_machine()
vm_data = compute_client.virtual_machines.get(resource_group_name,
virtual_machine_name,
expand = 'instanceView')
return vm_data
I have signed into my Azure US Government account using Visual Studio as well. The error stems from the compute_client.virtual_machines.get() command. I am 100% sure the credentials I am using are correct but I am really stuck on this. I've tried using ClientSecretCredential instead of DefaultAzureCredential and ran into the same ClientAuthenticationError. In addition, I'm not sure where this scope parameter that the error mentions should be passed in.

For Azure Subscriptions management, the scope should be {management-endpoint}/user_impersonation and not {management-endpoint}/.default. For example, in Azure Commercial the scope will be https://management.azure.com/user_impersonation.
I'm not 100% sure but the management endpoint for Azure Government is either https://management.usgovcloudapi.net/ or https://management.core.usgovcloudapi.net/. Based on the correct endpoint, your scope value should be either https://management.usgovcloudapi.net/user_impersonation or https://management.core.usgovcloudapi.net/user_impersonation.
Please try by changing that.
UPDATE
Looking at the GitHub issue here, it seems there's an issue with the SDK itself. Please try the solution proposed here.

Not sure which version of the Python SDK you have, but I was able to load the latest modules and get the following code to run in the Azure US Government cloud and pull back VM data:
import os
from msrestazure.azure_cloud import AZURE_US_GOV_CLOUD as CLOUD
from azure.mgmt.compute import ComputeManagementClient
from azure.identity import DefaultAzureCredential
subscription_id = 'xxx-xxx-xxx-xxxx'
tenant_id = 'xxxx-xxxx-xxxx-xxxx'
resource_group_name = 'rgName'
vm_name = 'vmName'
credential = DefaultAzureCredential(
authority=CLOUD.endpoints.active_directory,
tenant_id=tenant_id)
compute_client = ComputeManagementClient(
credential, subscription_id,
base_url=CLOUD.endpoints.resource_manager,
credential_scopes=[CLOUD.endpoints.resource_manager + '/.default'])
vm_data = compute_client.virtual_machines.get(
resource_group_name,
vm_name,
expand = 'instanceView')
print(f"{vm_data.name}")
Some things to note:
You had a few of the authentication methods set as excluded, you may want to ensure the method you are expecting is not excluded
The latest SDK sets the environment in the import, I set it to "CLOUD" so that the same code can be used for various cloud by simply changing the import statement
The latest SDK does seem to want '/.default' as part of the credential_scopes

Create linked service with key Vault using python

Here is my problem, I am trying to create linked service using python sdk and I was successful if I provide the storage account name and key. But I would like to create Linked service with key vaults reference, the below runs fine and creates the linked service. However when I go to datafactory and test connection.. it fails.. Please help!
store = LinkedServiceReference(reference_name ='LS_keyVault_Dev')
storage_string = AzureKeyVaultSecretReference( store=store, secret_name = 'access_key')
ls_azure_storage = AzureStorageLinkedService(connection_string=storage_string)
ls = adf_client.linked_services.create_or_update(rg_name, df_name, ls_name, ls_azure_storage)
Error Message
Invalid storage connection string provided to 'AzureTableConnection'. Check the storage connection string in configuration. No valid combination of account information found.

I test your code, it created the linked service successfully, and I navigate to the portal to Test connection, it also works, you could follow the steps below.
1.Navigate to the azure keyvault in the portal -> Secrets -> Create a secret, I'm not sure why you can use access_key as the name of the secret, pey my test, it is invalid. So in my sample, I use accesskey as the name of the secret, then store the Connection string of the storage account.
2.Navigate to the Access policies of the keyvault, add the MSI of your data factory with correct secret permission. If you did not enable the MSI of the data factory, follow this link to generate it, this is used to for the Azure Key Vault linked service to access your keyvault secret.
3.Navigate to the Azure Key Vault linked service of your data factory, make sure the connection is successful.
4.Use the code below to create the storage linked service.
Version of the libraries:
azure-common==1.1.23
azure-mgmt-datafactory==0.9.0
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.datafactory import DataFactoryManagementClient
from azure.mgmt.datafactory.models import *
subscription_id = '<subscription-id>'
credentials = ServicePrincipalCredentials(client_id='<client-id>', secret='<client-secret>', tenant='<tenant-id>')
adf_client = DataFactoryManagementClient(credentials, subscription_id)
rg_name = '<resource-group-name>'
df_name = 'joyfactory'
ls_name = 'storageLinkedService'
store = LinkedServiceReference(reference_name ='AzureKeyVault1') # AzureKeyVault1 is the name of the Azure Key Vault linked service
storage_string = AzureKeyVaultSecretReference( store=store, secret_name = 'accesskey')
ls_azure_storage = AzureStorageLinkedService(connection_string=storage_string)
ls = adf_client.linked_services.create_or_update(rg_name, df_name, ls_name, ls_azure_storage)
print(ls)
5.Go back to the linked service page, refresh and test the connection, it works fine.

How to list Azure vms using python?

I am trying to list Azure VMs using python code. could someone help me with this issue?
I have already tried to go through the code on the Microsoft website but it is not clear to me.

First, you need to follow the section Register your client application with Azure AD of Azure offical document Azure REST API Reference to register an application with Azure AD on Azure portal for getting the required parameters client_id and secret for Authentication to list VMs.
And, you continue to get the other required parameters subscription_id and tenant_id.
Then, to create a virtual environment in Python to install Azure SDK for Python via pip install azure or just install Azure Compute Management for Python via pip install azure-mgmt-compute.
Here is my sample code.
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.compute import ComputeManagementClient
credentials = ServicePrincipalCredentials(
client_id='<your client id>',
secret='<your client secret>',
tenant='<your tenant id>'
)
subscription_id = '<your subscription id>'
client = ComputeManagementClient(credentials, subscription_id)
If just list VMs by resource group, to use function list(resource_group_name, custom_headers=None, raw=False, **operation_config) as the code below.
resource_group_name = '<your resource group name>'
vms_by_resource_group = client.virtual_machines.list(resource_group_name)
Or you want to list all VMs in your subscription, to use function list_all(custom_headers=None, raw=False, **operation_config) as the code below.
all_vms = client.virtual_machines.list_all()
As references, there are two SO threads I think which may help for understanding deeper: How to get list of Azure VMs (non-classic/Resource Managed) using Java API and Is it anyway to get ftpsState of azure web app (Azure Function app) using Python SDK?.
Hope it helps.

Using Azure Key Vault and Active Directory to Retrieve Secrets

For a Python code base I would like to have developers accessing application secrets using Azure Key Vault, with the idea that when we deploy, the application also should be able to connect. Hence, I'm thinking Active Directory.
However, I can not find any examples on the interweb that show this with the Python SDK. Initially, I would think to retrieve the CLI user:
from azure.common.credentials import get_azure_cli_credentials
credentials, subscription_id, tenant_id = get_azure_cli_credentials(with_tenant=True)
and then use this retrieved set of credentials to access the key vault:
from azure.keyvault import KeyVaultClient
vault_url = "https://########.vault.azure.net/"
secret_name = "########"
secret_version = "########"
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)
However, I retrieve an error that:
azure.keyvault.v7_0.models.key_vault_error_py3.KeyVaultErrorException: Operation returned an invalid status code 'Unauthorized'
I can confirm that credentials, subscription_id and tenant_id are correct, and that using the CLI, I can succesfully retrieve the secret content. So it must be some Python SDK-specific thing.
Any ideas?

It looks like this is a bug in the Python SDK.
https://github.com/Azure/azure-sdk-for-python/issues/5096
You can use your own AD username and password with the UserPassCredentials class. It's not the logged in user, but's it's probably as close as you'll get for now.
EG:
from azure.common.credentials import UserPassCredentials
credentials = UserPassCredentials('username','password')
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)

I tried the same thing and had a different error ("...audience is invalid...") until I changed your first function call adding the resource parameter:
credentials, subscription_id, tenant_id =
get_azure_cli_credentials(resource='https://vault.azure.net', with_tenant=True)
With this change I was able to access secrets using the same code you show.

What about this code snippet? Comparing your code to the example, I don't see where you're setting the client_id or the tenant.

You’ll want to set the access policy for the key vault to allow the authenticated user to access secrets. This can be done in the portal. Bear in mind that key vault has an upper limit of 16 access definitions, so you’ll probably want to grant access to a group and add your users to that group.

As #8forty pointed out, adding a resource='https://vault.azure.net' parameter to your get_azure_cli_credentials call will resolve the issue.
However, there are new packages for working with Key Vault in Python that replace azure-keyvault:
azure-keyvault-certificates (Migration guide)
azure-keyvault-keys (Migration guide)
azure-keyvault-secrets (Migration guide)
azure-identity is also the package that should be used with these for authentication.
If you want to authenticate your Key Vault client with the credentials of the logged in CLI user, you can use the AzureCliCredential class:
from azure.identity import AzureCliCredential
from azure.keyvault.secrets import SecretClient
credential = AzureCliCredential()
vault_url = "https://{vault-name}.vault.azure.net"
secret_name = "secret-name"
client = SecretClient(vault_url, credential)
secret = client.get_secret(secret_name)
print(secret.value)
(I work on the Azure SDK in Python)

how to obtain GCR access token with python / listing docker images

The access token im getting with gcloud auth print-access-token is obviously a different access token than the one i can get with some basic python code:
export GOOGLE_APPLICATION_CREDENTIALS=/the-credentials.json
from oauth2client.client import GoogleCredentials
credentials = GoogleCredentials.get_application_default()
credentials.get_access_token()
What i am trying to do is get a token that would work with:
curl -u _token:<mytoken> https://eu.gcr.io/v2/my-project/my-docker-image/tags/list
I'd prefer not to install gcloud utility as a dependency for my app, hence my tries to obtain the access token progrmatically via oath google credentials

I know this is a very old question, but I just got faced with the exact same problem of requiring an ACCESS_TOKEN in Python and not being able to generate it, and managed to make it work.
What you need to do is to use the variable credentials.token, except it won't exist once you first create the credentials object, returning None. In order to generate a token, the credentials must be used by a google.cloud library, which in my case was done by using the googleapiclient.discovery.build method:
sqladmin = googleapiclient.discovery.build('sqladmin', 'v1beta4', credentials=credentials)
response = sqladmin.instances().get(project=PROJECT_ID, instance=INSTANCE_ID).execute()
print(json.dumps(response))
After which the ACCESS_TOKEN could be properly generated using
access_token = credentials.token
I've also tested it using google.cloud storage as a way to test credentials, and it also worked, by just trying to access a bucket in GCS through the appropriate Python library:
from google.oauth2 import service_account
from google.cloud import storage
PROJECT_ID = your_project_id_here
SCOPES = ['https://www.googleapis.com/auth/sqlservice.admin']
SERVICE_ACCOUNT_FILE = '/path/to/service.json'
credentials = service_account.Credentials.from_service_account_file(
SERVICE_ACCOUNT_FILE, scopes=SCOPES)
try:
list(storage.Client(project=PROJECT_ID, credentials=credentials).bucket('random_bucket').list_blobs())
except:
print("Failed because no bucket exists named 'random_bucket' in your project... but that doesn't matter, what matters is that the library tried to use the credentials and in doing so generated an access_token, which is what we're interested in right now")
access_token = credentials.token
print(access_token)

So I think there are a few questions:
gcloud auth print-access-token vs GoogleCredentials.get_application_default()
gcloud doesn't set application default credentials by default anymore when performing a gcloud auth login, so the access_token you're getting from gcloud auth print-access-token is going to be the one corresponding to the used you used to login.
As long as you follow the instructions to create ADC's for a service account, that account has the necessary permissions, and the environment from which you are executing the script has access to the ENV var and the adc.json file, you should be fine.
How to make curl work
The Docker Registry API specifies that a token exchange should happen, swapping your Basic auth (i.e. Authorization: Basic base64(_token:<gcloud_access_token>)) for a short-lived Bearer token. This process can be a bit involved, but is documented here under "How to authenticate" and "Requesting a Token". Replace auth.docker.io/token with eu.gcr.io/v2/token and service=registry.docker.io with service=eu.gcr.io, etc. Use curl -u oauth2accesstoken:<mytoken> here.
See also: How to list images and tags from the gcr.io Docker Registry using the HTTP API?
Avoid the question entirely
We have a python lib that might be relevant to your needs:
https://github.com/google/containerregistry