I was told to move my bash script that reports on VM backup status, also reports VMs that are not being backed up to Azure automation account. I picked python since Automation Account doesn't have bash and I have done python scripts before for system admin purposes. I am not a python developer, and I need help navigate Azure python SDK classes.
I need to find the "Backup Items" in portal from one of the python SDK modules, to retrieve the VM information from backup vault. I've tried azure.mgmt.recoveryservices and azure.mgmt.recoveryservicesbackup. I can get vault information from azure.mgmt.recoveryservices, which I can use to query more information about vault, hopefully the VM information. My guess is azure.mgmt.recoveryservicesbackup. But I am lost in azure.mgmt.recoveryservicesbackup.jobs.models. I can't tell among hundreds of classes, which one would give that information.
I'd like to use the output from vault backup to against the list of VMs, to find out which ones are not being backed up.
I've looked: https://learn.microsoft.com/en-us/python/api/azure-mgmt-recoveryservicesbackup/azure.mgmt.recoveryservicesbackup.activestamp?view=azure-python https://azure.github.io/azure-sdk/releases/latest/all/python.html, https://www.programcreek.com/python/?ClassName=azure.mgmt.recoveryservicesbackup&submit=Search.
any help would much appreciated!
thanks.
Using python to get the backup of VM
You can use the below code snippet to get VM Backup details.
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.network import NetworkManagementClient
import requests
SUBSCRIPTION_ID = '<Your Subscription ID>'
VM_NAME = '<VM Name>'
credentials = ServicePrincipalCredentials(
client_id='<Client id>',
secret='<Client Secret>',
tenant='<Your Tenent Id>'
)
# Creating base URL
BASE_API_URL = "https://management.azure.com/Subscriptions/<Subscription>/resourceGroups/<Resourece group name>/providers/Microsoft.RecoveryServices/vaults/your_vault_name/backupProtectedItems?api-version=2019-05-13&"
# Add the required filter to fetch the Exact details
customFilter="$filter=backupManagementType eq 'AzureIaasVM' and itemType eq 'VM' and policyName eq 'DailyPolicy'"
#Adding the Base API url with custom filter
BASE_URL = BASE_API_URL + customFilter
header = {
"Authorization": 'Bearer '+ credentials.token["access_token"]
}
response = requests.get(BASE_URL, headers=header)
# here you can handle the response to know the details of backup
print(response.content)
...
Refer here to achieve using Azure cli.
what I was looking for is in "backup_protected_item", in RecoveryServicesBackupClient constructor, here is sample code.
from azure.mgmt.recoveryservicesbackup import RecoveryServicesBackupClient
backup_items = backup_client.backup_protected_items.list(resource_group_name='rg_xxx', vault_name=var_vault)
print(backup_items.properties)
Related
I'm trying to create a simple script that lists out the virtual machines on my Azure US Government account. However, I am faced with this error:
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
VisualStudioCodeCredential: Azure Active Directory error '(invalid_scope) AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://management.azure.com/.default https://management.core.usgovcloudapi.net/.default is not valid. static scope limit exceeded.
This is the code I have used:
def get_access_to_virtual_machine():
subscription_id = key.SUBSCRIPTION_ID
credentials = DefaultAzureCredential(authority = AZURE_US_GOV_CLOUD.endpoints.active_directory,
tenant_id = key.TENANT_ID,
exclude_environment_credential = True,
exclude_managed_identity_credential = True,
exclude_shared_token_cache_credential = True)
compute_client = ComputeManagementClient(credential = credentials,
subscription_id = subscription_id,
base_url = AZURE_US_GOV_CLOUD.endpoints.resource_manager,
credential_scopes = [AZURE_US_GOV_CLOUD.endpoints.active_directory_resource_id + '.default'])
return compute_client
def get_azure_vm(resource_group_name, virtual_machine_name):
compute_client = get_access_to_virtual_machine()
vm_data = compute_client.virtual_machines.get(resource_group_name,
virtual_machine_name,
expand = 'instanceView')
return vm_data
I have signed into my Azure US Government account using Visual Studio as well. The error stems from the compute_client.virtual_machines.get() command. I am 100% sure the credentials I am using are correct but I am really stuck on this. I've tried using ClientSecretCredential instead of DefaultAzureCredential and ran into the same ClientAuthenticationError. In addition, I'm not sure where this scope parameter that the error mentions should be passed in.
For Azure Subscriptions management, the scope should be {management-endpoint}/user_impersonation and not {management-endpoint}/.default. For example, in Azure Commercial the scope will be https://management.azure.com/user_impersonation.
I'm not 100% sure but the management endpoint for Azure Government is either https://management.usgovcloudapi.net/ or https://management.core.usgovcloudapi.net/. Based on the correct endpoint, your scope value should be either https://management.usgovcloudapi.net/user_impersonation or https://management.core.usgovcloudapi.net/user_impersonation.
Please try by changing that.
UPDATE
Looking at the GitHub issue here, it seems there's an issue with the SDK itself. Please try the solution proposed here.
Not sure which version of the Python SDK you have, but I was able to load the latest modules and get the following code to run in the Azure US Government cloud and pull back VM data:
import os
from msrestazure.azure_cloud import AZURE_US_GOV_CLOUD as CLOUD
from azure.mgmt.compute import ComputeManagementClient
from azure.identity import DefaultAzureCredential
subscription_id = 'xxx-xxx-xxx-xxxx'
tenant_id = 'xxxx-xxxx-xxxx-xxxx'
resource_group_name = 'rgName'
vm_name = 'vmName'
credential = DefaultAzureCredential(
authority=CLOUD.endpoints.active_directory,
tenant_id=tenant_id)
compute_client = ComputeManagementClient(
credential, subscription_id,
base_url=CLOUD.endpoints.resource_manager,
credential_scopes=[CLOUD.endpoints.resource_manager + '/.default'])
vm_data = compute_client.virtual_machines.get(
resource_group_name,
vm_name,
expand = 'instanceView')
print(f"{vm_data.name}")
Some things to note:
You had a few of the authentication methods set as excluded, you may want to ensure the method you are expecting is not excluded
The latest SDK sets the environment in the import, I set it to "CLOUD" so that the same code can be used for various cloud by simply changing the import statement
The latest SDK does seem to want '/.default' as part of the credential_scopes
Is there a way to get the email of the Runtime Service Account programmatically from a Cloud Function?
I know I can "guess" the default App Engine account (since it is always <project-id>#appspot.gserviceaccount.com), but this is not what I want.
I was expecting to have some Environment Variable or something with this information, but I was not able to find any.
For older runtimes (Node.js 8, Python 3.7 and Go 1.11), you can use FUNCTION_IDENTITY environment variable, as described here.
Example in Python:
import os
service_account_email = os.environ.get('FUNCTION_IDENTITY')
For newer runtimes, you need to query the Metadata Server, as in this example:
import requests
def query_metadata(entry):
response = requests.get('http://metadata.google.internal/computeMetadata/v1/' + entry, headers={'Metadata-Flavor': 'Google'})
return response.content.decode("utf-8")
service_account_email = query_metadata('instance/service-accounts/default/email')
You can also write a function that supports both runtimes (older and newer) using:
service_account_email = os.environ.get('FUNCTION_IDENTITY') or query_metadata('instance/service-accounts/default/email')
For a list of metadata endpoints that you can use to query for information, check here.
If you're using the newer runtimes, you can get the runtime service account of your Cloud Function programmatically using REST API. Here's an example based from this answer:
import requests
import json
def get_sa():
FUNCTION_NAME = 'func_name'
PROJECT_ID = 'proj_id'
REGION = 'us-central1'
# Get the access token from the metadata server
metadata_server_token_url = 'http://metadata.google.internal/computeMetadata/v1/instance/service-accounts/default/token?scopes=https://www.googleapis.com/auth/cloud-platform'
token_request_headers = {'Metadata-Flavor': 'Google'}
token_response = requests.get(metadata_server_token_url, headers=token_request_headers)
token_response_decoded = token_response.content.decode("utf-8")
access_token = json.loads(token_response_decoded)['access_token']
# Call functions.get() to retrieve Cloud Functions information
response = requests.get('https://cloudfunctions.googleapis.com/v1/projects/{}/locations/{}/functions/{}'.format(PROJECT_ID, REGION, FUNCTION_NAME),
headers={
'Accept': 'application/json',
'Content-Type': 'application/json',
'Authorization': 'Bearer {}'.format(access_token)
})
print(response.json()['serviceAccountEmail'])
get_sa()
Note that you can only get an access token from the metadata server if your app is deployed in GCP (Compute Engine, Cloud Functions, etc.). If your app is on your local machine, you need to use a service account and an auth library to generate the access token. Here's the auth library reference if you're using Python.
To learn more about the given REST API method, see projects.locations.functions.get().
I'm using Azure Python SDK to deploy Azure VM. I can create VM with Network Security Group without any issue via the Azure portal. However, I failed to create a Network Security Group by using API like:
async_nsg_create=network_client.network_security_groups.begin_create_or_update(
GROUP_NAME,
NSG_NAME,
nsg_parameters
)
It always complains that I "does not have authorization to perform action 'Microsoft.Network/networkSecurityGroups/write'".
However, I can create a Network Security Group via the Azure portal by clicking "create a resource" or add new source in Resource Group. I suspect I may have to create NSG via ResourceManagementClient, but I couldn't find any useful info in API doc:https://learn.microsoft.com/en-us/python/api/azure-mgmt-resource/azure.mgmt.resource.resourcemanagementclient?view=azure-python#models-api-version--2020-06-01--
I checked the solution in this issue: enter link description here, but failed at step: resource_client.providers.register('Microsoft.Compute') and it complains:"does not have authorization to perform action 'Microsoft.Compute/register/action'"
The error means your client does not have the permission to do the operations, you need to add it as an RBAC role in your resource group/subscription.
However, I can create a Network Security Group via the Azure portal by clicking "create a resource" or add new source in Resource Group.
In the portal, your are using the account logged in the portal, if you are using the code here, it uses the credentials of the service principal, it is different.
Here is a complete sample works for me, you follow the steps below.
1.Register an application with Azure AD and create a service principal.
2.Get values for signing in and create a new application secret.
3.Navigate to the resource group or the subscription -> Access control (IAM) -> Add -> add service principal of the AD App as an RBAC role e.g. Contributor, details follow this.
4.Then use the code below.
from azure.identity import ClientSecretCredential
from azure.mgmt.network import NetworkManagementClient
from azure.mgmt.network.v2020_06_01.models import NetworkSecurityGroup
from azure.mgmt.network.v2020_06_01.models import SecurityRule
tenant_id = "<tenant-id>"
client_id = "<client-id>"
client_secret = "<client-secret>"
subscription_id = "<subscription-id>"
credential = ClientSecretCredential(tenant_id, client_id, client_secret)
network_client = NetworkManagementClient(credential, subscription_id)
resource_group_name = "<group-name>"
nsg_name = "testnsg"
nsg_params = NetworkSecurityGroup(id= "testnsg", location="UK South", tags={ "name" : "testnsg" })
nsg = network_client.network_security_groups.begin_create_or_update(resource_group_name, "testnsg", parameters=nsg_params)
print(nsg.result().as_dict())
5.Check in the portal:
Update:
If you want to use the user account, you just need to use AzureCliCredential.
1.Install the Azure CLI, then login your account with az login in a local terminal, e.g. powershell.
2.After login, change the code like below and run it.
from azure.identity import ClientSecretCredential
from azure.mgmt.network import NetworkManagementClient
from azure.mgmt.network.v2020_06_01.models import NetworkSecurityGroup
from azure.mgmt.network.v2020_06_01.models import SecurityRule
subscription_id = "<subscription-id>"
credential = AzureCliCredential()
network_client = NetworkManagementClient(credential, subscription_id)
resource_group_name = "<group-name>"
nsg_name = "testnsg"
nsg_params = NetworkSecurityGroup(id= "testnsg", location="UK South", tags={ "name" : "testnsg" })
nsg = network_client.network_security_groups.begin_create_or_update(resource_group_name, "testnsg", parameters=nsg_params)
print(nsg.result().as_dict())
OBJECTIVE I have an Azure VM set up with a system assigned managed identity. I want to be able to:
Allow users to access blobs inside storage account using the VM
Ensure the users are not able to access the blobs external to the VM
Use Python - most of our users are Python but not Powershell literate.
Setup details:
Storage account: sa030802util. Container: testutils. Blob: hello3.txt
Managed identity and roles. VM has a system assigned managed identity and contributor, storage account contributor, storage blob data contributor roles for sa030802util.
METHODS
I have tried four methods to solve this problem.
Partially successful method 1: Python. In Python, I have been able to access the sa030802util storage account using the below code, derived from link, link and link. The problem is that this uses the storage account and keys rather than relying solely on the managed identity for the VM. My fear is that this leaves the possibility that users could extract the storage keys and gain access to the blobs outside the VM.
Pros: in Python. Con: not using managed identity to authenticate. BlockBlobService can't use MSI to authenticate (yet).
Partially successful method 2: Powershell. In Powershell, I have found two ways to access the blob using the managed identity. The challenge is that neither create a credential that I can easily substitute into Python as explained below. This first method is drawn from the Microsoft-taught Pluralsight course on Implementing Managed Identities for Microsoft Azure Resources (link). It uses the Az module.
Pros: uses managed identity, relatively simple. Cons: not in Python. Does not generate a credential that could be used in Python.
Partially successful method 3: Powershell. This method is drawn from link. It uses the VM managed identity to generate a SAS credential and access Azure Storage.
Pros: uses managed identity and generates SAS credential, which is potentially valuable as BlockBlobService in Python can accept a SAS token. Cons: not in Python. Overkill for Powershell itself given method 2 above achieves the same thing with less effort. I trialled it because I wanted to see if I could extract the SAS credential for use in Python.
Unsuccessful method 4: Python and Powershell. I thought I might be able to generate a SAS token in Powershell using method 3, then slot the token in to the BlockBlobService code from method 1. What I have isn't working. I suspect the reason is that the SAS credential was created for the testutils container, and the Python BlockBlobService needs a SAS credential for the sa030802util storage account.
Pro: would allow me to rely on the managed identity of the VM to access Azure Storage. Con: doesn't work!
QUESTIONS
My questions are:
Am I right in thinking it's better to rely on the VM managed identity and / or SAS credential than the account keys, if I want to make sure that users can only access the storage account inside the VM?
Is there a way to cobble together code that lets me use Python to access the data? Is method 4 promising or a waste of time?
CODE
Method 1: Python
from azure.mgmt.storage import StorageManagementClient
from azure.mgmt.storage.models import StorageAccountCreateParameters
from msrestazure.azure_active_directory import MSIAuthentication
from azure.mgmt.resource import SubscriptionClient
from azure.storage.blob import BlockBlobService
# find credentials and subscription id
credentials = MSIAuthentication()
subscription_client = SubscriptionClient(credentials)
subscription = next(subscription_client.subscriptions.list())
subscription_id = subscription.subscription_id
# find storage keys
storage_client = StorageManagementClient(credentials, subscription_id)
storage_account = storage_client.storage_accounts.get_properties("<resourcegroup>", "sa030802util")
storage_keys = storage_client.storage_accounts.list_keys("<resourcegroup>", "sa030802util")
storage_keys = {v.key_name: v.value for v in storage_keys.keys}
# create BlockBlobService and for e.g. print blobs in container
account_name = "sa030802util"
account_key = storage_keys["key1"]
container_name = "testutils"
block_blob_service = BlockBlobService(account_name = account_name, account_key = account_key)
print("List blobs in container")
generator = block_blob_service.list_blobs(container_name)
for blob in generator:
print("Blob name: " + blob.name)
The output of this code is:
List blobs in container
Blob name: hello3.txt
Method 2: Powershell
Connect-AzAccount -MSI -Subscription <subscriptionid>
$context = New-AzStorageContext -StorageAccountName sa030802util
Get-AzStorageBlob -Name testutils -Context $context
The output of this code is:
Name BlobType Length ContentType LastModified AccessTier SnapshotTime IsDeleted
---- -------- ------ ----------- ------------ ---------- ------------ ---------
hello3.txt BlockBlob 15 application/octet-stream 2019-08-02 05:45:33Z Hot False
Method 3: Powershell
# to get an access token using the VM's identity and use it to call Azure Resource Manager
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Method GET -Headers #{Metadata="true"}
$ content = $response.Content | ConvertFrom-Json
#ArmToken = $content.access_token
# to get SAS credential from Azure Resource Manager to make storage calls
## convert parameters to JSON
$params = #{canonicalizedResource="/blob/sa030802util/testutils"; signedResource="c"; signedPermission="rcwl"; signedProtocol="https"; signedExpiry="2019-08-30T00:00:00Z"}
$jsonParams = $params | ConvertTo-Json
## call storage listServiceSas endpoint to create SAS credential
$sasResponse = Invoke-WebRequest -Uri https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resourceGroup>/providers/Microsoft.Storage/storageAccounts/sa030802util/listServiceSas/?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F -Method POST -Body $jsonParams -Headers #{Authorization = "Bearer $ArmToken"} -UseBasicParsing
## extract SAS credential from response
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
# as example, list contents of container
$context = New-AzStorageContext -StorageAccountName sa030802util -SasToken $sasCred
Get-AzStorageBlob -Name testutils -Context $context
The output of this code is the same as for Method 2.
Method 4: Python and Powershell
Powershell code
# to get an access token using the VM's identity and use it to call Azure Resource Manager
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Method GET -Headers #{Metadata="true"}
$content = $response.Content | ConvertFrom-Json
$ArmToken = $content.access_token
# to get SAS credential from Azure Resource Manager to make storage calls
## convert parameters to JSON
$params = #{canonicalizedResource="/blob/sa030802util/testutils"; signedResource="c"; signedPermission="rcwl"; signedProtocol="https"; signedExpiry="2019-08-30T00:00:00Z"}
$jsonParams = $params | ConvertTo-Json
## call storage listServiceSas endpoint to create SAS credential
$sasResponse = Invoke-WebRequest -Uri https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resourceGroup>/providers/Microsoft.Storage/storageAccounts/sa030802util/listServiceSas/?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F -Method POST -Body $jsonParams -Headers #{Authorization = "Bearer $ArmToken"} -UseBasicParsing
## extract SAS credential from response
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
# then export the SAS credential ready to be used in Python
Python code
from azure.storage.blob import BlockBlobService, PublicAccess
import os
# import SAS credential
with open("cred.txt") as f:
line = f.readline()
# create BlockBlobService
block_blob_service = BlockBlobService(account_name = "sa030802util", sas_token=line)
# print content of testutils container
generator = block_blob_service.list_blobs("testutils")
for blob in generator:
print(blob.name)
The Python code returns the following error:
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:<subscriptionid>
Time:2019-08-05T05:33:40.0175771Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcwl
2019-08-30T00:00:00.0000000Z
/blob/sa030802util/testutils
https
2018-03-28
</AuthenticationErrorDetail></Error>
Very interesting post, unfortunately I'm not Python expert but this may help :https://github.com/Azure-Samples/resource-manager-python-manage-resources-with-msi
if I want to make sure that users can only access the storage account inside the VM?
You can achieve this without MSI: https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security
MSI does provide an additional layer of security and it also somewhat simplifies management as you don't need to manage keys/SAS tokens but it's not an absolute requirement and you can build secure designs without it.
Good luck!
In the Azure SDK for Python, create a BlobServiceClient then use its get_blob_client method to retrieve a BlobClient class. Then use download_blob on that client to get at the blob contents.
BlobServiceClient takes a credentials argument to which you can pass MSIAuthentication()
You can use azure key vault to store the connection string of the storage account as a secret and retreive the credentials from there to connect to your desired container
For a Python code base I would like to have developers accessing application secrets using Azure Key Vault, with the idea that when we deploy, the application also should be able to connect. Hence, I'm thinking Active Directory.
However, I can not find any examples on the interweb that show this with the Python SDK. Initially, I would think to retrieve the CLI user:
from azure.common.credentials import get_azure_cli_credentials
credentials, subscription_id, tenant_id = get_azure_cli_credentials(with_tenant=True)
and then use this retrieved set of credentials to access the key vault:
from azure.keyvault import KeyVaultClient
vault_url = "https://########.vault.azure.net/"
secret_name = "########"
secret_version = "########"
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)
However, I retrieve an error that:
azure.keyvault.v7_0.models.key_vault_error_py3.KeyVaultErrorException: Operation returned an invalid status code 'Unauthorized'
I can confirm that credentials, subscription_id and tenant_id are correct, and that using the CLI, I can succesfully retrieve the secret content. So it must be some Python SDK-specific thing.
Any ideas?
It looks like this is a bug in the Python SDK.
https://github.com/Azure/azure-sdk-for-python/issues/5096
You can use your own AD username and password with the UserPassCredentials class. It's not the logged in user, but's it's probably as close as you'll get for now.
EG:
from azure.common.credentials import UserPassCredentials
credentials = UserPassCredentials('username','password')
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)
I tried the same thing and had a different error ("...audience is invalid...") until I changed your first function call adding the resource parameter:
credentials, subscription_id, tenant_id =
get_azure_cli_credentials(resource='https://vault.azure.net', with_tenant=True)
With this change I was able to access secrets using the same code you show.
What about this code snippet? Comparing your code to the example, I don't see where you're setting the client_id or the tenant.
You’ll want to set the access policy for the key vault to allow the authenticated user to access secrets. This can be done in the portal. Bear in mind that key vault has an upper limit of 16 access definitions, so you’ll probably want to grant access to a group and add your users to that group.
As #8forty pointed out, adding a resource='https://vault.azure.net' parameter to your get_azure_cli_credentials call will resolve the issue.
However, there are new packages for working with Key Vault in Python that replace azure-keyvault:
azure-keyvault-certificates (Migration guide)
azure-keyvault-keys (Migration guide)
azure-keyvault-secrets (Migration guide)
azure-identity is also the package that should be used with these for authentication.
If you want to authenticate your Key Vault client with the credentials of the logged in CLI user, you can use the AzureCliCredential class:
from azure.identity import AzureCliCredential
from azure.keyvault.secrets import SecretClient
credential = AzureCliCredential()
vault_url = "https://{vault-name}.vault.azure.net"
secret_name = "secret-name"
client = SecretClient(vault_url, credential)
secret = client.get_secret(secret_name)
print(secret.value)
(I work on the Azure SDK in Python)