We Keep Coding

azure python SDK retrieve backup items from recoverservices(backup)

I was told to move my bash script that reports on VM backup status, also reports VMs that are not being backed up to Azure automation account. I picked python since Automation Account doesn't have bash and I have done python scripts before for system admin purposes. I am not a python developer, and I need help navigate Azure python SDK classes.
I need to find the "Backup Items" in portal from one of the python SDK modules, to retrieve the VM information from backup vault. I've tried azure.mgmt.recoveryservices and azure.mgmt.recoveryservicesbackup. I can get vault information from azure.mgmt.recoveryservices, which I can use to query more information about vault, hopefully the VM information. My guess is azure.mgmt.recoveryservicesbackup. But I am lost in azure.mgmt.recoveryservicesbackup.jobs.models. I can't tell among hundreds of classes, which one would give that information.
I'd like to use the output from vault backup to against the list of VMs, to find out which ones are not being backed up.
I've looked: https://learn.microsoft.com/en-us/python/api/azure-mgmt-recoveryservicesbackup/azure.mgmt.recoveryservicesbackup.activestamp?view=azure-python https://azure.github.io/azure-sdk/releases/latest/all/python.html, https://www.programcreek.com/python/?ClassName=azure.mgmt.recoveryservicesbackup&submit=Search.
any help would much appreciated!
thanks.

Using python to get the backup of VM
You can use the below code snippet to get VM Backup details.
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.resource import ResourceManagementClient
from azure.mgmt.compute import ComputeManagementClient
from azure.mgmt.network import NetworkManagementClient
import requests
SUBSCRIPTION_ID = '<Your Subscription ID>'
VM_NAME = '<VM Name>'
credentials = ServicePrincipalCredentials(
client_id='<Client id>',
secret='<Client Secret>',
tenant='<Your Tenent Id>'
)
# Creating base URL
BASE_API_URL = "https://management.azure.com/Subscriptions/<Subscription>/resourceGroups/<Resourece group name>/providers/Microsoft.RecoveryServices/vaults/your_vault_name/backupProtectedItems?api-version=2019-05-13&"
# Add the required filter to fetch the Exact details
customFilter="$filter=backupManagementType eq 'AzureIaasVM' and itemType eq 'VM' and policyName eq 'DailyPolicy'"
#Adding the Base API url with custom filter
BASE_URL = BASE_API_URL + customFilter
header = {
"Authorization": 'Bearer '+ credentials.token["access_token"]
}
response = requests.get(BASE_URL, headers=header)
# here you can handle the response to know the details of backup
print(response.content)
...
Refer here to achieve using Azure cli.

what I was looking for is in "backup_protected_item", in RecoveryServicesBackupClient constructor, here is sample code.
from azure.mgmt.recoveryservicesbackup import RecoveryServicesBackupClient
backup_items = backup_client.backup_protected_items.list(resource_group_name='rg_xxx', vault_name=var_vault)
print(backup_items.properties)

Using Azure Key Vault and Active Directory to Retrieve Secrets

For a Python code base I would like to have developers accessing application secrets using Azure Key Vault, with the idea that when we deploy, the application also should be able to connect. Hence, I'm thinking Active Directory.
However, I can not find any examples on the interweb that show this with the Python SDK. Initially, I would think to retrieve the CLI user:
from azure.common.credentials import get_azure_cli_credentials
credentials, subscription_id, tenant_id = get_azure_cli_credentials(with_tenant=True)
and then use this retrieved set of credentials to access the key vault:
from azure.keyvault import KeyVaultClient
vault_url = "https://########.vault.azure.net/"
secret_name = "########"
secret_version = "########"
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)
However, I retrieve an error that:
azure.keyvault.v7_0.models.key_vault_error_py3.KeyVaultErrorException: Operation returned an invalid status code 'Unauthorized'
I can confirm that credentials, subscription_id and tenant_id are correct, and that using the CLI, I can succesfully retrieve the secret content. So it must be some Python SDK-specific thing.
Any ideas?

It looks like this is a bug in the Python SDK.
https://github.com/Azure/azure-sdk-for-python/issues/5096
You can use your own AD username and password with the UserPassCredentials class. It's not the logged in user, but's it's probably as close as you'll get for now.
EG:
from azure.common.credentials import UserPassCredentials
credentials = UserPassCredentials('username','password')
client = KeyVaultClient(credentials)
secret = client.get_secret(vault_url, secret_name, secret_version)
print(secret)

I tried the same thing and had a different error ("...audience is invalid...") until I changed your first function call adding the resource parameter:
credentials, subscription_id, tenant_id =
get_azure_cli_credentials(resource='https://vault.azure.net', with_tenant=True)
With this change I was able to access secrets using the same code you show.

What about this code snippet? Comparing your code to the example, I don't see where you're setting the client_id or the tenant.

You’ll want to set the access policy for the key vault to allow the authenticated user to access secrets. This can be done in the portal. Bear in mind that key vault has an upper limit of 16 access definitions, so you’ll probably want to grant access to a group and add your users to that group.

As #8forty pointed out, adding a resource='https://vault.azure.net' parameter to your get_azure_cli_credentials call will resolve the issue.
However, there are new packages for working with Key Vault in Python that replace azure-keyvault:
azure-keyvault-certificates (Migration guide)
azure-keyvault-keys (Migration guide)
azure-keyvault-secrets (Migration guide)
azure-identity is also the package that should be used with these for authentication.
If you want to authenticate your Key Vault client with the credentials of the logged in CLI user, you can use the AzureCliCredential class:
from azure.identity import AzureCliCredential
from azure.keyvault.secrets import SecretClient
credential = AzureCliCredential()
vault_url = "https://{vault-name}.vault.azure.net"
secret_name = "secret-name"
client = SecretClient(vault_url, credential)
secret = client.get_secret(secret_name)
print(secret.value)
(I work on the Azure SDK in Python)

Blob.generate_signed_url() failing to AttributeError

So I'm trying to produce temporary globally readable URLs for my Google Cloud Storage objects using the google-cloud-storage Python library (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) - more specifically the Blob.generate_signed_url() method. I doing this from within a Compute Engine instance in a command line Python script. And I keep getting the following error:
AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'oauth2cl
ient.service_account.ServiceAccountCredentials'> just contains a token. see https://google-cloud-python.readthedocs
.io/en/latest/core/auth.html?highlight=authentication#setting-up-a-service-account for more details.
I am aware that there are issues with doing this from within GCE (https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/50) but I have created a new Service Account credentials following the instructions here: https://cloud.google.com/storage/docs/access-control/create-signed-urls-program and my key.json file most certainly includes a private key. Still I am seeing that error.
This is my code:
keyfile = "/path/to/my/key.json"
credentials = ServiceAccountCredentials.from_json_keyfile_name(keyfile)
expiration = timedelta(3) # valid for 3 days
url = blob.generate_signed_url(expiration, method="GET",
credentials=credentials)
I've read through the issue tracker here https://github.com/GoogleCloudPlatform/google-cloud-python/issues?page=2&q=is%3Aissue+is%3Aopen and nothing related jumps out so I am assuming this should work. Cannot see what's going wrong here.

I was having the same issue. Ended up fixing it by starting the storage client directly from the service account json.
storage_client = storage.Client.from_service_account_json('path_to_service_account_key.json')
I know I'm late to the party but hopefully this helps!

Currently, it's not possible to use blob.generate_signed_url without explicitly referencing credentials. (Source: Google-Cloud-Python documentation) However, you can do a workaround, as seen here, which consists of:
signing_credentials = compute_engine.IDTokenCredentials(
auth_request,
"",
service_account_email=credentials.service_account_email
)
signed_url = signed_blob_path.generate_signed_url(
expires_at_ms,
credentials=signing_credentials,
version="v4"
)

A much complete snippet for those asking where other elements come from. cc #AlbertoVitoriano
from google.auth.transport import requests
from google.auth import default, compute_engine
credentials, _ = default()
# then within your abstraction
auth_request = requests.Request()
credentials.refresh(auth_request)
signing_credentials = compute_engine.IDTokenCredentials(
auth_request,
"",
service_account_email=credentials.service_account_email
)
signed_url = signed_blob_path.generate_signed_url(
expires_at_ms,
credentials=signing_credentials,
version="v4"
)

Python Azure sdk: How to retrieve secrets from keyvault?

I need to retrieve secrets from keyvault. This is my code so far:
from azure.mgmt.keyvault import KeyVaultManagementClient
from azure.common.credentials import ServicePrincipalCredentials
subscription_id = 'x'
# See above for details on creating different types of AAD credentials
credentials = ServicePrincipalCredentials(
client_id = 'x',
secret = 'x',
tenant = 'x'
)
kv_client = KeyVaultManagementClient(credentials, subscription_id)
for vault in kv_client.vaults.list():
print(vault)
But I am getting this error:
msrestazure.azure_exceptions.CloudError: Azure Error:
AuthorizationFailed Message: The client 'x' with object id 'x' does
not have authorization to perform action
'Microsoft.Resources/subscriptions/resources/read' over scope
'/subscriptions/x'.
Now I am able to access the same keyvault with same credentials using C# code/ POwershell so there is definitely nothing wrong with authorization. Not sure why it isnt working using SDK. Please help.

If you are looking to access via a ServicePrincipalCredentials instance, you can just use:
from azure.keyvault import KeyVaultClient, KeyVaultAuthentication
from azure.common.credentials import ServicePrincipalCredentials
credentials = None
def auth_callback(server, resource, scope):
credentials = ServicePrincipalCredentials(
client_id = '',
secret = '',
tenant = '',
resource = "https://vault.azure.net"
)
token = credentials.token
return token['token_type'], token['access_token']
client = KeyVaultClient(KeyVaultAuthentication(auth_callback))
secret_bundle = client.get_secret("https://vault_url", "secret_id", "")
print(secret_bundle.value)
This assumes that you don't want to pass a version. If you do, you can substitute the last parameter for it.

I run your code sample above and it is able to list the key vaults without any issue, hence it is not a code issue.
I have assigned the Contributor role to my AD application on the subscription where the key vault is provisioned and set the Access Policies to allow GET & LIST permissions for Key and Secret to the AD application.
The versions of my Azure Python packages used running under Python 3.6.2 runtime environment:
azure.common (1.1.8)
azure.mgmt.keyvault (0.40.0)
msrestazure(0.4.13)
I'll recommend you to try on the Python runtime version and Azure Python packages versions which is verified working.
Addendum:
If the above Python runtime environment version as well as Azure Python packages also does not work for you, you should probably consider creating a new issue in the Azure SDK for Python GitHub as it is working with the same credential with Azure .NET SDK as well as PowerShell.

You can also get secret by the name of the secret instead of ID:
secret_bundle = client.get_secret(<VAULT URL>, "<NAME>", "")

There are some good answers already, but the Azure SDK has since released new packages for working with Key Vault in Python that replace azure-keyvault:
azure-keyvault-certificates (Migration guide)
azure-keyvault-keys (Migration guide)
azure-keyvault-secrets (Migration guide)
azure-identity is also the package that should be used with these for authentication.
Documentation for working with the secrets library can be found on the azure-sdk-for-python GitHub repository, and here's a sample for retrieving secrets as you were doing:
from azure.identity import DefaultAzureCredential
from azure.keyvault.secrets import SecretClient
credential = DefaultAzureCredential()
secret_client = SecretClient(
vault_url="https://my-key-vault.vault.azure.net/",
credential=credential
)
secret = secret_client.get_secret("secret-name")
You can provide the same credentials that you used for ServicePrincipalCredentials by setting environment variables corresponding to the client_id, secret, and tenant:
export AZURE_CLIENT_ID="client_id"
export AZURE_CLIENT_SECRET="secret"
export AZURE_TENANT_ID="tenant"
(I work on the Azure SDK in Python)

One can use the below class from azure.identity i.e ClientSecretCredential, find the below code ex: snippet
from azure.identity import ClientSecretCredential
from azure.keyvault.secrets import SecretClient
TENANT= <TenantId-in-string>
CLIENT_ID = <ClientId-in-string>
CLIENT_SECRET= <ClientSecret-in-string>
credential = ClientSecretCredential(TENANT,CLIENT_ID,CLIENT_SECRET)
VAULT_URL= <AzureVault-url-in-string>
client = SecretClient(vault_url=VAULT_URL, credential=credential)
print(client)
example_secret = client.get_secret(<secret_name_in_string>)
print(example_secret.value)

Azure python sdk - getting the machine state

Using the python api for azure, I want to get the state of one of my machines.
I can't find anywhere to access this information.
Does someone know?
After looking around, I found this:
get_with_instance_view(resource_group_name, vm_name)
https://azure-sdk-for-python.readthedocs.org/en/latest/ref/azure.mgmt.compute.computemanagement.html#azure.mgmt.compute.computemanagement.VirtualMachineOperations.get_with_instance_view

if you are using the legacy api (this will work for classic virtual machines), use
from azure.servicemanagement import ServiceManagementService
sms = ServiceManagementService('your subscription id', 'your-azure-certificate.pem')
your_deployment = sms.get_deployment_by_name('service name', 'deployment name')
for role_instance in your_deployment.role_instance_list:
print role_instance.instance_name, role_instance.instance_status
if you are using the current api (will not work for classic vm's), use
from azure.common.credentials import UserPassCredentials
from azure.mgmt.compute import ComputeManagementClient
import retry
credentials = UserPassCredentials('username', 'password')
compute_client = ComputeManagementClient(credentials, 'your subscription id')
#retry.retry(RuntimeError, tries=3)
def get_vm(resource_group_name, vm_name):
'''
you need to retry this just in case the credentials token expires,
that's where the decorator comes in
this will return all the data about the virtual machine
'''
return compute_client.virtual_machines.get(
resource_group_name, vm_name, expand='instanceView')
#retry.retry((RuntimeError, IndexError,), tries=-1)
def get_vm_status(resource_group_name, vm_name):
'''
this will just return the status of the virtual machine
sometime the status may be unknown as shown by the azure portal;
in that case statuses[1] doesn't exist, hence retrying on IndexError
also, it may take on the order of minutes for the status to become
available so the decorator will bang on it forever
'''
return compute_client.virtual_machines.get(resource_group_name, vm_name, expand='instanceView').instance_view.statuses[1].display_status

If you are using Azure Cloud Services, you should use the Role Environment API, which provides state information regarding the current instance of your current service instance.
https://msdn.microsoft.com/en-us/library/azure/microsoft.windowsazure.serviceruntime.roleenvironment.aspx

In the new API resource manager
There's a function:
get_with_instance_view(resource_group_name, vm_name)
It's the same function as get machine, but it also returns an instance view that contains the machine state.
https://azure-sdk-for-python.readthedocs.org/en/latest/ref/azure.mgmt.compute.computemanagement.html#azure.mgmt.compute.computemanagement.VirtualMachineOperations.get_with_instance_view

Use this method get_deployment_by_name to get the instances status:
subscription_id = '****-***-***-**'
certificate_path = 'CURRENT_USER\\my\\***'
sms = ServiceManagementService(subscription_id, certificate_path)
result=sms.get_deployment_by_name("your service name","your deployment name")
You can get instance status via "instance_status" property.
Please see this post https://stackoverflow.com/a/31404545/4836342

As mentioned in other answers the Azure Resource Manager API has an instance view query to show the state of running VMs.
The documentation listing for this is here: VirtualMachineOperations.get_with_instance_view()
Typical code to get the status of a VM is something like this:
resource_group = "myResourceGroup"
vm_name = "myVMName"
creds = azure.mgmt.common.SubscriptionCloudCredentials(…)
compute_client = azure.mgmt.compute.ComputeManagementClient(creds)
vm = compute_client.virtual_machines.get_with_instance_view(resource_group, vm_name).virtual_machine
# Index 0 is the ProvisioningState, index 1 is the Instance PowerState, display_status will typically be "VM running, VM stopped, etc.
vm_status = vm.instance_view.statuses[1].display_status

There is no direct way to get the state of a virtual machine while listing them.
But, we can list out the vms by looping into them to get the instance_view of a machine and grab its power state.
In the code block below, I am doing the same and dumping the values into a .csv file to make a report.
import csv
from azure.common.credentials import ServicePrincipalCredentials
from azure.mgmt.compute import ComputeManagementClient
def get_credentials():
subscription_id = "*******************************"
credential = ServicePrincipalCredentials(
client_id="*******************************",
secret="*******************************",
tenant="*******************************"
)
return credential, subscription_id
credentials, subscription_id = get_credentials()
# Initializing compute client with the credentials
compute_client = ComputeManagementClient(credentials, subscription_id)
resource_group_name = "**************"
json_list = []
json_object = {"Vm_name": "", "Vm_state": "", "Resource_group": resource_group_name}
# listing out the virtual machine names
vm_list = compute_client.virtual_machines.list(resource_group_name=resource_group_name)
# looping inside the list of virtual machines, to grab the state of each machine
for i in vm_list:
vm_state = compute_client.virtual_machines.instance_view(resource_group_name=resource_group_name, vm_name=i.name)
json_object["Vm_name"] = i.name
json_object["Vm_state"] = vm_state.statuses[1].code
json_list.append(json_object)
csv_columns = ["Vm_name", "Vm_state", "Resource_group"]
f = open("vm_state.csv", 'w+')
csv_file = csv.DictWriter(f, fieldnames=csv_columns)
csv_file.writeheader()
for i in json_list:
csv_file.writerow(i)
To grab the state of a single virtual machine, where you know its resource_group_name and vm_name, just use the block below.
vm_state = compute_client.virtual_machines.instance_view(resource_group_name="foo_rg_name", vm_name="foo_vm_name")
power_state = vm_state.statuses[1].code
print(power_state)

As per the new API reference, this worked for me
vm_status = compute_client.virtual_machines.instance_view(GROUP_NAME, VM_NAME).statuses[1].code
it will return any one of these states, based on the current state
"PowerState/stopped", "PowerState/running","PowerState/stopping", "PowerState/starting"