We Keep Coding

Azure Blob Storage with Python, create containers but not list them?

Azure Blob Storage v12.13.1
Python 3.9.15
I have no problem creating containers...
## Create the container
blob_service_client = BlobServiceClient(account_url=sas_url)
container_client = blob_service_client.create_container(container_name)
but when I go to list them
all_containers = blob_service_client.list_containers()
for i,r in enumerate(all_containers):
print(r)
I get this error...
HttpResponseError: This request is not authorized to perform this operation using this resource type.
RequestId:a04349e6-b01e-0010-58ac-fa6495000000
Appreciate any suggestions!

More than likely you are encountering this error is because your SAS token does not have list (l) permission.
Please try creating a blob service client with a SAS URL that has list permission in it.

Azure BlobServiceClient Error InvalidResourceName

Having problem uploading file to azure blob storage container, using azure.storage.blob for python 2.7. (I know i should use newer python, but it's a part of big ROS application, hence not just so to upgrade it all.)
from azure.storage.blob import BlobServiceClient
...
container_name = "operationinput"
self.back_up_root = "~/backup/sql/lp/"
self.back_up_root = os.path.expanduser(self.back_up_root)
file = 'test.sql'
try:
client = BlobServiceClient.from_connection_string(conn_str=connection_string)
blob = client.get_blob_client(container='container_name', blob='datafile')
except Exception as err:
print(str(err))
with open(self.back_up_root + file, "rb") as data:
blob.upload_blob(data)
I get the following error:
azure.core.exceptions.HttpResponseError: The specifed resource name contains invalid characters.
RequestId:3fcb6c26-101e-007e-596d-1c7d61000000
Time:2022-02-07T21:58:17.1308670Z
ErrorCode:InvalidResourceName
All post i have found refers to people using capital letters or so, but i have:
operationinput
datafile
All should be within specification.
Any ideas?

We have tried with below sample code to upload files to Azure blob storage (Container ) using SAS token , and can able to achieve it successfully.
Code sample:-
from azure.storage.blob import BlobClient
upload_file_path="C:\\Users\\Desktop\\filename"
sas_url="https://xxx.blob.core.windows.nethttps://cloudsh3D?sastoken"
client = BlobClient.from_blob_url(sas_url)
with open(upload_file_path,'rb') as data:
client.upload_blob(data)
print("**file uploaded**")
To generate SAS url and connection string we have selected as below:-
For more information please refer this Microsoft Documentation: Allow or disallow public read access for a storage account

How to get the URL of all the objects in our aws s3 bucket programmatically using python?

I am posting this here because I found it really hard to find the function to get all objects from our s3 bucket using python. When I tried to find get_object_data function, I was directed to downloading the object function.
So, how do we get the data of all the objects in our AWS s3 bucket using boto3(aws sdk for python)?

import boto3 to your python shell
make a connection to your AWS account and specify the resource(s3-bucket here) you want to access?
(make sure that the IAM credentials you are giving have access to that resource)
get the data required
The code looks something like this
import boto3
s3_resource = boto3.resource(service_name='s3',
region_name='<your bucket region>'
aws_access_key_id='<your access key id>'
aws_secret_access_key='<your secret access key>')
a = s3_resource.Bucket('<your bucket name>')
for obj in a.objects.all():
#object URL
print("https://<your bucket name>.s3.<your bucket region>.amazonaws.com/" + obj.key)
#if you want to print all the data of object, just print obj

How do I generate a pre-signed URL in Google Cloud Function that links to a specific bucket GCS?

How do I add the private key file generated with service account to the cloud function to authenticate ?
The following code returns an auth credential error, it says I need a private key to sign credentials. How do I create this and pass in the cloud function?
import datetime
from google.cloud import storage
def generate_download_signed_url_v4(bucket_name, blob_name):
"""Generates a v4 signed URL for downloading a blob.
Note that this method requires a service account key file. You can not use
this if you are using Application Default Credentials from Google Compute
Engine or from the Google Cloud SDK.
"""
# bucket_name = 'your-bucket-name'
# blob_name = 'your-object-name'
storage_client = storage.Client()
bucket = storage_client.bucket(bucket_name)
blob = bucket.blob(blob_name)
url = blob.generate_signed_url(
version="v4",
# This URL is valid for 15 minutes
expiration=datetime.timedelta(minutes=15),
# Allow GET requests using this URL.
method="GET",
)
print("Generated GET signed URL:")
print(url)
print("You can use this URL with any user agent, for example:")
print("curl '{}'".format(url))
return url

I can help you to find information on how to create a signed url in GCF using Python. You need to use Cloud Storage Python Client libraries, as explained in this example on official GCP documentation. Also this example on Medium can help you.

How to access Azure storage account from VM using Python and managed identity / SAS credentials

OBJECTIVE I have an Azure VM set up with a system assigned managed identity. I want to be able to:
Allow users to access blobs inside storage account using the VM
Ensure the users are not able to access the blobs external to the VM
Use Python - most of our users are Python but not Powershell literate.
Setup details:
Storage account: sa030802util. Container: testutils. Blob: hello3.txt
Managed identity and roles. VM has a system assigned managed identity and contributor, storage account contributor, storage blob data contributor roles for sa030802util.
METHODS
I have tried four methods to solve this problem.
Partially successful method 1: Python. In Python, I have been able to access the sa030802util storage account using the below code, derived from link, link and link. The problem is that this uses the storage account and keys rather than relying solely on the managed identity for the VM. My fear is that this leaves the possibility that users could extract the storage keys and gain access to the blobs outside the VM.
Pros: in Python. Con: not using managed identity to authenticate. BlockBlobService can't use MSI to authenticate (yet).
Partially successful method 2: Powershell. In Powershell, I have found two ways to access the blob using the managed identity. The challenge is that neither create a credential that I can easily substitute into Python as explained below. This first method is drawn from the Microsoft-taught Pluralsight course on Implementing Managed Identities for Microsoft Azure Resources (link). It uses the Az module.
Pros: uses managed identity, relatively simple. Cons: not in Python. Does not generate a credential that could be used in Python.
Partially successful method 3: Powershell. This method is drawn from link. It uses the VM managed identity to generate a SAS credential and access Azure Storage.
Pros: uses managed identity and generates SAS credential, which is potentially valuable as BlockBlobService in Python can accept a SAS token. Cons: not in Python. Overkill for Powershell itself given method 2 above achieves the same thing with less effort. I trialled it because I wanted to see if I could extract the SAS credential for use in Python.
Unsuccessful method 4: Python and Powershell. I thought I might be able to generate a SAS token in Powershell using method 3, then slot the token in to the BlockBlobService code from method 1. What I have isn't working. I suspect the reason is that the SAS credential was created for the testutils container, and the Python BlockBlobService needs a SAS credential for the sa030802util storage account.
Pro: would allow me to rely on the managed identity of the VM to access Azure Storage. Con: doesn't work!
QUESTIONS
My questions are:
Am I right in thinking it's better to rely on the VM managed identity and / or SAS credential than the account keys, if I want to make sure that users can only access the storage account inside the VM?
Is there a way to cobble together code that lets me use Python to access the data? Is method 4 promising or a waste of time?
CODE
Method 1: Python
from azure.mgmt.storage import StorageManagementClient
from azure.mgmt.storage.models import StorageAccountCreateParameters
from msrestazure.azure_active_directory import MSIAuthentication
from azure.mgmt.resource import SubscriptionClient
from azure.storage.blob import BlockBlobService
# find credentials and subscription id
credentials = MSIAuthentication()
subscription_client = SubscriptionClient(credentials)
subscription = next(subscription_client.subscriptions.list())
subscription_id = subscription.subscription_id
# find storage keys
storage_client = StorageManagementClient(credentials, subscription_id)
storage_account = storage_client.storage_accounts.get_properties("<resourcegroup>", "sa030802util")
storage_keys = storage_client.storage_accounts.list_keys("<resourcegroup>", "sa030802util")
storage_keys = {v.key_name: v.value for v in storage_keys.keys}
# create BlockBlobService and for e.g. print blobs in container
account_name = "sa030802util"
account_key = storage_keys["key1"]
container_name = "testutils"
block_blob_service = BlockBlobService(account_name = account_name, account_key = account_key)
print("List blobs in container")
generator = block_blob_service.list_blobs(container_name)
for blob in generator:
print("Blob name: " + blob.name)
The output of this code is:
List blobs in container
Blob name: hello3.txt
Method 2: Powershell
Connect-AzAccount -MSI -Subscription <subscriptionid>
$context = New-AzStorageContext -StorageAccountName sa030802util
Get-AzStorageBlob -Name testutils -Context $context
The output of this code is:
Name BlobType Length ContentType LastModified AccessTier SnapshotTime IsDeleted
---- -------- ------ ----------- ------------ ---------- ------------ ---------
hello3.txt BlockBlob 15 application/octet-stream 2019-08-02 05:45:33Z Hot False
Method 3: Powershell
# to get an access token using the VM's identity and use it to call Azure Resource Manager
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Method GET -Headers #{Metadata="true"}
$ content = $response.Content | ConvertFrom-Json
#ArmToken = $content.access_token
# to get SAS credential from Azure Resource Manager to make storage calls
## convert parameters to JSON
$params = #{canonicalizedResource="/blob/sa030802util/testutils"; signedResource="c"; signedPermission="rcwl"; signedProtocol="https"; signedExpiry="2019-08-30T00:00:00Z"}
$jsonParams = $params | ConvertTo-Json
## call storage listServiceSas endpoint to create SAS credential
$sasResponse = Invoke-WebRequest -Uri https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resourceGroup>/providers/Microsoft.Storage/storageAccounts/sa030802util/listServiceSas/?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F -Method POST -Body $jsonParams -Headers #{Authorization = "Bearer $ArmToken"} -UseBasicParsing
## extract SAS credential from response
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
# as example, list contents of container
$context = New-AzStorageContext -StorageAccountName sa030802util -SasToken $sasCred
Get-AzStorageBlob -Name testutils -Context $context
The output of this code is the same as for Method 2.
Method 4: Python and Powershell
Powershell code
# to get an access token using the VM's identity and use it to call Azure Resource Manager
$response = Invoke-WebRequest -Uri 'http://169.254.169.254/metadata/identity/oauth2/token?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F' -Method GET -Headers #{Metadata="true"}
$content = $response.Content | ConvertFrom-Json
$ArmToken = $content.access_token
# to get SAS credential from Azure Resource Manager to make storage calls
## convert parameters to JSON
$params = #{canonicalizedResource="/blob/sa030802util/testutils"; signedResource="c"; signedPermission="rcwl"; signedProtocol="https"; signedExpiry="2019-08-30T00:00:00Z"}
$jsonParams = $params | ConvertTo-Json
## call storage listServiceSas endpoint to create SAS credential
$sasResponse = Invoke-WebRequest -Uri https://management.azure.com/subscriptions/<subscription_id>/resourceGroups/<resourceGroup>/providers/Microsoft.Storage/storageAccounts/sa030802util/listServiceSas/?api-version=2018-02-01&resource=https%3A%2F%2Fmanagement.azure.com%2F -Method POST -Body $jsonParams -Headers #{Authorization = "Bearer $ArmToken"} -UseBasicParsing
## extract SAS credential from response
$sasContent = $sasResponse.Content | ConvertFrom-Json
$sasCred = $sasContent.serviceSasToken
# then export the SAS credential ready to be used in Python
Python code
from azure.storage.blob import BlockBlobService, PublicAccess
import os
# import SAS credential
with open("cred.txt") as f:
line = f.readline()
# create BlockBlobService
block_blob_service = BlockBlobService(account_name = "sa030802util", sas_token=line)
# print content of testutils container
generator = block_blob_service.list_blobs("testutils")
for blob in generator:
print(blob.name)
The Python code returns the following error:
AzureHttpError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature. ErrorCode: AuthenticationFailed
<?xml version="1.0" encoding="utf-8"?><Error><Code>AuthenticationFailed</Code><Message>Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
RequestId:<subscriptionid>
Time:2019-08-05T05:33:40.0175771Z</Message><AuthenticationErrorDetail>Signature did not match. String to sign used was rcwl
2019-08-30T00:00:00.0000000Z
/blob/sa030802util/testutils
https
2018-03-28
</AuthenticationErrorDetail></Error>

Very interesting post, unfortunately I'm not Python expert but this may help :https://github.com/Azure-Samples/resource-manager-python-manage-resources-with-msi
if I want to make sure that users can only access the storage account inside the VM?
You can achieve this without MSI: https://learn.microsoft.com/en-us/azure/storage/common/storage-network-security
MSI does provide an additional layer of security and it also somewhat simplifies management as you don't need to manage keys/SAS tokens but it's not an absolute requirement and you can build secure designs without it.
Good luck!

In the Azure SDK for Python, create a BlobServiceClient then use its get_blob_client method to retrieve a BlobClient class. Then use download_blob on that client to get at the blob contents.
BlobServiceClient takes a credentials argument to which you can pass MSIAuthentication()

You can use azure key vault to store the connection string of the storage account as a secret and retreive the credentials from there to connect to your desired container