We Keep Coding

How does client= translate.TranslationServiceClient() work in conjunction with os.environ['GOOGLE_APPLICATION_CREDENTIALS']

I am using python and azure function app to send a document to be translated using the google cloud translation api.
I am trying to load the credentials from a tempfile (json) using the below code. The idea is to later download the json file from blob storage and store it in a temp file but I am not thinking about the blob storage for now.
key= {cred info}
f= tempfile.NamedTemporaryFile(suffix='.json', mode='a+')
json.dump(key, f)
f.flush()
f.seek(0)
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = f.name
client= translate.TranslationServiceClient()
But when I run this I get the following error:
Exception: PermissionError: [Errno 13] Permission denied:
How can I correctly load the creds from a temp file?. Also what is the relationship between translate.TranslationServiceClient() and os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = f.name? Does the TranslationServiceClient() get the creds from the environment variable?
I have been looking at this problem for a while now and I cannot find a good solution. Any help would be amazing!
edit:
when I change it to
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = f.read()
I get a different error:
System.Private.CoreLib: Exception while executing function:
Functions.Trigger. System.Private.CoreLib: Result: Failure
Exception: DefaultCredentialsError:
EDIT 2:
Its really weird, but it works when I read the file just before like so:
contents= f.read()
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = f.name
client= translate.TranslationServiceClient()
Any ideas why?

Any application which connects to any GCP Product requires credentials to authenticate. Now there are many ways how this authentication works.
According to the Google doc
Additionally, we recommend you use Google Cloud Client Libraries for your application. Google Cloud Client Libraries use a library called Application Default Credentials (ADC) to automatically find your service account credentials. ADC looks for service account credentials in the following order:
If the environment variable GOOGLE_APPLICATION_CREDENTIALS is set, ADC uses the service account key or configuration file that the variable points to.
If the environment variable GOOGLE_APPLICATION_CREDENTIALS isn't set, ADC uses the service account that is attached to the resource that is running your code.
This service account might be a default service account provided by Compute Engine, Google Kubernetes Engine, App Engine, Cloud Run, or Cloud Functions. It might also be a user-managed service account that you created.
If ADC can't use any of the above credentials, an error occurs.
There are also modules provided by Google that can be used to pass the credentials.
If you already have the JSON value as dictionary then you can simply pass dictionary in from_service_account_info(key)
Example:
key = json.load(open("JSON File Path")) # loading my JSON file into dictionary
client = translate.TranslationServiceClient().from_service_account_info(key)
In your case you already have the key as dictionary
As for the error you are getting, I believe that has to be something with the temp file. Because GOOGLE_APPLICATION_CREDENTIALS needs full access to the JSON file path to read from it.

List files on Azure BlobStorage with python API

This code tries to list the files in the in a blob storage:
#!/usr/bin/env python3
import os
from azure.storage.blob import BlobServiceClient, BlobClient, ContainerClient, __version__
from datetime import datetime, timedelta
import azure.cli.core as az
print(f"Azure Blob storage v{__version__} - Python quickstart sample")
account_name = "my_account"
container_name = "my_container"
path_on_datastore = "test/path"
def _create_sas(expire=timedelta(seconds=10)) -> str:
cli = az.get_default_cli()
expire_date = datetime.utcnow() + expire
expiry_string = datetime.strftime(expire_date, "%Y-%m-%dT%H:%M:%SZ")
cmd = ["storage", "container", "generate-sas", "--name", container_name, "--account-name",
account_name, "--permissions", "lr", "--expiry", expiry_string, "--auth-mode", "login", "--as-user"]
if cli.invoke(cmd) != 0:
raise RuntimeError("Could not receive a SAS token for user {}#{}".format(
account_name, container_name))
return cli.result.result
sas = _create_sas()
blob_service_client = BlobServiceClient(
account_url=f"{account_name}.blob.core.windows.net", container_name=container_name, credential=sas)
container_client = blob_service_client.create_container(container_name)
blob_list = container_client.list_blobs()
for blob in blob_list:
print("\t" + blob.name)
That code worked quite fine a few weeks ago, but then we always get the error:
azure.core.exceptions.ClientAuthenticationError: Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
Does someone know what can be wrong?
PS. using Azure blob storage package of version 12.3.2.
[Edit]
Because of security concerns we are not allowed to use account keys here.

I'm not entirely sure what is wrong with your code, but it looks like your SAS token is not the expected format. Have you tested if the the SAS URL works in a browser?
Additionally, your _create_sas function seems to be creating the SAS signature with an Azure CLI command. I don't think you need to do this because the azure-storage-blob package has methods such as generate_account_sas to generate a SAS signature. This will eliminate a lot of complexity because you don't need to worry about the SAS signature format.
from datetime import datetime, timedelta
from azure.storage.blob import (
BlobServiceClient,
generate_account_sas,
ResourceTypes,
AccountSasPermissions,
)
from azure.core.exceptions import ResourceExistsError
account_name = "<account name>"
account_url = f"https://{account_name}.blob.core.windows.net"
container_name = "<container name>"
# Create SAS token credential
sas_token = generate_account_sas(
account_name=account_name,
account_key="<account key>",
resource_types=ResourceTypes(container=True),
permission=AccountSasPermissions(read=True, write=True, list=True),
expiry=datetime.utcnow() + timedelta(hours=1),
)
Which gives this SAS signature read, write and list permissions on blob containers, with an expiry time of 1 hour. You can change this to your liking.
We can then create the BlobServiceClient with this SAS signature as a credential, then create the container client to list the blobs.
# Create Blob service client to interact with storage account
# Use SAS token as credential
blob_service_client = BlobServiceClient(account_url=account_url, credential=sas_token)
# First try to create container
try:
container_client = blob_service_client.create_container(name=container_name)
# If container already exists, fetch the client
except ResourceExistsError:
container_client = blob_service_client.get_container_client(container=container_name)
# List blobs in container
for blob in container_client.list_blobs():
print(blob.name)
Note: The above is using version azure-storage-blob==12.5.0, which is the latest package. This is not too far ahead of your version, so I would probably update your code to work with latest functionality, as also provided in the documentation.
Update
If you are unable to use account keys for security reasons, then you can create a service principal and give it a Storage Blob Data Contributor role to your storage account. This gets created as an AAD application, which will have access to your storage account.
To get this setup, you can use this guide from the documentation.
Sample Code
from azure.identity import DefaultAzureCredential
from azure.storage.blob import BlobServiceClient
token_credential = DefaultAzureCredential()
blob_service_client = BlobServiceClient(
account_url="https://<my_account_name>.blob.core.windows.net",
credential=token_credential
)

It looks like the module is deprecated:
Starting with v5.0.0, the 'azure' meta-package is deprecated and cannot be
installed anymore. Please install the service specific packages prefixed by
azure needed for your application.
The complete list of available packages can be found at:
https://aka.ms/azsdk/python/all
A more comprehensive discussion of the rationale for this decision can be found
in the following issue:
https://github.com/Azure/azure-sdk-for-python/issues/10646

Google Authentication in python

I am trying to get authentication to use the google translation API. Currently on my local machine I simply do this:
from google.cloud import translate_v2 as translate
os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = path_to_cred_json_file
translate_client = translate.Client()
which works fine. However, I wish to do this on AWS where I have I stored the credential json file in AWS secrets. In the documentation for translate.Client I see this:
Init signature:
translate.Client(
target_language='en',
credentials=None,
...
)
...
:type credentials: :class:`~google.auth.credentials.Credentials`
However, if I read in the json file and try to pass it in as the credentials argument it chucks an error.
The only answer I have for now in AWS is to read the secret, write it out as a json file, and then set os.environ["GOOGLE_APPLICATION_CREDENTIALS"] = path_to_cred_json_file, which will work, but was told by a data engineer is a security risk.
So the question is how do I get this google.auth.credentials.Credentials object without reading a physical file. I have access to the plain text version of the json file in memory (via AWS secrets). I'm really new to AWS in general so go easy on me.

Thanks to #miles-budnek and this github comment, found the answer.
Supposing I have the json string as a dictionary called secret:
from google.cloud import translate_v2 as translate
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_info(secret)
t_client = translate.Client(credentials=credentials)

sas generated blob url sends bad request 400 to azure ocr api

I have uploaded some images to azure blob container for my azure ocr api to read the image and send back the output.
I have retrieved list of blobs from azure container using
blob_service.list_blobs().
Each of these retrieved blobs are now given to shared access method
generate_blob_shared_access_signature(container_name='ocr-images',blob_name=blob.name,permission=PublicAccess.OFF,expiry='se=2015-04-30T02%3A23%3A26Z',start='st=2015-04-29T22%3A18%3A26Z')
output of shared access method is a SAS token which is then given to
blob_service.make_blob_url(container_name='ocr-images',blob_name=blob.name, sas_token=sas)
for generating url for each image
The URL generated when passed to Azure ocr api, shows error
requests.exceptions.HTTPError: 400 Client Error: Bad Request for url:https://westeurope.api.cognitive.microsoft.com/vision/v2.0/ocr?language=unk&detectOrientation=false
But when url is generated manually from azure portal, it works perfectly. Can some one help me with this issue?

You're getting this error because you're using the method generate_blob_shared_access_signature incorrectly.
There are a number of issues with the values you're passing to this method.
For permission, you need to provide one of the possible values of BlobPermissions. Assuming you would want to read the blob contents, I would recommend that you use BlobPermissions.READ permission.
Both of your start and expiry dates are in the past. Furthermore, you just need to specify the date value and not include st= and se=.
Please try with the following code:
generate_blob_shared_access_signature(container_name='ocr-images',blob_name=blob.name,permission=BlobPermissions.READ,expiry='2019-04-09',start='2019-04-08')

In case of 404 error while creating url from SAS token blob it is required to create a SAS token from BlobSharedAccesssignature.
Here is my code:
from azure.storage.blob import BlockBlobService
from azure.storage.blob.models import BlobPermissions
from azure.storage.blob.sharedaccesssignature import BlobSharedAccessSignature
account_name = data_dict['blob_storage_account_name']
account_key = data_dict['blob_storage_account_key']
top_level_container_name = data_dict['blob_container_name']
blob_service = BlockBlobService(account_name, account_key)
blob_shared = BlobSharedAccessSignature(account_name, account_key)
Once you have object from BlobSharedAccessSignature, call generate_blob method which create sas token for individual blob from your container
sas = blob_shared.generate_blob(container_name=top_level_container_name, blob_name=blob.name,
permission=BlobPermissions.READ, expiry='2019-04-10',
start='2019-04-09')
sas_url= 'https://'+account_name+'.blob.core.windows.net'+'/'+top_level_container_name+'/'+blob.name+'?'+sas

C++ Azure file upload: Server failed to authenticate the request

I'm trying to upload a file to my azure file service using the following C++ code (which is referenced from official documentation here):
const utility::string_t storage_connection_string(U("DefaultEndpointsProtocol=https;AccountName=myaccount;"
"AccountKey=mykey"));
// Retrieve storage account from connection string.
azure::storage::cloud_storage_account storage_account =
azure::storage::cloud_storage_account::parse(storage_connection_string);
// Create the Azure Files client.
azure::storage::cloud_file_client file_client =
storage_account.create_cloud_file_client();
// Get a reference to the share.
azure::storage::cloud_file_share share =
file_client.get_share_reference(_XPLATSTR("myfileservice"));
//Get a reference to the root directory for the share.
azure::storage::cloud_file_directory root_dir = share.get_root_directory_reference();
// Upload a file from a file.
azure::storage::cloud_file file =
root_dir.get_file_reference(_XPLATSTR("my-sample-file"));
file.upload_from_file(_XPLATSTR("test.pdf"));
This gives out the following error:
terminate called after throwing an instance of 'azure::storage::storage_exception'
what(): Server failed to authenticate the request. Make sure the value of Authorization header is formed correctly including the signature.
I've searched the internet and found it should be something related to authentication using SAS. I tried many example codes online but still cannot succeed.
However, I can do similar file upload by the following Python code without SAS authentication and without any trouble:
import os
from azure.storage.file import FileService, ContentSettings
currentDir = os.path.dirname(os.path.abspath(__file__))
#credentials for azure account
file_service = FileService(account_name='myaccount', account_key='mykey')
#path for azure file
global pathStr
pathStr = 'https://myaccount.file.core.windows.net/myfileservice/'
#function for uploading file onto Azure container
def upload(myfile):
file_service.create_file_from_path(
'myfileservice',
'files',
test + '.pdf',
myfile,
content_settings=ContentSettings(content_type='text/pdf')
)
print('finished uploading file.')
What could be the solution for this?