Blob.generate_signed_url() failing to AttributeError - python

So I'm trying to produce temporary globally readable URLs for my Google Cloud Storage objects using the google-cloud-storage Python library (https://googlecloudplatform.github.io/google-cloud-python/latest/storage/blobs.html) - more specifically the Blob.generate_signed_url() method. I doing this from within a Compute Engine instance in a command line Python script. And I keep getting the following error:
AttributeError: you need a private key to sign credentials.the credentials you are currently using <class 'oauth2cl
ient.service_account.ServiceAccountCredentials'> just contains a token. see https://google-cloud-python.readthedocs
.io/en/latest/core/auth.html?highlight=authentication#setting-up-a-service-account for more details.
I am aware that there are issues with doing this from within GCE (https://github.com/GoogleCloudPlatform/google-auth-library-python/issues/50) but I have created a new Service Account credentials following the instructions here: https://cloud.google.com/storage/docs/access-control/create-signed-urls-program and my key.json file most certainly includes a private key. Still I am seeing that error.
This is my code:
keyfile = "/path/to/my/key.json"
credentials = ServiceAccountCredentials.from_json_keyfile_name(keyfile)
expiration = timedelta(3) # valid for 3 days
url = blob.generate_signed_url(expiration, method="GET",
credentials=credentials)
I've read through the issue tracker here https://github.com/GoogleCloudPlatform/google-cloud-python/issues?page=2&q=is%3Aissue+is%3Aopen and nothing related jumps out so I am assuming this should work. Cannot see what's going wrong here.

I was having the same issue. Ended up fixing it by starting the storage client directly from the service account json.
storage_client = storage.Client.from_service_account_json('path_to_service_account_key.json')
I know I'm late to the party but hopefully this helps!

Currently, it's not possible to use blob.generate_signed_url without explicitly referencing credentials. (Source: Google-Cloud-Python documentation) However, you can do a workaround, as seen here, which consists of:
signing_credentials = compute_engine.IDTokenCredentials(
auth_request,
"",
service_account_email=credentials.service_account_email
)
signed_url = signed_blob_path.generate_signed_url(
expires_at_ms,
credentials=signing_credentials,
version="v4"
)

A much complete snippet for those asking where other elements come from. cc #AlbertoVitoriano
from google.auth.transport import requests
from google.auth import default, compute_engine
credentials, _ = default()
# then within your abstraction
auth_request = requests.Request()
credentials.refresh(auth_request)
signing_credentials = compute_engine.IDTokenCredentials(
auth_request,
"",
service_account_email=credentials.service_account_email
)
signed_url = signed_blob_path.generate_signed_url(
expires_at_ms,
credentials=signing_credentials,
version="v4"
)

Related

TypeError: Consumer key must be string or bytes, not NoneType when authentication

Planning to use variables to authorize tweepy with Twitter's API, but it shows the following:
TypeError: Consumer key must be string or bytes, not NoneType when authentication
Code below:
import tweepy
import os
consumera = os.environ.get('TWICK')
consumerb = os.environ.get('TWICS')
accessa = os.environ.get('TWIAT')
accessb = os.environ.get('TWIATS')
auth = tweepy.OAuthHandler(consumera, consumerb)
auth.set_access_token(accessa, accessb)
api = tweepy.API(auth)
status = api.update_status(status="Test tweet.")
I've tried using other authentication methods, such as:
consumera = os.environ.get('TWICK')
consumerb = os.environ.get('TWICS')
accessa = os.environ.get('TWIAT')
accessb = os.environ.get('TWIATS')
client = tweepy.Client(consumer_key=consumera,
consumer_secret=consumerb,
access_token=accessa,
access_token_secret=accessb
)
Any fix?
Additional info: I was planning to run it through GitHub actions but without exposing the keys in the source code - and I put all the keys as secrets, so it requires environmental access. Tested on Ubuntu (both local and under GitHub Actions), but no luck.
Perhaps a typo problem? NoneType is caused because one of the methods is receiving None, which is returned by get() if the environment variable does not exist.
Check if your environment variables' names are correct.
If your problem persists please indicate in which line the error occurs.

Invalid scope for ComputeManagementClient for Azure US Government account?

I'm trying to create a simple script that lists out the virtual machines on my Azure US Government account. However, I am faced with this error:
azure.core.exceptions.ClientAuthenticationError: DefaultAzureCredential failed to retrieve a token from the included credentials.
Attempted credentials:
VisualStudioCodeCredential: Azure Active Directory error '(invalid_scope) AADSTS70011: The provided request must include a 'scope' input parameter. The provided value for the input parameter 'scope' is not valid. The scope https://management.azure.com/.default https://management.core.usgovcloudapi.net/.default is not valid. static scope limit exceeded.
This is the code I have used:
def get_access_to_virtual_machine():
subscription_id = key.SUBSCRIPTION_ID
credentials = DefaultAzureCredential(authority = AZURE_US_GOV_CLOUD.endpoints.active_directory,
tenant_id = key.TENANT_ID,
exclude_environment_credential = True,
exclude_managed_identity_credential = True,
exclude_shared_token_cache_credential = True)
compute_client = ComputeManagementClient(credential = credentials,
subscription_id = subscription_id,
base_url = AZURE_US_GOV_CLOUD.endpoints.resource_manager,
credential_scopes = [AZURE_US_GOV_CLOUD.endpoints.active_directory_resource_id + '.default'])
return compute_client
def get_azure_vm(resource_group_name, virtual_machine_name):
compute_client = get_access_to_virtual_machine()
vm_data = compute_client.virtual_machines.get(resource_group_name,
virtual_machine_name,
expand = 'instanceView')
return vm_data
I have signed into my Azure US Government account using Visual Studio as well. The error stems from the compute_client.virtual_machines.get() command. I am 100% sure the credentials I am using are correct but I am really stuck on this. I've tried using ClientSecretCredential instead of DefaultAzureCredential and ran into the same ClientAuthenticationError. In addition, I'm not sure where this scope parameter that the error mentions should be passed in.
For Azure Subscriptions management, the scope should be {management-endpoint}/user_impersonation and not {management-endpoint}/.default. For example, in Azure Commercial the scope will be https://management.azure.com/user_impersonation.
I'm not 100% sure but the management endpoint for Azure Government is either https://management.usgovcloudapi.net/ or https://management.core.usgovcloudapi.net/. Based on the correct endpoint, your scope value should be either https://management.usgovcloudapi.net/user_impersonation or https://management.core.usgovcloudapi.net/user_impersonation.
Please try by changing that.
UPDATE
Looking at the GitHub issue here, it seems there's an issue with the SDK itself. Please try the solution proposed here.
Not sure which version of the Python SDK you have, but I was able to load the latest modules and get the following code to run in the Azure US Government cloud and pull back VM data:
import os
from msrestazure.azure_cloud import AZURE_US_GOV_CLOUD as CLOUD
from azure.mgmt.compute import ComputeManagementClient
from azure.identity import DefaultAzureCredential
subscription_id = 'xxx-xxx-xxx-xxxx'
tenant_id = 'xxxx-xxxx-xxxx-xxxx'
resource_group_name = 'rgName'
vm_name = 'vmName'
credential = DefaultAzureCredential(
authority=CLOUD.endpoints.active_directory,
tenant_id=tenant_id)
compute_client = ComputeManagementClient(
credential, subscription_id,
base_url=CLOUD.endpoints.resource_manager,
credential_scopes=[CLOUD.endpoints.resource_manager + '/.default'])
vm_data = compute_client.virtual_machines.get(
resource_group_name,
vm_name,
expand = 'instanceView')
print(f"{vm_data.name}")
Some things to note:
You had a few of the authentication methods set as excluded, you may want to ensure the method you are expecting is not excluded
The latest SDK sets the environment in the import, I set it to "CLOUD" so that the same code can be used for various cloud by simply changing the import statement
The latest SDK does seem to want '/.default' as part of the credential_scopes

InvalidRequestError: Must provide an 'engine' parameter while invoking openAI API for text generation

I was trying this code given in OpenAI.
Link:- API for text generation
Code
import openai
prompt = """We’re releasing an API for accessing new AI models developed by OpenAI. Unlike most AI systems which are designed for one use-case, the API today provides a general-purpose “text in, text out” interface, allowing users to try it on virtually any English language task. You can now request access in order to integrate the API into your product, develop an entirely new application, or help us explore the strengths and limits of this technology."""
response = openai.Completion.create(model="davinci", prompt=prompt, stop="\n", temperature=0.9, max_tokens=100)
print(response)
I'm getting an error
Error
"Must provide an 'engine' parameter to create a %s" % cls, "engine".
openai.error.InvalidRequestError: Must provide an 'engine' parameter to create a <class 'openai.api_resources.completion.Completion'>
I'm using python 3.7.6
It seems you have confused the engine parameter with the model parameter. Please have a look at this documentation for the correct way to call: https://beta.openai.com/docs/developer-quickstart/python-bindings
Please change model = "davinci" to engine = "davinci"
and you should be good to go.
If you get the error code:
...
InvalidRequestError: Engine not found
One possible problem could be your account setting does not offer you access to the engine. For example, embedding engines are only available for "private beta." You may need to request access to it for your account.
The following code may get you the available engines to your account:
import openai
openai.api_key = your_openai_api_key
data = openai.Engine.list() for eng in data['data']:
print(eng['id'])
Here is complete prompt in a function, for a successful query:
import os
import openai
openai.api_key = os.environ["openai_key"]
start = "Your are a AI Search Engine, answer the following query with a witty answer and include validated facts only."
def generate(prompt):
start_sequence = "{}.{}".format(start,prompt)
completions = openai.Completion.create(
model="text-davinci-003",
prompt=start_sequence,
temperature=0.1,
max_tokens=256,
top_p=1,
frequency_penalty=0.51,
presence_penalty=0.5,
#stream = False,
#echo = True
)
message = completions.choices[0].text
print(message)
return message

python3 upload files to ondrive or sharepoint?

Anyone know if this is possible?
I just want to automate dropping some documents into my onedrive for business account.
I tried
import onedrivesdk
from onedrivesdk.helpers import GetAuthCodeServer
from onedrivesdk.helpers.resource_discovery import ResourceDiscoveryRequest
redirect_uri = 'http://localhost:8080'
client_id = 'appid'
client_secret = 'mysecret'
discovery_uri = 'https://api.office.com/discovery/'
auth_server_url='https://login.live.com/oauth20_authorize.srf?scope=wl.skydrive_update'
#auth_server_url='https://login.microsoftonline.com/common/oauth2/authorize',
auth_token_url='https://login.microsoftonline.com/common/oauth2/token'
http = onedrivesdk.HttpProvider()
auth = onedrivesdk.AuthProvider(http,
client_id,
auth_server_url=auth_server_url,
auth_token_url=auth_token_url)
auth_url = auth.get_auth_url(redirect_uri)
code = GetAuthCodeServer.get_auth_code(auth_url, redirect_uri)
auth.authenticate(code, redirect_uri, client_secret, resource=resource)
# If you have access to more than one service, you'll need to decide
# which ServiceInfo to use instead of just using the first one, as below.
service_info = ResourceDiscoveryRequest().get_service_info(auth.access_token)[0]
auth.redeem_refresh_token(service_info.service_resource_id)
client = onedrivesdk.OneDriveClient(service_info.service_resource_id + '/_api/v2.0/', auth, http)
I registered an APP and got a secret and id. But when I ran this I got scope is invalid errors. Plus it tries to launch a webpage which isn't great for a command line kinda environment. I think this SDK might be outdated as well because originally this script had login.microsoftonline, but that wasn't reachable so I changed it to login.live.com.
I wrote this sample code you posted. You replaced the auth_server_URLwith the authentication URL for Microsoft Account authentication, which can only be used to access OneDrive (the consumer product). You need to continue using the login.microsoftonline.com URL to log into your OneDrive for Business account.
You are correct that this pops up a dialog. However, you can write a little supporting code so that only happens the first time you log into a particular app. Follow these steps (assuming you are using the default implementation of AuthProvider:
Use the sample code above up through the line auth.redeem_refresh_token()
The AuthProvider will now have a Session object, which caches the credentials of the current user and session. Use AuthProvider.save_session() to save the credentials for later.
Next time you start your app, use AuthProvider.load_session() and AuthProvider.refresh_token() to retrieve the previous session and refresh the auth token. This will all be headless.
Take note that the default implementation of SessionBase (found here) uses Pickle and is not safe for product use. Make sure to create a new implementation of Session if you intend to deploy this app to other users.
Onerive's website shows "Not Yet" on "OneDrive SDK for Python" to "OneDrive for Business"
https://dev.onedrive.com/SDKs.htm
The github sample codes did not work for me either, it tried to popup a window of authentication, but IE can not find the address:
http://('https//login.microsoftonline.com/common/oauth2/authorize',)?redirect_uri=http%3A%2F%2Flocalhost%3A8080&client_id=034xxxx9-9xx8-4xxf-bexx-1bc5xxxxbd0c&response_type=code
or removed all the "-" in client id
http://('https//login.microsoftonline.com/common/oauth2/authorize',)?redirect_uri=http%3A%2F%2Flocalhost%3A8080&client_id=034xxxx99xx84xxfbexx1bc5xxxxbd0c&response_type=code
Either way, I got the same result, IE did not show the popup with a line "This page can’t be displayed"

Python: Facebook API showing invalid token

I am using facepy facebook graph API to access my mailbox/messages and I followed the following two approaches:
1st Approach:
I used the access token I got from Graph Explorer facebook page and use the below code:
from facepy import GraphAPI
graph = GraphAPI(token)
print graph.get('/me')
#Rest of the code
The above code worked fine and I was able to retrieve all my messages using an FQL Query. The problem arised when my auth_token expired after sometime.
So, after some googling I shifted to approach two:
Now, what I did was created a facebook app gave it read_mailbox permission and got it's id and key and then used get_application_access_token method of facepy to get the token.
After retrieving the token I used:
token = facepy.utils.get_application_access_token(app_id, key)
graph.get('/me')
## OUT: OAuthError: [2500] An active access token must be used to query information about the current user.
facepy.utils.get_extended_access_token(token, app_id, key)
# OUT: OAuthError: [1] No user access token specified
Now, you can see the error(commented #) generated on using the application token.
I believe the error I am getting is because facebook needs the user_token and I am supplying it with app_token.
So, is it possible to access user data using the app_token and if not how can one issue a extended token which can access user data.
Update:
So, I followed #Johannes suggestion and tried this but ran into error:
from facepy.utils import get_extended_access_token
from facepy import GraphAPI
token = "My user access token got from https://developers.facebook.com/tools/explorer"
long_lived_access_token = get_extended_access_token(token)
graph = GraphAPI(long_lived_access_token)
graph.get('/me')
Now, when I ran the above code I got
TypeError: get_extended_access_token() takes exactly 3 arguments (1 given)
So, I changed it to long_lived_access_token = get_extended_access_token(token, None, None) and got
facepy.exceptions.OAuthError
So, I again I changed it to long_lived_access_token = get_extended_access_token(token, app_id, key) and I got the same exception/error as above.
So, is this a bug or am I doing something wrong.
PS: I installed the latest version from git.
You're right in your assumption that you cannot use application access tokens to read a user's mailbox, but the error you're getting stems from the fact that you haven't initialized graph with an access token at all.
Be that as it may, you're on the right track in asking for how you can extend the user's access token. As you have already discovered, Facepy HEAD (soon to be version 0.9) has a function get_extended_access_token which accepts an existing short-lived user access token and extends it. Extended user access tokens last for 2 months, and you can read more about them in Facebook's documentation on the removal of offline_access permission.
If you want to use get_extended_access_token right now, you will have to install facepy from git:
$ pip install git+git://github.com/jgorset/facepy.git#b5153f460f2f52cef9a5e49a3b48b3fb8742356c
Once you've installed the right version of Facepy, you can extend an existing short-lived user access token and initialize a new instance of GraphAPI with it:
from facepy.utils import get_extended_access_token
from facepy import GraphAPI
long_lived_access_token, expires_at = get_extended_access_token(short_lived_access_token, application_id, application_secret_key)
graph = GraphAPI(long_lived_access_token)
graph.get('/me')
There is nothing wrong with the API you are just not interpreting the result in a right way.
If you try to print the result of long_lived_access_token = get_extended_access_token(token) it will not directly give you a long_lived_access_token instead it will provide you a tuple with contents:
long_lived_access_token = ('your token', datetime_object).
You can verify this by looking at the source code of utils.py. If you look at get_extended_access_token method it returns token, expires_at.
According to the facebook docs to get the extended access token one has to make request at the below endpoint
https://graph.facebook.com/oauth/access_token?
client_id=APP_ID&
client_secret=APP_SECRET&
grant_type=fb_exchange_token&
fb_exchange_token=EXISTING_ACCESS_TOKEN
and the response is something like token=mytoken&expire=5184000 where 5184000 means 60 days.
So, your final code will look something like:
from facepy.utils import get_extended_access_token
from facepy import GraphAPI
app_id = 'id'
key = 'key'
short_lived_access_token = 'short_token'
long_token = get_extended_access_token(short_token, id, key)
graph = GraphAPI(long_token[0])
print graph.get('/me')

Categories