We Keep Coding

Databricks API call fails on Azure DevOps pipelines using python script, but run successfully on Postman from local machine

In Azure databricks API I am trying to pull latest changes to main branch in each folder in Databricks repos by using Azure Databricks API. This is where I am refering to:
When I use postman to make the calls by posting requests to the following endpoint, it pulls successfully as shown below:
endpoint:
https://<databricks-workspace>.azuredatabricks.net/api/2.0/repos/<repo-id>
This is the header of the same request:
To explain more, the header is constructed by sending a bearer token, a management token and another field which contains subscription, resource group and databricks workspace as shown below:
/subscriptions/<Azure-subscription-id>/resourceGroups/<resourcegroup-name>/providers/Microsoft.Databricks/workspaces/<databricks-workspace-name>
As shown above it works perfectly well when I call it on my local machine with postman. But when I use the same thing by using Azure DevOps it fails with the error:
Exception: b'{"error_code":"PERMISSION_DENIED","message":"Missing Git provider credentials. Go to User Settings > Git Integration to add your personal access token."}'
Note that following this link I have already generated a PAT token in Azure DevOps and added it to my Service Principal, otherwise it wouldn't have worked on my postman. Still it is giving this error in DevOps pipeline as shown below:
This pipeline is doing exactly what I already did with postman. This pipeline is calling a python script which is constructing the request header and body as shown above in postman. The python code is as below but I am almost sure it is not the python script that is causing the issue as I have used the same method to list repos, get specific repo, create clusters and many more by the same methodology. I think it must be some administrative problem which I cannot pin point.
The python script:
import requests
import os
import json
## Constructing the header request
DBRKS_REQ_HEADERS = {
'Authorization': 'Bearer ' + os.environ['DBRKS_BEARER_TOKEN'],
'X-Databricks-Azure-Workspace-Resource-Id': '/subscriptions/'+ os.environ['DBRKS_SUBSCRIPTION_ID'] +'/resourceGroups/'+ os.environ['DBRKS_RESOURCE_GROUP'] +'/providers/Microsoft.Databricks/workspaces/' + os.environ['DBRKS_WORKSPACE_NAME'],
'X-Databricks-Azure-SP-Management-Token': os.environ['DBRKS_MANAGEMENT_TOKEN']}
TRIGGERING_BRANCH = "\"" + os.environ["TRIGGERING_BRANCH"] + "\""
print("TRIGGERING_BRANCH path is {}".format(TRIGGERING_BRANCH))
## Constructing the body request
body_json = '''
{
"branch": "main"
}
'''
## Checking the request body format is correct
print("Request body in json format:")
print(body_json)
## The prints are only for code tracing
DBRKS_REPOS_LIST_JSON = os.environ["DBRKS_REPOS_LIST"]
print("Type of DBRKS_REPOS_LIST_JSON is {}".format(type(DBRKS_REPOS_LIST_JSON)))
## This section extracts repo Ids from the variable containing repo Ids and branches to later construct url endpoint
str_obj = DBRKS_REPOS_LIST_JSON.replace('[','').replace(']','').replace('(','').replace(')','').replace('\'','').replace(' ','').split(',')
output = {}
str_to_list = [str_obj[i:i+2] for i in range(0, len(str_obj), 2)]
print("str_to_list")
print(str_to_list)
for e in str_to_list:
output[e[0]] = e[1]
print("output")
print(output)
repo_ids_for_main_branch = []
for key, value in output.items():
if value == 'main':
repo_ids_for_main_branch.append(key)
print("repo_ids_for_main_branch")
print(repo_ids_for_main_branch)
## This is the main part which is making the API call like postman:
for repo_id in repo_ids_for_main_branch:
dbrks_pull_repo_url = "https://"+os.environ['DBRKS_INSTANCE']+".azuredatabricks.net/api/2.0/repos/"+str(repo_id)
print("endpoint url is {}".format(dbrks_pull_repo_url))
response = requests.patch(dbrks_pull_repo_url, headers=DBRKS_REQ_HEADERS, data=body_json)
if response.status_code == 200:
print("Branch changed or pulled successfully!")
print(response.status_code)
print(response.content)
print('####################')
else:
print("Couldn't pull or change branch!")
raise Exception(response.content)
All the os variables in the code as passed from Azure DevOps pipeline to the script and I have checked their values by printing and they are all correct.
I would like to know what the root cause problem is and how I can resolve it.

A small operation is there to implement. The major issue was with GIT integration. This can be resolved with the following steps.
Enable support for arbitrary files in Databricks Repos
This operation is needed to implement. Follow the link for the series of steps.

Can't access Microsoft To Do list using the MS Graph API

I try to make programm in python to add appointments to Microsoft To Do
I'm using the Microsoft Graph API with following code:
import requests
import msal
import atexit
import os.path
TENANT_ID = 'X
CLIENT_ID = 'Y'
AUTHORITY = 'https://login.microsoftonline.com/' + TENANT_ID
ENDPOINT = 'https://graph.microsoft.com/v1.0'
SCOPES = [
'Files.ReadWrite.All',
'Sites.ReadWrite.All',
'User.Read',
'User.ReadBasic.All',
'Tasks.ReadWrite'
]
cache = msal.SerializableTokenCache()
if os.path.exists('token_cache.bin'):
cache.deserialize(open('token_cache.bin', 'r').read())
atexit.register(lambda: open('token_cache.bin', 'w').write(cache.serialize()) if cache.has_state_changed else None)
app = msal.PublicClientApplication(CLIENT_ID, authority=AUTHORITY, token_cache=cache)
accounts = app.get_accounts()
result = None
if len(accounts) > 0:
result = app.acquire_token_silent(SCOPES, account=accounts[0])
if result is None:
flow = app.initiate_device_flow(scopes=SCOPES)
if 'user_code' not in flow:
raise Exception('Failed to create device flow')
print(flow['message'])
result = app.acquire_token_by_device_flow(flow)
if 'access_token' in result:
print(result['access_token'])
result = requests.get(f'https://graph.microsoft.com/v1.0/me/todo/lists', headers={'Authorization': 'Bearer ' + result['access_token']})
print(result.json())
else:
raise Exception('no access token in result')
This is the error I get:
{'error': {'code': 'UnknownError', 'message': 'The service is unavailable.', 'innerError': {'date': '2022-01-25T18:52:03', 'request-id': 'X', 'client-request-id': 'X'}}}
I tried to google the error but I didn't found any solution that worked for me.

Just for Curiosity ,I would like to know have you entered your TENANT_Id and CLIENT_ID in place of "X" and "Y". if yes and still its not working.
Could you please try to run the URL - https://graph.microsoft.com/v1.0/me/todo/lists on Graph explorer and postman. let us know if you are still facing the issue.

could you please check your token expiry time (exp), might be it got expired. Go to jwt.ms to check token details, find the attached image to see where to find token time.
If the token got expired ,please the docs get new token.

I'm trying to get the direct download link for a file using the Google Drive API

I'm trying to get the direct download link for a file in Google Drive using the Google Drive API (v3), but I'm also trying to do this without making the file publicly shared.
Here is what I've tried:
https://www.googleapis.com/drive/v3/files/**FILE_ID**?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key=**API_KEY**
Now this works if the file is shared publicly. But when the file isn't shared publicly you get this message:
{'error': {'errors': [{'domain': 'global', 'reason': 'notFound', 'message': 'File not found: 10k0Qogwcz7k0u86m7W2HK-LO7bk8xAF8.', 'locationType': 'parameter', 'location': 'fileId'}], 'code': 404, 'message': 'File not found: 10kfdsfjDHJ38-UHJ34D82.'}}
After doing some googling I found a post on stack overflow saying that I need to add a request header with my access token, but this doesn't work and the application just hangs
Here is the full code:
### SETTING UP GOOGLE API
scopes = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', scopes)
credentials = tools.run_flow(flow, store)
accessToken = credentials.access_token
refreshToken = credentials.refresh_token
drive = build('drive', 'v3', credentials=credentials)
### SENDING REQUEST
req_url = "https://www.googleapis.com/drive/v3/files/"+file_id+"?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key="+GOOGLE_API_KEY
headers={'Authorization': 'Bearer %s' % accessToken}
request_content = json.loads((requests.get(req_url)).content)
print(request_content)
------------------ EDIT: ------------------
I've gotten really close to an answer, but I can't seem to figure out why this doesn't work.
So I've figured out previously that alt=media generates a download link for the file, but when the file is private this doesn't work.
I just discovered that you can add &access_token=.... to access private files, so I came up with this API call:
https://www.googleapis.com/drive/v3/files/**FILE_ID**?supportsAllDrives=true&alt=media&access_token=**ACCESS_TOKEN**&key=**API_KEY**
When I go to that url on my browser I get this message:
We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
I find this confusing because if I remove alt=media, I am able to call on that request and I get some metadata about the file.

I believe your goal as follows.
From I'm trying to get the direct download link for a file in Google Drive using the Google Drive API (v3),, I understand that you want to retrieve webContentLink.
The file that you want to retrieve the webContentLink is the files except for Google Docs files.
You have already been able to get the file metadata using Drive API. So your access token can be used for this.
Modification points:
When the file is not shared, the API key cannot be used. By this, https://www.googleapis.com/drive/v3/files/**FILE_ID**?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key=**API_KEY** returns File not found. I think that the reason of this issue is due to this.
When I saw your script in your question, it seems that you want to download the file content.
In your script, headers is not used. So in this case, the access token is not used.
In the method of "Files: get", there is no includeItemsFromAllDrives.
In your script, I think that an error occurs at credentials.access_token. How about this? If my understanding is correct, please try to modify to accessToken = credentials.token.
In Drive API v3, the default response values don't include webContentLink. So in this case, the field value is required to be set like fields=webContentLink.
When your script is modified, it becomes as follows.
Modified script:
file_id = '###' # Please set the file ID.
req_url = "https://www.googleapis.com/drive/v3/files/" + file_id + "?supportsAllDrives=true&fields=webContentLink"
headers = {'Authorization': 'Bearer %s' % accessToken}
res = requests.get(req_url, headers=headers)
obj = res.json()
print(obj.get('webContentLink'))
Or, you can use drive = build('drive', 'v3', credentials=credentials) in your script, you can also use the following script.
file_id = '###' # Please set the file ID.
drive = build('drive', 'v3', credentials=credentials)
request = drive.files().get(fileId=file_id, supportsAllDrives=True, fields='webContentLink').execute()
print(request.get('webContentLink'))
Note:
In this modified script,
When the file is in the shared Drive and you don't have the permissions for retrieving the file metadata, an error occurs.
When your access token cannot be used for retrieving the file metadata, an error occurs.
So please be careful above points.
When * is used for fields, all file metadata can be retrieved.
Reference:
Files: get
Added:
You want to download the binary data from the Google Drive by the URL.
The file size is large like "2-10 gigabytes".
In this case, unfortunately, webContentLink cannot be used. Because in the case of the such large file, webContentLink is redirected. So I think that the method that the file is publicly shared and use the API key is suitable for achieving your goal. But, you cannot publicly shared the file.
From this situation, as a workaround, I would like to propose to use this method. This method is "One Time Download for Google Drive". At Google Drive, when the publicly shared file is downloaded, even when the permission of file is deleted under the download, the download can be run. This method uses this.
Flow
In this sample script, the API key is used.
Request to Web Apps with the API key and the file ID you want to download.
At Web Apps, the following functions are run.
Permissions of file of the received file ID are changed. And the file is started to be publicly shared.
Install a time-driven trigger. In this case, the trigger is run after 1 minute.
When the function is run by the time-driven trigger, the permissions of file are changed. And sharing file is stopped. By this, the shared file of only one minute can be achieved.
Web Apps returns the endpoint for downloading the file of the file ID.
After you got the endpoint, please download the file using the endpoint in 1 minute. Because the file is shared for only one minute.
Usage:
1. Create a standalone script
In this workaround, Google Apps Script is used as the server side. Please create a standalone script.
If you want to directly create it, please access to https://script.new/. In this case, if you are not logged in Google, the log in screen is opened. So please log in to Google. By this, the script editor of Google Apps Script is opened.
2. Set sample script of Server side
Please copy and paste the following script to the script editor. At that time, please set your API key to the variable of key in the function doGet(e).
Here, please set your API key in the function of doGet(e). In this Web Apps, when the inputted API key is the same, the script is run.
function deletePermission() {
const forTrigger = "deletePermission";
const id = CacheService.getScriptCache().get("id");
const triggers = ScriptApp.getProjectTriggers();
triggers.forEach(function(e) {
if (e.getHandlerFunction() == forTrigger) ScriptApp.deleteTrigger(e);
});
const file = DriveApp.getFileById(id);
file.setSharing(DriveApp.Access.PRIVATE, DriveApp.Permission.NONE);
}
function checkTrigger(forTrigger) {
const triggers = ScriptApp.getProjectTriggers();
for (var i = 0; i < triggers.length; i++) {
if (triggers[i].getHandlerFunction() == forTrigger) {
return false;
}
}
return true;
}
function doGet(e) {
const key = "###"; // <--- API key. This is also used for checking the user.
const forTrigger = "deletePermission";
var res = "";
if (checkTrigger(forTrigger)) {
if ("id" in e.parameter && e.parameter.key == key) {
const id = e.parameter.id;
CacheService.getScriptCache().put("id", id, 180);
const file = DriveApp.getFileById(id);
file.setSharing(DriveApp.Access.ANYONE_WITH_LINK, DriveApp.Permission.VIEW);
var d = new Date();
d.setMinutes(d.getMinutes() + 1);
ScriptApp.newTrigger(forTrigger).timeBased().at(d).create();
res = "https://www.googleapis.com/drive/v3/files/" + id + "?alt=media&key=" + e.parameter.key;
} else {
res = "unavailable";
}
} else {
res = "unavailable";
}
return ContentService.createTextOutput(res);
}
3. Deploy Web Apps
On the script editor, Open a dialog box by "Publish" -> "Deploy as web app".
Select "Me" for "Execute the app as:".
Select "Anyone, even anonymous" for "Who has access to the app:". This is a test case.
If Only myself is used, only you can access to Web Apps. At that time, please use your access token.
Click "Deploy" button as new "Project version".
Automatically open a dialog box of "Authorization required".
Click "Review Permissions".
Select own account.
Click "Advanced" at "This app isn't verified".
Click "Go to ### project name ###(unsafe)"
Click "Allow" button.
Click "OK"
4. Test run: Client side
This is a sample script of python. Before you test this, please confirm the above script is deployed as Web Apps. And please set the URL of Web Apps, the file ID and your API key.
import requests
url1 = "https://script.google.com/macros/s/###/exec"
url1 += "?id=###fileId###&key=###your API key###"
res1 = requests.get(url1)
url2 = res1.text
res2 = requests.get(url2)
with open("###sampleFilename###", "wb") as f:
f.write(res2.content)
In this sample script, at first, it requests to the Web Apps using the file ID and API key, and the file is shared publicly in 1 minute. And then, the file can be downloaded. After 1 minute, the file is not publicly shared. But the download of the file can be kept.
Note:
When you modified the script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.
References:
One Time Download for Google Drive
Web Apps
Taking advantage of Web Apps with Google Apps Script

CloudKit Server-to-Server auth: Keep getting 401 Authentication failed

I have been recently exploring the CloudKit and related frameworks. I got the communication with my app working, as well as with my website using CloudKitJS. Where I am struggling is the Server-to-Server communication (which I would need for exporting data from public database in csv.
I have tried Python package requests-cloudkit, which others were suggesting. I have created a Server-to-Server token, and have copied only the key between START and END line once creating the eckey.pem file. I then got this code:
from requests_cloudkit import CloudKitAuth
from restmapper import restmapper
import json
KEY_ID = '[my key ID from CK Dashboard]'
SECRET_FILE_KEY = 'eckey.pem'
AUTH = CloudKitAuth(KEY_ID, SECRET_FILE_KEY)
PARAMS = {
'query':{
'recordType': '[my record type]'
},
}
CloudKit = restmapper.RestMapper("https://api.apple-cloudkit.com/database/1/[my container]/development/")
cloudkit = CloudKit(auth=AUTH)
response = cloudkit.POST.public.records.query(json.dumps(PARAMS))
I am then getting the 401 Authentication failed response. I am stuck on this for days, so I would be grateful for any help or advice. 😊

Creating the server-to-server key is an important first step, but in order to make HTTP requests after that, you have to sign each request.
Look for the Authenticate Web Service Requests section near the bottom of this documentation page.
It's a little bit convoluted, but you have to carefully construct signed headers to include with each request you make. I'm not familiar with how to do it in Python, but here's how I do it in NodeJS which may help:
//Get the timestamp in a very specific format
let date = moment().utc().format('YYYY-MM-DD[T]HH:mm:ss[Z]')
//Construct the subpath
let endpoint = '/records/lookup'
let path = '/database/1/iCloud.*****/development/public'
let subpath = path+endpoint
//Get the key file
let privateKeyFile = fs.readFileSync('../../'+SECRET_FILE_KEY, 'utf8')
//Make a string out of your JSON query
let query = {
recordType: '[my record type]'
}
let requestBody = JSON.stringify(query)
//Hash the query
let bodyHash = crypto.createHash('sha256').update(requestBody, 'utf8').digest('base64')
//Assemble the components you just generated in a special format
//[Current date]:[Request body]:[Web service URL subpath]
let message = date+':'+bodyHash+':'+subpath
//Sign it
let signature = crypto.createSign('RSA-SHA256').update(message).sign(privateKeyFile, 'base64')
//Assemble your headers and include them in your HTTP request
let headers = {
'X-Apple-CloudKit-Request-KeyID': KEY_ID,
'X-Apple-CloudKit-Request-ISO8601Date': date,
'X-Apple-CloudKit-Request-SignatureV1': signature
}
This is a bit hairy at first, but I just put all this stuff in a function that I reuse whenever I need to make a request.
Apple's documentation has pretty much been abandoned and it's hard to find good help with CloudKit Web Services these days.

Microsoft Sharepoint Authentication using Python

I am trying to upload documents to the SharePoint online.
Target url: https://companyURL.sharepoint.com/sites/A/B/Info_documents/C
My aim is to search folder C, and if X subfolder is present in folder C, I need to upload a file. To accomplish this, I have generated client_id and client_secret by navigating to http://{sharepointsite}/_layouts/15/AppRegNew.aspx. In the XML permissions, I gave the following code:
I am using https://github.com/vgrem/Office365-REST-Python-Client for this implementation. When trying to use the following code snippet to see, if I am having the access to sharepoint using client_id and client_secret, i am seeing different errors:
import json
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.runtime.client_request import ClientRequest
from office365.runtime.utilities.request_options import RequestOptions
from office365.sharepoint.client_context import ClientContext
app_settings = {
'url': 'https://companyURL.sharepoint.com/sites/A/B/Info_documents/C',
'client_id': 'xxxxxxxx',
'client_secret': 'xxxxxx',
}
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()
print("Web site title: {0}".format(web.properties['Title']))
Error:
ClientRequestException: ('-2147024891, System.UnauthorizedAccessException', 'Access denied. You do not have permission to perform this action or access this resource.', '403 Client Error: Forbidden for url: https://companyURL.sharepoint.com/sites/A/B/Info_documents/C_api/Web')
But I gave the permissions, not sure whether I am doing wrong or selected the wrong module.
Please help.
Following is the XML code i gave when generating the client_ID and client_secret:
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read"/>
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web/list" Right="Write"/>
</AppPermissionRequests>

Given the description
My aim is to search folder C, and if X subfolder is present in folder
C, I need to upload a file.
and the the following link format:
https://companyURL.sharepoint.com/sites/A/B/Info_documents/C
your folder structure could be represented like this:
https://companyURL.sharepoint.com/sites/A/B/ <-this is the actual site url
|
Info_documents <-library
|
C <-folder
Since AuthenticationContext class accepts the first parameter to be site url, app_settings needs to be updated like this:
app_settings = {
'url': 'https://companyURL.sharepoint.com/sites/A/B/', //need to refer to site url
'client_id': '--client id goes here--',
'client_secret': '--client secret goes here--',
}
Now comes the turn of App principal, since the requested permissions are applicable per web (scope: http://sharepoint/content/sitecollection/web), the second step (granting permissions) needs to be accomplished per specified web:
https://companyURL.sharepoint.com/sites/A/B/_layouts/15/appinv.aspx
Example
Here is an example which demonstrates how to verify if sub folder exists under the parent folder:
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'],
client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
folder_url = "Info_documents/C" #folder url where to find
folder_name = "X" #folder name to find
result = ctx.web.get_folder_by_server_relative_url(folder_url).folders.filter("Name eq '{0}'".format(folder_name))
ctx.load(result)
ctx.execute_query()
if len(result) > 0:
print("Folder has been found: {0}".format(result[0].properties["Name"]))