I wont upload 2 file to google drive with python and google script (not API)
I would also like to send the name of the file
I got this answer to my first question
You can do this by publishing a web app to run as "Me"(you) with access: "Anyone, even anonymous".
Server side:
function doPost(e){
const FOLDER_ID = '###FOLDER_ID###';
var name = e.parameter.name;
saveFile(e.postData.contents,name,FOLDER_ID);
return 'Success';
}
function saveFile(data,name,id) {
data = Utilities.newBlob(Utilities.base64DecodeWebSafe(data));
DriveApp.getFolderById(id)
.createFile(data.setName(name))
}
Client side:
import requests, base64
url = '###WEB_APP_PUBLISHED_URL###'
name = '###FILENAME###'
requests.post(url+'?name='+name,data=base64.urlsafe_b64encode(open('##UPLOAD FILE PATH##','rb').read()))
This answer helps me send one file without its name
I saw you could do it that way
But I did not understand what I was changing on the server side
files = {'file': ('report.xls', open('report.xls', 'rb'), 'application/vnd.ms-excel', {'Expires': '0'})}
What I need to change on the server side to get the file?
thanks
Related
Hi I am new to logic apps. I have a python code for sending mail with attachments using Azure Logic Apps. When I provide a static file path in Get Blob Content the mail is working fine with the attachment. But when I am trying to the send the file path dynamically from azure databricks it is not receiving at the get blob content in logic app. I have also tried few examples using initialise variables but it is not working.
import requests
def send_email(_to, _subject, _body, file):
email_body = "{0} <br><br><br> <h6> Hi Hello World </h6>".format(_body)
task = { "body_data_1": email_body , "subject_name_1": _subject , "email_to_list_1": _to, "attach" : file }
logic_app_post_url = 'https://prod-27.centralindia.logic.azure.com:443/workflows/*****'
resp = requests.post(logic_app_post_url, json=task)
print(resp.status_code)
if resp.status_code == 202:
print('Email Sent!')
email_to = "abc#outlook.com"
email_subject = "Testing Logic App Send Mail with Dynamic File"
email_body = "Hi This is a test mail Dynamic"
file_path = "/super-store/output/orders/_SUCCESS"
send_email(email_to, email_subject, email_body, file_path)
Logic App Design
Logic App Run Error Details
HTTP Input
HTTP Output
Get Blob Content Input
Get Blob Content Output
We have observed Get Blob Content (V2) is being used to send the attachments dynamically from azure Data bricks. Instead, we suggest using Get Blob Content using Path(V2) connector.
Wherein Get Blob Content (V2) connector will retrieve blob contents using id and Get Blob Content using Path (V2) connector retrieves blob contents using path.
Example with snippets:
Result:
I'm trying to get the direct download link for a file in Google Drive using the Google Drive API (v3), but I'm also trying to do this without making the file publicly shared.
Here is what I've tried:
https://www.googleapis.com/drive/v3/files/**FILE_ID**?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key=**API_KEY**
Now this works if the file is shared publicly. But when the file isn't shared publicly you get this message:
{'error': {'errors': [{'domain': 'global', 'reason': 'notFound', 'message': 'File not found: 10k0Qogwcz7k0u86m7W2HK-LO7bk8xAF8.', 'locationType': 'parameter', 'location': 'fileId'}], 'code': 404, 'message': 'File not found: 10kfdsfjDHJ38-UHJ34D82.'}}
After doing some googling I found a post on stack overflow saying that I need to add a request header with my access token, but this doesn't work and the application just hangs
Here is the full code:
### SETTING UP GOOGLE API
scopes = 'https://www.googleapis.com/auth/drive'
store = file.Storage('storage.json')
credentials = store.get()
if not credentials or credentials.invalid:
flow = client.flow_from_clientsecrets('client_secret.json', scopes)
credentials = tools.run_flow(flow, store)
accessToken = credentials.access_token
refreshToken = credentials.refresh_token
drive = build('drive', 'v3', credentials=credentials)
### SENDING REQUEST
req_url = "https://www.googleapis.com/drive/v3/files/"+file_id+"?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key="+GOOGLE_API_KEY
headers={'Authorization': 'Bearer %s' % accessToken}
request_content = json.loads((requests.get(req_url)).content)
print(request_content)
------------------ EDIT: ------------------
I've gotten really close to an answer, but I can't seem to figure out why this doesn't work.
So I've figured out previously that alt=media generates a download link for the file, but when the file is private this doesn't work.
I just discovered that you can add &access_token=.... to access private files, so I came up with this API call:
https://www.googleapis.com/drive/v3/files/**FILE_ID**?supportsAllDrives=true&alt=media&access_token=**ACCESS_TOKEN**&key=**API_KEY**
When I go to that url on my browser I get this message:
We're sorry...
... but your computer or network may be sending automated queries. To protect our users, we can't process your request right now.
I find this confusing because if I remove alt=media, I am able to call on that request and I get some metadata about the file.
I believe your goal as follows.
From I'm trying to get the direct download link for a file in Google Drive using the Google Drive API (v3),, I understand that you want to retrieve webContentLink.
The file that you want to retrieve the webContentLink is the files except for Google Docs files.
You have already been able to get the file metadata using Drive API. So your access token can be used for this.
Modification points:
When the file is not shared, the API key cannot be used. By this, https://www.googleapis.com/drive/v3/files/**FILE_ID**?alt=media&supportsAllDrives=True&includeItemsFromAllDrives=True&key=**API_KEY** returns File not found. I think that the reason of this issue is due to this.
When I saw your script in your question, it seems that you want to download the file content.
In your script, headers is not used. So in this case, the access token is not used.
In the method of "Files: get", there is no includeItemsFromAllDrives.
In your script, I think that an error occurs at credentials.access_token. How about this? If my understanding is correct, please try to modify to accessToken = credentials.token.
In Drive API v3, the default response values don't include webContentLink. So in this case, the field value is required to be set like fields=webContentLink.
When your script is modified, it becomes as follows.
Modified script:
file_id = '###' # Please set the file ID.
req_url = "https://www.googleapis.com/drive/v3/files/" + file_id + "?supportsAllDrives=true&fields=webContentLink"
headers = {'Authorization': 'Bearer %s' % accessToken}
res = requests.get(req_url, headers=headers)
obj = res.json()
print(obj.get('webContentLink'))
Or, you can use drive = build('drive', 'v3', credentials=credentials) in your script, you can also use the following script.
file_id = '###' # Please set the file ID.
drive = build('drive', 'v3', credentials=credentials)
request = drive.files().get(fileId=file_id, supportsAllDrives=True, fields='webContentLink').execute()
print(request.get('webContentLink'))
Note:
In this modified script,
When the file is in the shared Drive and you don't have the permissions for retrieving the file metadata, an error occurs.
When your access token cannot be used for retrieving the file metadata, an error occurs.
So please be careful above points.
When * is used for fields, all file metadata can be retrieved.
Reference:
Files: get
Added:
You want to download the binary data from the Google Drive by the URL.
The file size is large like "2-10 gigabytes".
In this case, unfortunately, webContentLink cannot be used. Because in the case of the such large file, webContentLink is redirected. So I think that the method that the file is publicly shared and use the API key is suitable for achieving your goal. But, you cannot publicly shared the file.
From this situation, as a workaround, I would like to propose to use this method. This method is "One Time Download for Google Drive". At Google Drive, when the publicly shared file is downloaded, even when the permission of file is deleted under the download, the download can be run. This method uses this.
Flow
In this sample script, the API key is used.
Request to Web Apps with the API key and the file ID you want to download.
At Web Apps, the following functions are run.
Permissions of file of the received file ID are changed. And the file is started to be publicly shared.
Install a time-driven trigger. In this case, the trigger is run after 1 minute.
When the function is run by the time-driven trigger, the permissions of file are changed. And sharing file is stopped. By this, the shared file of only one minute can be achieved.
Web Apps returns the endpoint for downloading the file of the file ID.
After you got the endpoint, please download the file using the endpoint in 1 minute. Because the file is shared for only one minute.
Usage:
1. Create a standalone script
In this workaround, Google Apps Script is used as the server side. Please create a standalone script.
If you want to directly create it, please access to https://script.new/. In this case, if you are not logged in Google, the log in screen is opened. So please log in to Google. By this, the script editor of Google Apps Script is opened.
2. Set sample script of Server side
Please copy and paste the following script to the script editor. At that time, please set your API key to the variable of key in the function doGet(e).
Here, please set your API key in the function of doGet(e). In this Web Apps, when the inputted API key is the same, the script is run.
function deletePermission() {
const forTrigger = "deletePermission";
const id = CacheService.getScriptCache().get("id");
const triggers = ScriptApp.getProjectTriggers();
triggers.forEach(function(e) {
if (e.getHandlerFunction() == forTrigger) ScriptApp.deleteTrigger(e);
});
const file = DriveApp.getFileById(id);
file.setSharing(DriveApp.Access.PRIVATE, DriveApp.Permission.NONE);
}
function checkTrigger(forTrigger) {
const triggers = ScriptApp.getProjectTriggers();
for (var i = 0; i < triggers.length; i++) {
if (triggers[i].getHandlerFunction() == forTrigger) {
return false;
}
}
return true;
}
function doGet(e) {
const key = "###"; // <--- API key. This is also used for checking the user.
const forTrigger = "deletePermission";
var res = "";
if (checkTrigger(forTrigger)) {
if ("id" in e.parameter && e.parameter.key == key) {
const id = e.parameter.id;
CacheService.getScriptCache().put("id", id, 180);
const file = DriveApp.getFileById(id);
file.setSharing(DriveApp.Access.ANYONE_WITH_LINK, DriveApp.Permission.VIEW);
var d = new Date();
d.setMinutes(d.getMinutes() + 1);
ScriptApp.newTrigger(forTrigger).timeBased().at(d).create();
res = "https://www.googleapis.com/drive/v3/files/" + id + "?alt=media&key=" + e.parameter.key;
} else {
res = "unavailable";
}
} else {
res = "unavailable";
}
return ContentService.createTextOutput(res);
}
3. Deploy Web Apps
On the script editor, Open a dialog box by "Publish" -> "Deploy as web app".
Select "Me" for "Execute the app as:".
Select "Anyone, even anonymous" for "Who has access to the app:". This is a test case.
If Only myself is used, only you can access to Web Apps. At that time, please use your access token.
Click "Deploy" button as new "Project version".
Automatically open a dialog box of "Authorization required".
Click "Review Permissions".
Select own account.
Click "Advanced" at "This app isn't verified".
Click "Go to ### project name ###(unsafe)"
Click "Allow" button.
Click "OK"
4. Test run: Client side
This is a sample script of python. Before you test this, please confirm the above script is deployed as Web Apps. And please set the URL of Web Apps, the file ID and your API key.
import requests
url1 = "https://script.google.com/macros/s/###/exec"
url1 += "?id=###fileId###&key=###your API key###"
res1 = requests.get(url1)
url2 = res1.text
res2 = requests.get(url2)
with open("###sampleFilename###", "wb") as f:
f.write(res2.content)
In this sample script, at first, it requests to the Web Apps using the file ID and API key, and the file is shared publicly in 1 minute. And then, the file can be downloaded. After 1 minute, the file is not publicly shared. But the download of the file can be kept.
Note:
When you modified the script of Web Apps, please redeploy the Web Apps as new version. By this, the latest script is reflected to the Web Apps. Please be careful this.
References:
One Time Download for Google Drive
Web Apps
Taking advantage of Web Apps with Google Apps Script
I have been trying to get my Python notebook in Watson Studio to connect to and retrieve data from my cloud storage for hours and read many tutorials but no idea why it is not working, code below:
credentials = {
'BUCKET': 'openai-data',
'URL': 'https://s3.private.us-east.cloud-object-storage.appdomain.cloud',
'SECRET_KEY': '',
'API_KEY': '*********************', #this key has been *d out but the value is there in my code
'RESOURCE_INSTANCE_ID': 'crn:v1:bluemix:public:cloud-object-storage:global:a/e448d36ef93f4d3ca55077db903d3461:51ce6e50-4e92-41d0-b002-5023e815cadc::',
'FILE': 'test_Y_smallsample.h5',
'ACCESS_KEY': ''
}
from ibm_botocore.client import Config
import ibm_boto3
cos = ibm_boto3.resource(service_name='s3',
ibm_api_key_id=credentials['API_KEY'],
ibm_service_instance_id=credentials['RESOURCE_INSTANCE_ID'],
ibm_auth_endpoint='https://iam.bluemix.net/oidc/token',
config=Config(signature_version='oauth'),
endpoint_url=credentials['URL'])
files = cos.Bucket('openai-data').objects.all()
for file in files:
print("Item: {0} ({1} bytes).".format(file.key, file.size))
this produces the error:
CredentialRetrievalError: Error when retrieving credentials from https://iam.bluemix.net/oidc/token: HttpCode(400) - Retrieval of tokens from server failed.
same result if I use "https://iam.cloud.ibm.com/identity/token" for the ibm_auth_endpoint
I've also tried a separate connection with HMAC credentials as well, but can't find any tutorials that show you how to incorporate those either...
Please help!
Thanks
This can be caused due to the use of invalid apikey. To get an apikey value, go to the 'Service credentials' menu of your storage instance and then click on "View credentials" of the "WDP-Project-Management-..." and see the value in the 'apikey' field.
This issue does not seem to be caused by invalid endpoint_url, bu anyway, to get the endpoint_url, go to Buckets > 'your bucket' > Configuration, and then choose an endpoint according to you needs (Private/Public/Direct). When filling the field, use "https://".
See below a code snippet to get data from a cvs file directly:
import pandas as pd
import ibm_boto3
from ibm_botocore.client import Config
cos = ibm_boto3.client(service_name='s3',
ibm_api_key_id='<apikey>',
ibm_auth_endpoint="https://iam.ng.bluemix.net/oidc/token",
config=Config(signature_version='oauth'),
endpoint_url="<endpoint>")
obj = cos.get_object(Bucket='<bucket_name>', Key='<csv_file>')
df = pd.read_csv(obj['Body'])
I am trying to upload documents to the SharePoint online.
Target url: https://companyURL.sharepoint.com/sites/A/B/Info_documents/C
My aim is to search folder C, and if X subfolder is present in folder C, I need to upload a file. To accomplish this, I have generated client_id and client_secret by navigating to http://{sharepointsite}/_layouts/15/AppRegNew.aspx. In the XML permissions, I gave the following code:
I am using https://github.com/vgrem/Office365-REST-Python-Client for this implementation. When trying to use the following code snippet to see, if I am having the access to sharepoint using client_id and client_secret, i am seeing different errors:
import json
from office365.runtime.auth.authentication_context import AuthenticationContext
from office365.runtime.client_request import ClientRequest
from office365.runtime.utilities.request_options import RequestOptions
from office365.sharepoint.client_context import ClientContext
app_settings = {
'url': 'https://companyURL.sharepoint.com/sites/A/B/Info_documents/C',
'client_id': 'xxxxxxxx',
'client_secret': 'xxxxxx',
}
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'], client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
web = ctx.web
ctx.load(web)
ctx.execute_query()
print("Web site title: {0}".format(web.properties['Title']))
Error:
ClientRequestException: ('-2147024891, System.UnauthorizedAccessException', 'Access denied. You do not have permission to perform this action or access this resource.', '403 Client Error: Forbidden for url: https://companyURL.sharepoint.com/sites/A/B/Info_documents/C_api/Web')
But I gave the permissions, not sure whether I am doing wrong or selected the wrong module.
Please help.
Following is the XML code i gave when generating the client_ID and client_secret:
<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web" Right="Read"/>
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection/web/list" Right="Write"/>
</AppPermissionRequests>
Given the description
My aim is to search folder C, and if X subfolder is present in folder
C, I need to upload a file.
and the the following link format:
https://companyURL.sharepoint.com/sites/A/B/Info_documents/C
your folder structure could be represented like this:
https://companyURL.sharepoint.com/sites/A/B/ <-this is the actual site url
|
Info_documents <-library
|
C <-folder
Since AuthenticationContext class accepts the first parameter to be site url, app_settings needs to be updated like this:
app_settings = {
'url': 'https://companyURL.sharepoint.com/sites/A/B/', //need to refer to site url
'client_id': '--client id goes here--',
'client_secret': '--client secret goes here--',
}
Now comes the turn of App principal, since the requested permissions are applicable per web (scope: http://sharepoint/content/sitecollection/web), the second step (granting permissions) needs to be accomplished per specified web:
https://companyURL.sharepoint.com/sites/A/B/_layouts/15/appinv.aspx
Example
Here is an example which demonstrates how to verify if sub folder exists under the parent folder:
context_auth = AuthenticationContext(url=app_settings['url'])
context_auth.acquire_token_for_app(client_id=app_settings['client_id'],
client_secret=app_settings['client_secret'])
ctx = ClientContext(app_settings['url'], context_auth)
folder_url = "Info_documents/C" #folder url where to find
folder_name = "X" #folder name to find
result = ctx.web.get_folder_by_server_relative_url(folder_url).folders.filter("Name eq '{0}'".format(folder_name))
ctx.load(result)
ctx.execute_query()
if len(result) > 0:
print("Folder has been found: {0}".format(result[0].properties["Name"]))
I am trying to update a file that I created with some new data. Essentially, i am trying to append contents to an already created file. I have tried different ways using different attributes but nothing seems to work so far. The migration from v2 to v3 seems to have made things harder to develop in python.
This is my code so far
def updateFile(fileID, contents):
credentials = get_credentials() #this function gets the credentials for oauth
http = credentials.authorize(httplib2.Http())
service = discovery.build('drive', 'v3', http=http)
# First retrieve the file from the API.
#fileCreated = service.files().get(fileId=fileID).execute()
#print(dir(fileCreated.update()))
# File's new content.
file_metadata = { 'name' : 'notes.txt', 'description' : 'this is a test' }
fh = BytesIO(contents.encode())
media = MediaIoBaseUpload(fh,
mimetype='text/plain')
# Send the request to the API.
#print(BytesIO(contents.encode()).read())
print(fileID)
updated_file = service.files().update(
body=file_metadata,
#uploadType = 'media',
fileId=fileID,
#fields = fileID,
media_body=media).execute()
I have tried using MediaFileUpload (it works only for uploading a file) but I am appending a 'string' that is real time generated using a voice interface and is not stored in any file. So I ended up using MediaIoBaseUpload.
The code runs without any error, but the file is not updated. Any help is appreciated.
okay, so i figured out what was wrong with my code. Apparently, nothing is wrong with this chunk of code that I posted. I realized that I have not converted the document I created to a "google document" hence the code didnt update the document. I changed the mime type of the document i create to be a google document, and now the code works fine :)
You can also use service.files().get_media(fileId=fileID).execute() to get the file content and append new content to it.