I have been successfully uploading files to a google-drive-folder with PyDrive. But, when it comes to uploading files to a folder in a google-drive-teamdrive-folder which is shared with me, the following code is not working.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)
location_to_save = "D:\images"
mImageLoc = location_to_save + "\\abcd.jpg"
#[...Code to fetch and save the file as abcd.jpg ...]
gfolder_id = "1H1gjBKcpiHJtnXKVxWQEC1CS8t4Gswjj" #This is a google drive folder id. I am replacing this with a teamdrive folder id, but that does not work
gfile_title = mImageLoc.split("\\")[-1] # returns abcd.jpg
http = gdrive.auth.Get_Http_Object()
f = gdrive.CreateFile({"parents": [{"kind": "drive#fileLink", "id": gfolder_id}],
'title': gfile_title})
f.SetContentFile(mImageLoc)
f.Upload(param={"http": http})
The error message I am recieving is: pydrive.files.ApiRequestError: <HttpError 404 when requesting https://www.googleapis.com/upload/drive/v2/files?alt=json&uploadType=resumable returned "File not found: 0AG-N4DqGC1nbUk9PVA">
'0AG-N4DqGC1nbUk9PVA' is the teamdrive's folder id here.
I have been searching for means to upload files to Teamdrives with PyDrive but in vain. I see in the pydrive's github pages that they added the teamdrives support approx 8 month ago. But I cannot find any documentation on how to use that. Can anyone suggest where I am being wrong please?
For uploading, try making a file called "settings.yaml" and saving it in your working directory, as per the instructions here:
https://pythonhosted.org/PyDrive/oauth.html
You will need the client id and client secret found in the client_secrets.json file which should also be in your directory after you authorised access to the Google API.
Test it out with the following code to make a text file in a folder in the team drive:
parent_folder_id = 'YYYY'
f = drive.CreateFile({
'title': 'test.txt',
'parents': [{
'kind': 'drive#fileLink',
'teamDriveId': team_drive_id,
'id': parent_folder_id
}]
})
f.SetContentString('Hello World')
f.Upload(param={'supportsTeamDrives': True})
# where XXXX and YYYY are the team drive and target folder ids found from the end of the URLS when you open them in your browser.
Related
I have a bit of python code using PyDrive, to download files from a shared google drive folder. It looks like this.
from pydrive2.auth import GoogleAuth
from pydrive2.drive import GoogleDrive
gauth = GoogleAuth()
gauth.CommandLineAuth()
drive = GoogleDrive(gauth)
file_list = drive.ListFile({'q': "'XXXXXXXXXXXXXXXXXXXXXXXX' in parents and trashed=false"}).GetList()
for file2 in file_list:
string = "'"+ file2['id'] +"'" + " in parents and trashed=false"
file_list2 = drive.ListFile({'q': string}).GetList()
print(file2['title'], file2['id'])
for file3 in file_list2:
file3.GetContentFile('/data/' + file3['title'])
print('downloaded: ', file3['title'], file3['id'])
This downloads roughly a TB of data in many tens of thousands of files. (The original file link is not XXXXXXXX, I just added that to avoid using the actual file ID in this example). The code works great for like the first 25k files, when it suddenly fails with
code: 403,
message: Daily Limit for Unauthenticated Use Exceeded. Continued use requires signup.
I sign in successfully with OAuth and my google drive api is enabled and it has a quota of 10,000,000 requests a day (far more than I am using). Why does it think I am still using an anonymous account and hitting me with this error?
It is my understanding that because it is a file that I have in my shared drive and because I am signed in I shouldn't be encountering this, whats up with that?
Can anyone tell me how to put project id of shared drive in GoogleAuth()?
I have tried the below chunk of code but none of them are working:
auth = GoogleAuth({'id': 'projectid'})
auth = GoogleAuth({'project_id': 'projectid'})
auth = GoogleAuth({'project': 'projectid'})
Below is my piece of code where I am trying to upload a .csv file to a shared drive. I assume that the project ID is the string after the last '/' in the URL which appears after we double click on the desired drive folder.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
auth = GoogleAuth()
drive = GoogleDrive(gauth)
file1 = drive.CreateFile()
file1.SetContentFile('file_name.csv')
file1.Upload()`
I am performing LDA on a simple wikipedia dump file, but the code I am following needs to output the articles to a file. I need some guidance as python and colab are really broad and I can't seem to find an answer to this specific problem. Here's my code for mounting google drive:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate the user
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
# Get your file
fileId ='xxxx'
fileName = 'simplewiki-20170820-pages-meta-current-reduced.xml'
downloaded = drive.CreateFile({'id': fileId})
downloaded.GetContentFile(fileName)
and here's the culprit, this code is trying to create a file from the article
if not article_txt == None and not article_txt == "" and len(article_txt) > 150 and is_ascii(article_txt):
outfile = dir_path + str(i+1) +"_article.txt"
f = codecs.open(outfile, "w", "utf-8")
f.write(article_txt)
f.close()
print (article_txt)
I have tried so many things already and I can't recall them all. Basically, what I need to know is how to convert this code so that it would work with google drive. I've been trying so many solutions for hours now. Something I recall doing is converting this code into this
file_obj = drive.CreateFile()
file_obj['title'] = "file name"
But then I got an error 'expected str, bytes or os.PathLike object, not GoogleDriveFile'. It's not the question of how to upload a file and open it with colab, as I already know how to do that with the XML file, what I need to know is how to generate files through my colab script and place them to the same folder as my script. Any help would be appreciated. Thanks!
I am not sure whether the problem is with generating the files or copying them to google drive, if it is the latter, a simpler approach would be to mount your drive directly to the instance as follows
from google.colab import drive
drive.mount('drive')
You can then access any item in your drive as if it were a hard disk and copy your files using bash commands:
!cp filename 'drive/My Drive/folder1/'
Another alternative is to use shutil :
import shutil
shutil.copy(filename, 'drive/My Drive/folder1/')
Wanted to try out python, and google colaboratory seemed the easiest option.I have some files in my google drive, and wanted to upload them into google colaboratory.
so here is the code that i am using:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
# 2. Create & upload a file text file.
uploaded = drive.CreateFile({'xyz.csv': 'C:/Users/abc/Google Drive/def/xyz.csv'})
uploaded.Upload()
print('Uploaded file with title {}'.format(uploaded.get('title')))
import pandas as pd
xyz = pd.read_csv('Untitled.csv')
Basically, for user "abc", i wanted to upload the file xyz.csv from the folder "def".
I can upload the file, but when i ask for the title it says the title is "Untitled".
when i ask for the Id of the file that was uploaded, it changes everytime, so i can not use the Id.
How do i read the file??? and set a proper file name???
xyz = pd.read_csv('Untitled.csv') doesnt work
xyz = pd.read_csv('Untitled') doesnt work
xyz = pd.read_csv('xyz.csv') doesnt work
Here are some other links that i found..
How to import and read a shelve or Numpy file in Google Colaboratory?
Load local data files to Colaboratory
To read a csv file from my google drive into colaboratory, I needed to do the following steps:
1) I first needed to authorize colaboratory to access my google drive with PyDrive. I used their code example for that. (pasted below)
2) I also needed to log into my drive.google.com to find the target id of the file i wanted to download. I found this by right clicking on the file and copying the shared link for the ID. The id looks something like this: '1BH-rffqv_1auzO7tdubfaOwXzf278vJK'
3) Then I ran downloaded.GetContentFile('myName.csv') - putting in the name i wanted (in your case it is xyz.csv)
This seems to work for me!
I used the code they provided in their example:
# Code to read csv file into colaboratory:
!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# 1. Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
#2. Get the file
downloaded = drive.CreateFile({'id':'1BH-rffqv_1auzO7tdubfaOwXzf278vJK'}) # replace the id with id of file you want to access
downloaded.GetContentFile('xyz.csv')
#3. Read file as panda dataframe
import pandas as pd
xyz = pd.read_csv('xyz.csv')
Okay I'm pretty sure I'm quite late, but I'd like to put this out there, just in case.
I think the easiest way you could do this is by
from google.colab import drive
drive.mount("/content/drive")
This will generate a link, click on it and sign in using Google OAuth, paste the key in the colab cell and you're connected!
check out the list of available files in the side bar on the left side and copy the path of the file you want to access. Read it as you would, with any other file.
File create takes a file body i its first parameter. If you check the documentation for file create there are a number of fields you can fill out. In the example below you would add them to file_metadata comma separated.
file_metadata = {'name': 'photo.jpg'}
media = MediaFileUpload('files/photo.jpg',
mimetype='image/jpeg')
file = drive_service.files().create(body=file_metadata,
media_body=media,
fields='id').execute()
I suggest you read the file upload section of the documentation to get a better idea how upload works and which files can actually be read from within google drive. I am not sure that this is going to give you access to Google colaborate
Possible fix for your code.
I am not a python dev but my guess would be you can set your title by doing this.
uploaded = drive.CreateFile({'xyz.csv': 'C:/Users/abc/Google Drive/def/xyz.csv',
'name': 'xyz.csv'})
I think it's that simple with this command
# Mount Google Drive
import os
from google.colab import drive
drive.mount('/content/drive')
!pwd
!ls
import pandas as pd
df = pd.read_csv('Untitled.csv')
It will require authorization with your Google OAuth, and create authorization key. put the key into the colab cell.
Please aware !, sometimes the file within google colab directory are not update or similar with google drive if you delete or add files in your Google Drive.
The flow of my desired program is:
Upload an xlsx spreadsheet to drive (it was created using pandas to_excel)
Convert it to Google Sheets format
Specify that it is editable by anyone with the link
Get the link and share it with someone who will enter information
Download the completed sheet
I am currently using PyDrive, which solves steps 1 and 5, but there are a few unsolved problems.
How can I convert to google sheets format? I tried to just specify the mimeType as 'application/vnd.google-apps.spreadsheet' when I created the file to upload with PyDrive, but that gave me an error.
How can I set the file to be editable by anyone with the link? Once that is set, I can get the sharing link easily enough with PyDrive.
UPDATE: conversion from xlsx to google sheets is easy with a convert=True flag. See below. I am still seeking a way to set the sharing settings of my new file to "anyone with the link can edit".
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)
test_file = drive.CreateFile({'title': 'testfile.xlsx'})
test_file.SetContentFile('testfile.xlsx')
test_file.Upload({'convert': True})
There is an Optional query parameter of "convert", for both the "INSERT" and "COPY" method;
convert=true,
Whether to convert this file to the corresponding Google Docs format. (Default: false)
There is a python example here:
Google Documentation - Copy
You need to use the Python client library for the code to work.
from apiclient import errors
from apiclient.http import MediaFileUpload
# ...
def insert_file(service, title, description, parent_id, mime_type, filename):
"""Insert new file.
Args:
service: Drive API service instance.
title: Title of the file to insert, including the extension.
description: Description of the file to insert.
parent_id: Parent folder's ID.
mime_type: MIME type of the file to insert.
filename: Filename of the file to insert.
Returns:
Inserted file metadata if successful, None otherwise.
"""
media_body = MediaFileUpload(filename, mimetype=mime_type, resumable=True)
body = {
'title': title,
'description': description,
'mimeType': mime_type
}
# Set the parent folder.
if parent_id:
body['parents'] = [{'id': parent_id}]
try:
file = service.files().insert(
body=body,
convert=true,
media_body=media_body).execute()
# Uncomment the following line to print the File ID
# print 'File ID: %s' % file['id']
return file
except errors.HttpError, error:
print 'An error occured: %s' % error
return None
I haven't tried this, so you'll need to test it.
In order to set the file to be editable for anyone with the link , you have to insert a new permission with the following information:
from apiclient import errors
# ...
def share_with_anyone(service, file_id):
"""Shares the file with anyone with the link
Args:
service: Drive API service instance.
file_id: ID of the file to insert permission for.
Returns:
The inserted permission if successful, None otherwise.
"""
new_permission = {
'type': "anyone",
'role': "writer",
'withLink': True
}
try:
return service.permissions().insert(
fileId=file_id, body=new_permission).execute()
except errors.HttpError, error:
print 'An error occurred: %s' % error
return None
then to get the link you go to : file["alternateLink"]