We Keep Coding

Google API/Python: Delete old files in Google Drive by filename in local folder, upload new filenames from local folder

I have a problem witch was hard to write in the title. I have this script with a lot of help from #Tanaike . This script is doing basically two things:
Deletes files from Google Drive folder by filenames which are in local folder CSVtoGD (using spreadsheet ID's)
then:
Upload list of CSV from local folder "CSVtoGD" to Google Drive folder
I have a big problem now and can not work it out. The script is deleting old files in google drive when there are the same filenames in CSVtoGD. When I add new file to local folder CSVtoGD, there is a error "list index out of range" and I got printed "No files found" like in the script. I was trying to make some modification but it was blind shoots. What I want this script to do is to delete from Google Drive folder ONLY files which are in local CSVtoGD folder and work on with rest of the files in CSVtoGD (just upload them). Anyone have some answer to that? Thank you :)
import gspread
import os
from googleapiclient.discovery import build
gc = gspread.oauth(credentials_filename='/users/user/credentials.json')
service = build("drive", "v3", credentials=gc.auth)
def getSpreadsheetId(filename):
q = "name='" + filename + "' and mimeType='application/vnd.google-apps.spreadsheet' and trashed=false"
res = service.files().list(q=q, fields="files(id)", corpora="allDrives", includeItemsFromAllDrives=True, supportsAllDrives=True).execute()
items = res.get("files", [])
if not items:
print("No files found.")
exit()
return items[0]["id"]
os.chdir('/users/user/CSVtoGD2')
files = os.listdir()
for filename in files:
fname = filename.split(".")
if fname[1] == "csv":
folder_id = '1z_pUvZyt5AoTNy-aKCKLmlNjdR2OPo'
oldSpreadsheetId = getSpreadsheetId(fname[0])
#print(oldSpreadsheetId)
sh = gc.del_spreadsheet(oldSpreadsheetId)
**# IF there are the same filenames in CSVtoGD folder on my Mac
#and the same filenames on Google Drive folder,
#those lines works well.
#Problem is when there are new files in CSVtoGD local folder on Mac.**
sh = gc.create(fname[0], folder_id)
content = open(filename, "r").read().encode("utf-8")
gc.import_csv(sh.id, content)

I believe your goal is as follows.
For example, when sample.csv is existing on your local PC and a Spreadsheet of sample is existing in your Google Drive, you want to delete the Spreadsheet of sample from your Google Drive.
When sample1.csv is existing on your local PC and the Spreadsheet of sample1 is NOT existing in your Google Drive, you want to upload sample1.csv to Google Drive.
In this case, how about the following modification?
Modified script:
import gspread
import os
from googleapiclient.discovery import build
from googleapiclient.http import MediaFileUpload
gc = gspread.oauth(credentials_filename='/users/user/credentials.json')
service = build("drive", "v3", credentials=gc.auth)
folder_id = '1z_pUvZyt5AoTNy-aKCKLmlNjdR2OPo' # Please set the folder ID you want to upload the file.
def getSpreadsheetId(filename, filePath):
q = "name='" + filename + "' and mimeType='application/vnd.google-apps.spreadsheet' and trashed=false"
res = service.files().list(q=q, fields="files(id)", corpora="allDrives", includeItemsFromAllDrives=True, supportsAllDrives=True).execute()
items = res.get("files", [])
if not items:
print("No files found.")
file_metadata = {
"name": filename,
"parents": [folder_id],
"mimeType": "application/vnd.google-apps.spreadsheet",
}
media = MediaFileUpload(filePath + "/" + filename + ".csv")
file = service.files().create(body=file_metadata, media_body=media, fields="id").execute()
id = file.get("id")
print("File was uploaded. The file ID is " + id)
exit()
return items[0]["id"]
filePath = '/users/user/CSVtoGD2'
os.chdir(filePath)
files = os.listdir()
for filename in files:
fname = filename.split(".")
if fname[1] == "csv":
oldSpreadsheetId = getSpreadsheetId(fname[0], filePath)
print(oldSpreadsheetId)
sh = gc.del_spreadsheet(oldSpreadsheetId)
sh = gc.create(fname[0], folder_id)
content = open(filename, "r").read().encode("utf-8")
gc.import_csv(sh.id, content)
When this script is run, the above flow is run.
Note:
In this modification, the CSV file is uploaded as a Google Spreadsheet. From your question, I thought that this might be your expected result. But, if you want to upload the CSV file as the CSV file, please remove "mimeType": "application/vnd.google-apps.spreadsheet", from file_metadata.
If an error related to the scope, please add the scope of https://www.googleapis.com/auth/drive and authorize the scopes again and test it again.
Reference:
Upload file data

I can see only files created with my script pydrive

As the title says I have an issue with pydrive. I ran the code given in the pydrive quickstart (https://googleworkspace.github.io/PyDrive/docs/build/html/quickstart.html) and I created a settings and credentials file to avoid entering my credentials all the time.
But when I run this code:
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
# Rename the downloaded JSON file to client_secrets.json
# The client_secrets.json file needs to be in the same directory as the script.
gauth = GoogleAuth()
drive = GoogleDrive(gauth)
# List files in Google Drive
fileList = drive.ListFile().GetList()
for drive_file in fileList:
print('title: %s, id: %s' % (drive_file['title'], drive_file['id']))
I can only see the files created with my script. For example if I add this before the list file:
folder = drive.ListFile({'q': "title = 'Python_test' and trashed=false"}).GetList()[0] # get the folder we just created
file = drive.CreateFile({'title': "test.txt", 'parents': [{'id': folder['id']}]})
file.Upload()
I only see the folder and the file ID I just created... And if I add manually a file on my drive (on my browser for example), it doesn't appear.
Anyone got an idea of what's going on?

I just found the problem, it was on my settings.yaml file, I added only this oauth_scope:
oauth_scope:
- https://www.googleapis.com/auth/drive.file
but this gives only access to the files created by the app. To correct that I needed to remove the .file like this
oauth_scope:
- https://www.googleapis.com/auth/drive
If you want more details about the different scopes, check this link:
https://developers.google.com/identity/protocols/oauth2/scopes

Why does Google Drive API corrupt my file extension when uploading files through Python?

I have created a Python script that pulls files with a .xlsx format from a folder on my computer and uploads the file to a specific folder in my Google Drive. This is using the pydrive package in Python. The script runs with no issues, and the files are uploaded as expected. However, for some reason, when the uploaded Google Drive file is downloaded and re-opened, Excel gives the following error message:
Excel cannot open the file...because the file format or file extension is not valid. Verify that the file has not been corrupted and that the file extension matches the format of the file.
When I open the file directly on my computer, it opens fine with no issues. When I manually drag/upload the file into the Google Drive folder, and then re-download the file, it opens with no problem. The transformation seems to be coming from my Python script (see below).
Can anyone provide any help here? I have been trying different things and I keep getting the same result. I can provide more information if necessary.
Updated to add full Python Script:
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import glob,os,shutil
import datetime
os.chdir(os.path.dirname(os.path.abspath(__file__)))
gauth = GoogleAuth()
#gauth.LocalWebserverAuth()
# Try to load saved client credentials
gauth.LoadCredentialsFile("mycreds.txt")
if gauth.credentials is None:
# Authenticate if they're not there
gauth.LocalWebserverAuth()
elif gauth.access_token_expired:
# Refresh them if expired
gauth.Refresh()
else:
# Initialize the saved creds
gauth.Authorize()
# Save the current credentials to a file
gauth.SaveCredentialsFile("mycreds.txt")
drive = GoogleDrive(gauth)
#Google Drive Folder ID
fid = '[FOLDER ID PLACEHOLDER]'
#Check to see if today's folder is created on PC
date = datetime.date.today()
today = date.strftime('%Y-%m-%d')
starting_folder = '[FOLDER PLACEHOLDER]'
if not os.path.exists(starting_folder + "/" + today):
os.makedirs(starting_folder + "/" + today)
destination_folder = starting_folder + "/" + today
#Change directory to the folder where the bulk bid tools are stored
os.chdir("[FOLDER PLACEHOLDER]")
for file in glob.glob("*.xlsx"):
try:
print(file)
with open(file,"r") as f:
fn = os.path.basename(f.name)
file_drive = drive.CreateFile({'parents':[{'kind':"drive#parentReference",'id':fid}],'title':fn})
file_drive.Upload()
print("The file: " + fn + " has been uploaded.")
shutil.move(starting_folder + "/" + fn,destination_folder + "/" + fn)
except:
pass
print("All files have been uploaded")

You are not actually passing the information of the file to the request. Meaning, you are not really uploading the actual "bytes" of this file to Drive. Just creating an empty drive file with the same name in the determined folder.
If you look at the documentation for pyDrive you can see that after calling of CreateFile they use SetContentFile.
Copied from the documentation you can see an example like so:
file2 = drive.CreateFile()
file2.SetContentFile('hello.png') # This line needs to be added to your code
# with the name of the file in your computer
file2.Upload()
# Also check the mimetype afterwards to check the file has been correctly uploaded
print('Created file %s with mimeType %s' % (file2['title'],
file2['mimeType']))
# Created file hello.png with mimeType image/png
Also from the comments you say you are still running python2 code. Take into consideration that python 2 is "dead" and there will not be more security updates, neither development/support. You should really be considering change as a lot of package and modules are also dropping (or will start to do so) python 2 support.
More information of this issue in Sunsetting Python 2.

Upload one file to multi folder google driver via API with python

I want to upload a file to multi folder google driver via API but only one file will be save, not each file for each folder. (1 file in serveral folder)
Example using by hand: Add the Same File to Multiple Folders in Google Drive without Copying
Could you please help me! Thank you!

In inserting a file in a folder, you need to specify the correct folder ID in the parents property of the file. Using Python:
folder_id = '0BwwA4oUTeiV1TGRPeTVjaWRDY1E'
file_metadata = {
'name' : 'photo.jpg',
'parents': [ folder_id ]
}
media = MediaFileUpload('files/photo.jpg',
mimetype='image/jpeg',
resumable=True)
file = drive_service.files().create(body=file_metadata,
media_body=media,
fields='id').execute()
print 'File ID: %s' % file.get('id')
As further mentioned in Files: insert, setting the parents[] property in the request body will put the file in all of the provided folders. If no folders are provided in parents[] field, the file will be placed in the default root folder.
Hope that helps!

Download S3 Files with Boto

I am trying to set up an app where users can download their files stored in an S3 Bucket. I am able to set up my bucket, and get the correct file, but it won't download, giving me the this error: No such file or directory: 'media/user_1/imageName.jpg' Any idea why? This seems like a relatively easy problem, but I can't quite seem to get it. I can delete an image properly, so it is able to identify the correct image.
Here's my views.py
def download(request, project_id=None):
conn = S3Connection('AWS_BUCKET_KEY', 'AWS_SECRET_KEY')
b = Bucket(conn, 'BUCKET_NAME')
k = Key(b)
instance = get_object_or_404(Project, id=project_id)
k.key = 'media/'+str(instance.image)
k.get_contents_to_filename(str(k.key))
return redirect("/dashboard/")

The problem is that you are downloading to a local directory that doesn't exist (media/user1). You need to either:
Create the directory on the local machine first
Just use the filename rather than a full path
Use the full path, but replace slashes (/) with another character -- this will ensure uniqueness of filename without having to create directories
The last option could be achieved via:
k.get_contents_to_filename(str(k.key).replace('/', '_'))
See also: Boto3 to download all files from a S3 Bucket

Downloading files using boto3 is very simple, configure your AWS credentials at system level before using this code.
client = boto3.client('s3')
// if your bucket name is mybucket and the file path is test/abc.txt
// then the Bucket='mybucket' Prefix='test'
resp = client.list_objects_v2(Bucket="<your bucket name>", Prefix="<prefix of the s3 folder>")
for obj in resp['Contents']:
key = obj['Key']
//to read s3 file contents as String
response = client.get_object(Bucket="<your bucket name>",
Key=key)
print(response['Body'].read().decode('utf-8'))
//to download the file to local
client.download_file('<your bucket name>', key, key.replace('test',''))
replace is to locate the file in your local with s3 file name, if you don't replace it will try to save as 'test/abc.txt'.

import os
import boto3
import json
s3 = boto3.resource('s3', aws_access_key_id="AKIAxxxxxxxxxxxxJWB",
aws_secret_access_key="LV0+vsaxxxxxxxxxxxxxxxxxxxxxry0/LjxZkN")
my_bucket = s3.Bucket('s3testing')
# download file into current directory
for s3_object in my_bucket.objects.all():
# Need to split s3_object.key into path and file name, else it will give error file not found.
path, filename = os.path.split(s3_object.key)
my_bucket.download_file(s3_object.key, filename)