We Keep Coding

Upload new files to bucket from local drive

I would like to write a script which will detect new files (csv files in this case) that have been added to a folder then upload these new files to my AWS S3 Bucket. I would like them to maintain the original name. Currently the script i have allows me to manually select a file then upload it with a name of my choice.
hc = pd.read_csv((open(r'CSV PATH')))
s3 = boto3.client('s3',aws_access_key_id = 'ACCESSKEYID',
aws_secret_access_key = 'ACCESSKEY')
csv_buf = StringIO()
hc.to_csv(csv_buf, header = True, index = False)
csv_buf.seek(0)
s3.put_object(Bucket = 'BucketName', Body = csv_buf.getvalue(), Key = 'Original CSV Name from Above')
I assume i need the following section in the code:
Code to monitor said location (but only when running the app - does not need to run 24/7)
Code to pull new file from said location
Code to upload to S3 Bucket
Any Tips?

How to read file using sharepoint REST API in Python

I am an absolute beginner when it comes to working with REST APIs with python. We have received a share-point URL which has multiple folders and multiples files inside those folders in the 'document' section. I have been provided an 'app_id' and a 'secret_token'.
I am trying to access the .csv file and read them as a dataframe and perform operations.
The code for operation is ready after I downloaded the .csv and did it locally but I need help in terms of how to connect share-point using python so that I don't have to download such heavy files ever again.
I know there had been multiple queries already on this over stack-overflow but none helped to get to where I want.
I did the following and I am unsure of what to do next:
import json
from office365.runtime.auth.user_credential import UserCredential
from office365.sharepoint.client_context import ClientContext
from office365.runtime.http.request_options import RequestOptions
site_url = "https://<company-name>.sharepoint.com"
ctx = ClientContext(site_url).with_credentials(UserCredential("{app_id}", "{secret_token}"))
Above for site_url, should I use the whole URL or is it fine till ####.com?
This is what I have so far, next I want to read files from respective folders and convert them into a dataframe? The files will always be in .csv format
The example hierarchy of the folders are as follows:
Documents --> Folder A, Folder B
Folder A --> a1.csv, a2.csv
Folder B --> b1.csv, b2.csv
I should be able to move to whichever folder I want and read the files based on my requirement.
Thanks for the help.

This works for me, using a Sharepoint App Identity with an associated client Id and client Secret.
First, I demonstrate authenticating and reading a specific file, then getting a list of files from a folder and reading the first one.
import pandas as pd
import json
import io
from office365.sharepoint.client_context import ClientCredential
from office365.sharepoint.client_context import ClientContext
from office365.sharepoint.files.file import File
#Authentication (shown for a 'modern teams site', but I think should work for a company.sharepoint.com site:
site="https://<myteams.companyname.com>/sites/<site name>/<sub-site name>"
#Read credentials from a json configuration file:
spo_conf = json.load(open(r"conf\spo.conf", "r"))
client_credentials = ClientCredential(spo_conf["RMAppID"]["clientId"],spo_conf["RMAppID"]["clientSecret"])
ctx = ClientContext(site).with_credentials(client_credentials)
#Read a specific CSV file into a dataframe:
folder_relative_url = "/sites/<site name>/<sub site>/<Library Name>/<Folder Name>"
filename = "MyFileName.csv"
response = File.open_binary(ctx, "/".join([folder_relative_url, filename]))
df = pd.read_csv(io.BytesIO(response.content))
#Get a list of file objects from a folder and read one into a DataFrame:
def getFolderContents(relativeUrl):
contents = []
library = ctx.web.get_list(relativeUrl)
all_items = library.items.filter("FSObjType eq 0").expand(["File"]).get().execute_query()
for item in all_items: # type: ListItem
cur_file = item.file
contents.append(cur_file)
return contents
fldrContents = getFolderContents('/sites/<site name>/<sub site>/<Library Name>')
response2 = File.open_binary(ctx, fldrContents[0].serverRelativeUrl)
df2 = pd.read_csv(io.BytesIO(response2.content))
Some References:
Related SO thread.
Office365 library github site.
Getting a list of contents in a doc library folder.
Additional notes following up on comments:
The site path doesn't not include the full url for the site home page (ending in .aspx) - it just ends with the name for the site (or sub-site, if relevant to your case).
You don't need to use a configuration file to store your authentication credentials for the Sharepoint application identity - you could just replace spo_conf["RMAppID"]["clientId"] with the value for the Sharepoint-generated client Id and do similarly for the client Secret. But this is a simple example of what the text of a JSON file could look like:
{
"MyAppName":{
"clientId": "my-client-id",
"clientSecret": "my-client-secret",
"title":"name_for_application"
}
}

Uploading image string to Google Drive using pydrive

I need to upload an image string (as the one you get from requests.get(url).content) to google drive using the PyDrive package. I checked a similar question but the answer accepted there was to save it in a temporary file on a local drive and then upload that.
However, I cannot do that because of local storage and permission restrictions.
The accepted answer was previously to use SetContentString(image_string.decode('utf-8')) since
SetContentString requires a parameter of type str not bytes.
However the error: UnicodeDecodeError: 'utf-8' codec can't decode byte 0x89 in position 0: invalid start byte came up, as in the comments on that answer.
Is there any way to do this without using a temporary file, using PIL/BytesIO/anything that can convert it to be uploaded correctly as a string or somehow using PIL manipulated as an image and uploaded using SetContentFile()?
A basic example of what I'm trying to do is:
img_content = requests.get('https://i.imgur.com/A5gIh7W.jpeg')
file = drive.CreateFile({...})
file.setContentString(img_content.decode('utf-8'))
file.Upload()

When I saw the document (Upload and update file content) of pydrive, it says as follows.
Managing file content is as easy as managing file metadata. You can set file content with either SetContentFile(filename) or SetContentString(content) and call Upload() just as you did to upload or update file metadata.
And, I searched about the method for directly uploading the binary data to Google Drive. But, I couldn't find it. From this situation, I thought that there might not be such method. So, in this answer, I would like to propose to upload the binary data using requests module. In this case, the access token is retrieved from the authorization script of pydrive. The sample script is as follows.
Sample script:
from pydrive.auth import GoogleAuth
import io
import json
import requests
url = 'https://i.imgur.com/A5gIh7W.jpeg' # Please set the direct link of the image file.
filename = 'sample file' # Please set the filename on Google Drive.
folder_id = 'root' # Please set the folder ID. The file is put to this folder.
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
metadata = {
"name": filename,
"parents": [folder_id]
}
files = {
'data': ('metadata', json.dumps(metadata), 'application/json'),
'file': io.BytesIO(requests.get(url).content)
}
r = requests.post(
"https://www.googleapis.com/upload/drive/v3/files?uploadType=multipart",
headers={"Authorization": "Bearer " + gauth.credentials.access_token},
files=files
)
print(r.text)
Note:
In this script, it supposes that your URL is the direct link of the image file. Please be careful this.
In this case, uploadType=multipart is used. The official document says as follows. Ref
Use this upload type to quickly transfer a small file (5 MB or less) and metadata that describes the file, in a single request. To perform a multipart upload, refer to Perform a multipart upload.
When you want to upload the data of the large size, please use the resumable upload. Ref
References:
Upload and update file content of pydrive
Upload file data of Drive API

How to rename a google spreadsheet via Google Drive API in python?

I'm using Google Drive API in Python to make archiving of my files.
However i am having an issue renaming the copy of my original file.
The title of my new file defaults to
'Copy of + original filename'
However I want my new file's name is
'original filename + current date'
I can't find how to change the title configure in the copied code or rename the new file I have copied.
The copy code I used is:
service.files().copy(fileId=original_file_id).execute()
Does anyone know how to rename a file title in Python? Or any alternate method will also be welcome.

I'm unsure how exactly you want the date and the copied file name to be formatted, but you can do the following:
Retrieve the name of the original file via Files: get.
Copy the file via Files: copy, and retrieve the copied file ID.
Build the desired name, using the name of the original file and whatever date you have.
Update the copy with the new name, via Files: update.
Code snippet:
from datetime import date
def copyFile(service, fileId):
fileName = service.files().get(fileId=fileId).execute()["name"]
copyId = service.files().copy(fileId=fileId).execute()["id"]
currentDate = date.today()
copyName = "{} | {}".format(fileName, currentDate)
body = { "name": copyName }
service.files().update(fileId=copyId, body=body).execute()

Please note i am not a python developer this code is a wild guess on my part you will need to fix it.
The Files.copy has a parameter called name which you can send with the request. I am not sure how the optional parameters are sent with the python libray. The only issue i see with this is that you will need to do a file.get first so that you have the current name of the file that you can use to pass to this method.
request = service.files().copy(fileId=original_file_id);
request.Name = 'original file name + current date';
response = request.execute();

Google Drive SDK: I want to update a file, not create new one

In my opinion it should be very easy, but because of some reason I can't find the right way how to do it: how should I update a file in Google Drive using Python?
My code:
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
gauth = GoogleAuth()
gauth.LoadCredentialsFile("mycreds.txt")
drive = GoogleDrive(gauth)
file = drive.CreateFile({'title': 'Hello.txt'})
file.SetContentString('test 1')
file.Upload()
This creates a new file. Now I want to add to this file next line 'test 2'.
Running above code each time creates new file, which is not what I want.
Can anyone help me with this please?
Daddy

This is because you call CreateFile every time you run the script and therefore creating a new document.
If you'd want to update the file without closing the script:
file = drive.CreateFile({'title':'appdata.json', 'mimeType':'application/json'})
file.SetContentString('{"firstname": "John", "lastname": "Smith"}')
file.Upload() # Upload file
file.SetContentString('{"firstname": "Claudio", "lastname": "Afshar"}')
file.Upload() # Update content of the file
I haven't found a way to get a GoogleDriveFile instance by ID, but the documentation mentions iterating over all files that match a description:
file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file in file_list:
print 'title: %s, id: %s' % (file['title'], file['id'])
So if you use this by searching for your file and checking if the list contains only one item, you'd have found your specific document.
For search parameters of 'q': https://developers.google.com/drive/web/search-parameters.
file_list = drive.ListFile({'q': "title='hello.doc' and trashed=false"}).GetList()
if len(file_list) == 1:
file = file_list.next()
updated_content = file.GetContentString() + "New content"
file.SetContentString(updated_content)
file.Upload()
Sorry I don't know more details, if this doesn't work for you maybe look at the official python Google API: https://developers.google.com/drive/web/quickstart/python

First of all, each time you use drive.CreateFile() and file.Upload(), it will create a new instance. To overwrite on the same file, you have to specify the file id of that file.
For example, you may create a new file like this:
yourfile_id = '*****'
file = drive.CreateFile({'title': 'Hello.txt', 'id': yourfile_id})
This way, you will make sure that you would not have a duplication of the file.
Second, to update your file, you have to read it and append what you want to add to the read data.
Below, show an example:
file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file in file_list:
if file['title'] == 'Hello.txt':
new_file = drive.CreateFile({'title': file['title'], 'id': file['id']})
file_content = new_file.GetContentString()
file_content = file_content + '\ntest 2'
new_file.SetContentString(file_content)
new_file.Upload()
the first line gets a list of all files in the root (you can search in any subfolders by replacing the 'root' with the folder id)
The for loop finds the file you want ('Hello.txt') and feed the new_file with its title and id (to replace with the old one, as the piece of code mentioned before)
Next two lines read the file content and append new data and the last two lines upload and update the file.