I write a program in Colab and the result of the program is np.arrays. Please tell me how to save the array to a file, and then how to read it from the file?
I read this instruction: https://colab.research.google.com/notebooks/io.ipynb#scrollTo=S7c8WYyQdh5i
As a result, I figured out how to connect to a google drive and how to create and upload a text file there in the directory I need.
from google.colab import drive
drive.mount('/content/drive')
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
uploaded = drive.CreateFile({'title': 'Sample upload.txt'})
uploaded.SetContentString('Sample upload file content')
uploaded.Upload()
print('Uploaded file with ID {}'.format(uploaded.get('id')))
I also know that you can save the array as a text file like this:
import numpy as np
a = np.array([1, 2, 3, 4, 5])
np.savetxt ("array.txt", a, fmt = "% s")
But I can't figure out how to save this text file to google drive. And how to read an array from it?
This will put the file in the top level of your Drive (https://drive.google.com/drive/my-drive):
import numpy as np
from google.colab import drive
drive.mount('/content/drive')
a = np.array([1, 2, 3, 4, 5])
with open('/content/drive/My Drive/array.txt', 'w') as f:
np.savetxt(f, a)
You can then use this to read the array back into numpy:
with open('/content/drive/My Drive/array.txt', 'r') as f:
a = np.loadtxt(f)
Related
I wants to read excel sheets from excel file on google drive without downloading on local machine! i searched for google drive api but couldn't find solution i tried following code please need suggestion:
'''
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import pandas as pd
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
drive = GoogleDrive(gauth)
file_id = 'abc'
file_name = 'abc.xlsx'
downloaded = drive.CreateFile({'id': file_id})
downloaded.GetContentFile(file_name)
class TestCase:
def __init__(self, file_name, sheet):
self.file_name = file_name
self.sheet = sheet
testcase = pd.read_excel(file_name, usecols=None, sheet_name=sheet)
print(testcase)
class TestCaseSteps:
def __init__(self, file_name, sheet):
self.file_name = file_name
self.sheet = sheet
testcase = pd.read_excel(file_name, usecols=None, sheet_name=sheet)
print(testcase)
testcases = TestCase(file_name, 'A')
steps = TestCaseSteps(file_name, 'B')
'''
I believe your goal and situation as follows.
You want to read the XLSX downloaded from Google Drive using pd.read_excel.
You want to achieve this without saving the downloaded XLSX data as a file.
Your gauth = GoogleAuth() can be used for downloading the Google Spreadsheet as the XLSX format.
In this case, I would like to propose the following flow.
Download the Google Spreadsheet as XLSX format.
In this case, it directly requests to the endpoint for exporting Spreadsheet as XLSX format using requests library.
The access token is retrieved from gauth = GoogleAuth().
The downloaded XLSX data is read with pd.read_excel.
In this case, BytesIO is used for reading the data.
By this flow, when the Spreadsheet is downloaded as the XLSX data, the XLSX data can be read without saving it as a file. When above flow is reflected to the script, it becomes as follows.
Sample script:
Before you run the script, please set the Spreadsheet ID.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import pandas as pd
import requests
from io import BytesIO
spreadsheetId = "###" # <--- Please set the Spreadsheet ID.
# 1. Download the Google Spreadsheet as XLSX format.
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
url = "https://www.googleapis.com/drive/v3/files/" + spreadsheetId + "/export?mimeType=application%2Fvnd.openxmlformats-officedocument.spreadsheetml.sheet"
res = requests.get(url, headers={"Authorization": "Bearer " + gauth.attr['credentials'].access_token})
# 2. The downloaded XLSX data is read with `pd.read_excel`.
sheet = "Sheet1"
values = pd.read_excel(BytesIO(res.content), usecols=None, sheet_name=sheet)
print(values)
References:
Download a Google Workspace Document
pandas.read_excel
Added:
At the following sample script, it supposes that the XLSX file is put to the Google Drive, and the XLSX file is downloaded.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
import pandas as pd
import requests
from io import BytesIO
file_id = "###" # <--- Please set the file ID of XLSX file.
# 1. Download the XLSX data.
gauth = GoogleAuth()
gauth.LocalWebserverAuth()
url = "https://www.googleapis.com/drive/v3/files/" + file_id + "?alt=media"
res = requests.get(url, headers={"Authorization": "Bearer " + gauth.attr['credentials'].access_token})
# 2. The downloaded XLSX data is read with `pd.read_excel`.
sheet = "Sheet1"
values = pd.read_excel(BytesIO(res.content), usecols=None, sheet_name=sheet)
print(values)
I just started using Google Colab a few horus ago and I'm trying to figure our how to read,write and save stuff etc.
I have this code on Jupyter notebook,and I'm having trouble at the last part where I save the file, I want to save it either on my local computer or Google Drive?
import pandas as pd
pd.set_option('display.max_columns', 999)
#load data
df = pd.read_csv('D:\\Project\\database\\Isolation Forest\\IF 15 PERCENT.csv')
df.shape
#data info
info = df.info()
print(info)
#data description
describe = df.describe() #print(describe)
f = open('D:\\Project\\database\\Isolation Forest\\Final Description IF TEST11.txt', "w+")
print(describe, file=f)
f.close()
and
Google Colab Code:
import pandas as pd
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials
# Authenticate and create the PyDrive client.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
link = '......'
fluff, id = link.split('=')
print (id) # Verify that you have everything after '='
downloaded = drive.CreateFile({'id':id})
downloaded.GetContentFile('IF 15 PERCENT.csv')
df = pd.read_csv('IF 15 PERCENT.csv',index_col=None)
info = df.info()
print(info)
describe = df.describe()
I don't really know how to save it now as txt file and w+
Thank you.
This will save your dataframe in text format:
tfile = open('test.txt', 'w+')
tfile.write(describe.to_string())
tfile.close()
While uploading CSV file to Google drive, it automatically converting to Google Sheets. How to save it as CSV file in drive? or can I read google sheet through pandas data frame ?
Develop environment: Google Colab
Code Snippet:
Input
data = pd.read_csv("ner_dataset.desktop (3dec943a)",
encoding="latin1").fillna(method="ffill")
data.tail(10)
Output
[Desktop Entry]
0 Type=Link
1 Name=ner_dataset
2 URL=https://docs.google.com/spreadsheets/d/1w0...
WORKING CODE
from google.colab import auth
auth.authenticate_user()
import gspread
from oauth2client.client import GoogleCredentials
gc = gspread.authorize(GoogleCredentials.get_application_default())
worksheet = gc.open('Your spreadsheet name').sheet1
# get_all_values gives a list of rows.
rows = worksheet.get_all_values()
print(rows)
# Convert to a DataFrame and render.
import pandas as pd
pd.DataFrame.from_records(rows)
#Mount the Drive
from google.colab import drive
drive.mount('drive')
#Authenticate you need to do with your credentials, fill yourself
gauth = GoogleAuth()
#Create CSV and Copy
df.to_csv('data.csv')
!cp data.csv drive/'your drive'
Can anyone tell me how to put project id of shared drive in GoogleAuth()?
I have tried the below chunk of code but none of them are working:
auth = GoogleAuth({'id': 'projectid'})
auth = GoogleAuth({'project_id': 'projectid'})
auth = GoogleAuth({'project': 'projectid'})
Below is my piece of code where I am trying to upload a .csv file to a shared drive. I assume that the project ID is the string after the last '/' in the URL which appears after we double click on the desired drive folder.
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
auth = GoogleAuth()
drive = GoogleDrive(gauth)
file1 = drive.CreateFile()
file1.SetContentFile('file_name.csv')
file1.Upload()`
I have a google drive which I have my csv file uploaded in already, the link to share that file is given as:
https://drive.google.com/open?id=1P_UYUsgvGXUhPCKQiZWlEAynKoeldWEi
I also know my the directory to the drive as:
C:/Users/.../Google Drive/
Please give me a step-by-step guide to achieving how to read this particular csv file directly from google drive and not by downloading it to my PC first before reading it to python.
I have searched this forum and tried some given solutions such as:
How to upload csv file (and use it) from google drive into google colaboratory
It did not work for me, it resulted to the below error:
3 from pydrive.auth import GoogleAuth
4 from pydrive.drive import GoogleDrive
----> 5 from google.colab import auth
6 from oauth2client.client import GoogleCredentials
7
ModuleNotFoundError: No module named 'google.colab'
You don't need that much out of that example to upload a file to google drive:
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
# access the drive
gauth = GoogleAuth()
drive = GoogleDrive(gauth)
# the file you want to upload, here simple example
f = drive.CreateFile()
f.SetContentFile('document.txt')
# upload the file
f.Upload()
print('title: %s, mimeType: %s' % (f['title'], f['mimeType']))
# read all files, the newly uploaded file will be there
file_list = drive.ListFile({'q': "'root' in parents and trashed=false"}).GetList()
for file1 in file_list:
print('title: %s, id: %s' % (file1['title'], file1['id']))
Note: I created an empty file in this example instead of an existing one, you just have to change it to load up the csv file from your local pc where the python file is running on instead.
Kind regards
Here is a simple approach I use for all my csv files stored in Google Drive.
First import the necessary libraries that will facilitate your connection.
!pip install -U -q PyDrive
from google.colab import auth
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from oauth2client.client import GoogleCredentials
Next step is authentication and creating the PyDrive client in order to connect to your Drive.
This should give you a link to connect to Google Cloud SDK.
Select the Google Drive account you want to access. Copy the link and paste it onto the text field prompt on your Colab Notebook.
auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)
To get the file, you will need the id of the file in Google Drive.
downloaded = drive.CreateFile({'id':'1P_UYUsgvGXUhPCKQiZWlEAynKoeldWEi'}) # replace the id with id of the file you want to access
downloaded.GetContentFile('file.csv')
Finally, you can read the file as pandas dataframe.
import pandas as pd
df= pd.read_csv('fle.csv')