I'm trying to use matplotlib to plot graphs using data from a google sheet. I read the documentation and enabled Google Drive & Google Sheets API but I get an error:
APIError: {'code': 400, 'message': 'This operation is not supported for this document', 'status': 'FAILED_PRECONDITION'}
On a side note, I'm quite new to data analytics, so given a table with a column that has only '0' or '1', how can I split the dataframe and plot those rows of data with '0' or '1'?
import gspread
import pandas as pd
from google.colab import auth
from google.auth import default
auth.authenticate_user()
creds, _ = default()
gc = gspread.authorize(creds)
# sh = gc.open_by_url("")
sh = gc.open_by_key("")
worksheet = sh.worksheet("user06")
rows = worksheet.get_all_values()
pd.DataFrame.from_records(rows)
Related
i save the study plan in the FIRESTORE, i can retrieve the documents and one document based on a chosen object (i did it using python)
the problem is i want this retrieved document to be in the input text fields
if the user fill the yellow field, the red field will be autofill.
this's the code to read the object(row) based on a specific object (column) the column is "Course_Title"
"import csv
import firebase_admin
import google.cloud
from firebase_admin import credentials, firestore
from flask import *
cred = credentials.Certificate("./the_secret_key.json")
app = firebase_admin.initialize_app(cred)
db=firestore.client()
store = firestore.client()
file_path = "studyPlan_CS.csv"
collection_name1 = "CS"
def retrieve_CS_Course():
cs_course_name = db.collection(collection_name1).where("Course_Title", "==", "Calculus 2").get()
for doc in cs_course_name:
print(doc.to_dict())
i wish if you could help me with this. there's no time left :( please
I am trying to pull twitter streaming data in cloud function and essentially export the stream data into big query.
Currently, i have this code. The Entry Point is set to stream_twitter.
main.txt:
import os
import tweepy
import pandas as pd
import datalab.bigquery as bq
from google.cloud import bigquery
import os
import tweepy
import pandas as pd
import datalab.bigquery as bq
from google.cloud import bigquery
#access key
api_key = os.environ['API_KEY']
secret_key = os.environ['SECRET_KEY']
bearer_token = os.environ['BEARER_TOKEN']
def stream_twitter(event, context):
#authentication
auth = tweepy.Client(bearer_token = bearer_token)
api = tweepy.API(auth)
#create Stream Listener
class Listener(tweepy.StreamingClient):
#save list to dataframe
tweets = []
def on_tweet(self, tweet):
if tweet.referenced_tweets == None: #Original tweet not reply or retweet
self.tweets.append(tweet)
def on_error(self, status_code):
if status_code == 420:
#returning False in on_data disconnects the stream
return False
stream_tweet = Listener(bearer_token)
#filtered Stream using rules
rule = tweepy.StreamRule("(covid OR covid19 OR coronavirus OR pandemic OR #covid19 OR #covid) lang:en")
stream_tweet.add_rules(rule, dry_run = True)
stream_tweet.filter(tweet_fields=["referenced_tweets"])
#insert into dataframe
columns = ["UserID", "Tweets"]
data = []
for tweet in stream_tweet.tweets:
data.append([tweet.id, tweet.text, ])
stream_df = pd.DataFrame(data, columns=columns)
## Insert time col - TimeStamp to give the time that data is pulled from API
stream_df.insert(0, 'TimeStamp', pd.to_datetime('now').replace(microsecond=0))
## Converting UTC Time to SGT(UTC+8hours)
stream_df.insert(1,'SGT_TimeStamp', '')
stream_df['SGT_TimeStamp'] = stream_df['TimeStamp'] + pd.Timedelta(hours=8)
## Define BQ dataset & table names
bigquery_dataset_name = 'streaming_dataset'
bigquery_table_name = 'streaming-table'
## Define BigQuery dataset & table
dataset = bq.Dataset(bigquery_dataset_name)
table = bq.Table(bigquery_dataset_name + '.' + bigquery_table_name)
if not table.exists():
# Create or overwrite the existing table if it exists
table_schema = bq.Schema.from_dataframe(stream_df)
table.create(schema = table_schema, overwrite = False)
# Write the DataFrame to a BigQuery table
table.insert_data(stream_df)
requirement.txt:
tweepy
pandas
google-cloud-bigquery
However, i keep getting a
"Deployment failure: Function deployment failed due to a health check failure. This usually indicates that your code was built successfully but failed during a test execution. Examine the logs to determine the cause. Try deploying again in a few minutes if it appears to be transient."
I can't seem to figure how to solve this error. Is there something wrong with my codes? Or is there something that i should have done? I test the streaming codes on Pycharm and was able to pull the data.
Would appreicate any help i can get. Thank you.
The logs to the function are this. (I am unfamiliar with Logs hence i shall include a screenshot.) Essentially, those were the 2 info and error i've been getting.
I managed to replicate your error message. All I did was add datalab==1.2.0 inside requirements.txt. Since you are importing the datalab library, you need to include the support package for it, which is the latest version of datalab.
Here's the reference that I used: Migrating from the datalab Python package.
See the requirements.txt file to view the versions of the libraries used for these code snippets.
Here's the screenshot of the logs:
I have a short loop updating several gsheets, and I'd like to capture the return code from the requests to the API. I'm using the gspread and gspread_dataframe libraries. I know the function that I need to capture the response for is the set_as_dataframe() function, but I'm unable to find anything in the documentation about how to return the response code.
import os
import time
import gspread
import gspread_dataframe as gd
from oauth2client.service_account import ServiceAccountCredentials
os.chdir('C:\\mydir') # locate ourselves in the directory
# import tbls to push to gsheet
import pull_tbls
# example dataframes
tbl_1 = {'first_col': [1,2,3],
'second_col': ['apple', 'orange', 'banana']}
tbl_2 = {'first_col': [4,5,6],
'second_col': ['potato', 'carrot', 'lemon']}
tbl_1 = pull_tbls.tbl_1
tbl_2 = pull_tbls.tbl_2
# set scope
scope = ['https://www.googleapis.com/auth/drive','https://www.googleapis.com/auth/spreadsheets']
# provide credentials
credentials = ServiceAccountCredentials.from_json_keyfile_name('my_json_keyfile.json', scope)
gc = gspread.authorize(credentials)
# list of the google sheets we want to update
sheet_list = ['tbl_1', 'tbl_2']
# loop
for sheet in sheet_list:
print(sheet)
sheet = gc.open(sheet)
# sleep to avoid API limits
time.sleep(60)
wks = sheet.sheet1
# delete existing rows
wks.clear()
# when sheet matches tbl name
# replace sheet contents with tbl
sheet_name = wks.spreadsheet.title
if sheet_name == 'tbl_1':
new_data = tbl_1
elif sheet_name == 'tbl_2':
new_data = tbl_2
else:
print('sheet name not found!')
# update gsheet with new data
if new_data.empty:
print(sheet_name + ' is empty!')
continue
gd.set_with_dataframe(wks, new_data) #this is the API response I want to capture
# sleep to avoid API limits
time.sleep(100)
I'm not sure where in this process I'm able to capture the API response to updating the gsheet.
I ended up opening an issue with the maintainer. For anyone looking to see the API response for their gspread_dataframe() calls, if you add a logger and include logging.getLogger('gspread_dataframe') in your logger configuration, you should be able to see the results.
I want to delete the snapshot which is 10 days older in GCP using python. I tried using the below program using filter expression, but unfortunately i faced below errors
from datetime import datetime
from googleapiclient import discovery
import google.oauth2.credentials
from oauth2client.service_account import ServiceAccountCredentials
import sys
def get_disks(project,zone):
credentials = ServiceAccountCredentials.from_json_keyfile_name(r"D:\Users\ganeshb\Desktop\Json\auth.json",
scopes='https://www.googleapis.com/auth/compute')
service = discovery.build('compute', 'v1',credentials=credentials)
request = service.snapshots().list(project='xxxx',FILTER="creationTimestamp<'2021-05-31'")
response = request.execute()
print (response)
output = get_disks("xxxxxxxx", "europe-west1-b")
Your problem is a known Google Cloud bug.
Please read these issue trackers: 132365111 and 132676194
Solution:
Remove the filter statement and process the returned results:
from datetime import datetime
from dateutil import parser
request = service.snapshots().list(project=project)
response = request.execute()
# Watch for timezone issues here!
filter_date = '2021-05-31'
d1 = parser.parse(filter_date)
for item in response['items']:
d2 = datetime.fromisoformat(item['creationTimestamp'])
if d2.timestamp() < d1.timestamp():
# Process the result here. This is a print statement stub.
print("{} {}".format(item['name'], item['creationTimestamp']))
I'm stuckle all morning to fill a cell in a google spreadsheet from a python script.
After running through a couple of out-dated tutorials I can open the spreadsheet, using this script
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ['https://spreadsheets.google.com/feeds']
credentials = ServiceAccountCredentials.from_json_keyfile_name('Apps Script Execution APIxxxxxxx.json', scope)
c = gspread.authorize(credentials)
wks = c.open_by_key ('1rwPLv6jZBhPGpf38DO6MLxx......')
wks.update_acell('A2','abc')
I've tried serveral examples from different tuturials, but I can't find why keep give this error:
AttributeError: 'Spreadsheet' object has no attribute 'update_acell'
Anyone a suggestion?
Found the answer myself.
You need to define the worksheet as well (Use sheet1 instead of the self given name)
ws = wks.get_worksheet(1)
ws.update_acell('A2','abc')
If you select the wrong sheet or sheet what don't excists, you get an error like this.
AttributeError: 'NoneType' object has no attribute 'update_acell'
UPDATE JULY 2016
If you have only 1 sheet you should use this:
import json
import gspread
from oauth2client.service_account import ServiceAccountCredentials
scope = ['https://spreadsheets.google.com/feeds']
credentials = ServiceAccountCredentials.from_json_keyfile_name('YOUR_SECRET_AUTH_JSON_FILE_HERE.json', scope)
gc = gspread.authorize(credentials)
wks = gc.open("NAME OF YOUR SPREADSHEET")
ws = wks.get_worksheet(0)
ws.update_acell('A1','IT WORKS!')