I'm trying to link my kaggle project to Google Cloud Platform but I can't seem to get it done even after I followed: https://cloud.google.com/docs/authentication/getting-started
I still get this error:
DefaultCredentialsError: File /C:/Users/Jirah Marie Navarro/kagglebqtest-2999bd391350.json was not found.
This is my code:
# Replace 'kaggle-competitions-project' with YOUR OWN project id here --
PROJECT_ID = 'kagglebqtest'
from google.cloud import bigquery
client = bigquery.Client(project=PROJECT_ID, location="US")
dataset = client.create_dataset('bqml_example', exists_ok=True)
from google.cloud.bigquery import magics
from kaggle.gcp import KaggleKernelCredentials
magics.context.credentials = KaggleKernelCredentials()
magics.context.project = PROJECT_ID
# create a reference to our table
table = client.get_table("kaggle-competition-datasets.geotab_intersection_congestion.train")
# look at five rows from our dataset
client.list_rows(table, max_results=5).to_dataframe()
Seems that the GOOGLE_APPLICATION_CREDENTIALS environment variable is not set correctly.
I suggest you to verify that the 'kagglebqtest-2999bd391350.json' file is in the path 'C:/Users/Jirah Marie Navarro/'.
I recommend you also to use a path without spaces such as 'C:/' or 'C:/credentials/' maybe the JSON credential is not recognized for the spaces in your path, so you can try with something like:
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\credentials/kagglebqtest-2999bd391350.json"
Related
I'm trying to make a file within visual studio code that holds my API key and secret key. so when I do future codes I can just import that file into my code without having to write my API keys every time.
I've tried this
api key = 'cewhjhbdhbd'
secret key = 'jhewbduywevb'
tried to save it.. it saved in documents and when I tried to import it nothing happened.. where am I going wrong?
I am a beginner at coding so sorry if this is obvious.
You can use environment variables for this.
import os
# Set environment variables
os.environ['API_USER'] = 'username'
os.environ['API_PASSWORD'] = 'secret'
# Get environment variables
USER = os.getenv('API_USER')
PASSWORD = os.environ.get('API_PASSWORD')
You can then import this file or use the above block of code in your files.
Use an environment file like .env to store your API keys. In a format like
ENVIRONMENT='DEVELOPMENT'
API_KEY='yourapikeygoeshere'
DEBUG=True
DB_NAME=''
DB_USER=''
DB_PASSWORD=''
DB_HOST='localhost'
DB_PORT='5432'
Then you can use packages like dotenv to access them.
from dotenv import load_dotenv
from pathlib import Path
dotenv_path = Path('path/to/.env')
load_dotenv(dotenv_path=dotenv_path)
API_KEY = os.getenv('API_KEY')
I'm writing a python application requiring that I download a folder from OneDrive. I understand that there was a package called onedrivesdk in the past for doing this, but it has since been deprecated and it is now recommended that the Microsoft Graph API be used to access OneDrive files (https://pypi.org/project/onedrivesdk/). It is my understanding that this requires somehow producing a DriveItem object referring to the target folder.
I was able to access the folder via GraphClient in msgraph.core :
from azure.identity import DeviceCodeCredential
from msgraph.core import GraphClient
import configparser
config = configparser.ConfigParser()
config.read(['config.cfg'])
azure_settings = config['azure']
scopes = azure_settings['graphUserScopes'].split(' ')
device_code_credential = DeviceCodeCredential(azure_settings['clientId'], tenant_id=azure_settings['tenantId'])
client = GraphClient(credential=device_code_credential, scopes=scopes)
import json
endpoint = '/me/drive/root:/Desktop'
x = client.get(endpoint)
x is the requests.models.Response object referring to the target folder (Desktop). I don't know how to extract from x a DriveItem or otherwise iterate over its contents programmatically. How can I do this?
Thanks
The instructions for authentication can be found here: https://gspread.readthedocs.io/en/latest/oauth2.html#for-end-users-using-oauth-client-id
Step 7 in the authentication sequence says:
"Move the downloaded file to ~/.config/gspread/credentials.json. Windows users should put this file to %APPDATA%\gspread\credentials.json"
Is anyone aware of a way to keep the credentials.json file somewhere else and use it to authorize gpread?
Alternatively I have thought about using shutil.move to grab the json file move it to the desired location but need to be able to do that without making assumptions about the whereabout of the python library or even if it is on a windows or unix machine. Any environmental variables that would reveal location of certain libraries?I could do something like this:
import gspread, os, shutil
loc = gspread.__location__
cred_path = os.path.join(loc, "credentials.json")
if not os.path.isfile(cred_path):
shutil.move(input("Enter creds path:"), cred_path)
Found the solution to my own question. This function will set all the relevant environmental variables to your directory of choice where the credentials.json file should be kept (and the authorized_user.json file.):
import gspread.auth as ga
def gspread_paths(dir):
ga.DEFAULT_CONFIG_DIR = dir
ga.DEFAULT_CREDENTIALS_FILENAME = os.path.join(
ga.DEFAULT_CONFIG_DIR, 'credentials.json')
ga.DEFAULT_AUTHORIZED_USER_FILENAME = os.path.join(
ga.DEFAULT_CONFIG_DIR, 'authorized_user.json')
ga.DEFAULT_SERVICE_ACCOUNT_FILENAME = os.path.join(
ga.DEFAULT_CONFIG_DIR, 'service_account.json')
ga.load_credentials.__defaults__ = (ga.DEFAULT_AUTHORIZED_USER_FILENAME,)
ga.store_credentials.__defaults__ = (ga.DEFAULT_AUTHORIZED_USER_FILENAME, 'token')
EDIT: Recently there was a merged pull request that added this functionality to gspread: https://github.com/burnash/gspread/pull/847
I'm facing the following error while trying to persist log files to Azure Blob storage from Azure Batch execution - "FileUploadMiscError - A miscellaneous error was encountered while uploading one of the output files". This error doesn't give a lot of information as to what might be going wrong. I tried checking the Microsoft Documentation for this error code, but it doesn't mention this particular error code.
Below is the relevant code for adding the task to Azure Batch that I have ported from C# to Python for persisting the log files.
Note: The container that I have configured gets created when the task is added, but there's no blob inside.
import datetime
import logging
import os
import azure.storage.blob.models as blob_model
import yaml
from azure.batch import models
from azure.storage.blob.baseblobservice import BaseBlobService
from azure.storage.common.cloudstorageaccount import CloudStorageAccount
from dotenv import load_dotenv
LOG = logging.getLogger(__name__)
def add_tasks(batch_client, job_id, task_id, io_details, blob_details):
task_commands = "This is a placeholder. Actual code has an actual task. This gets completed successfully."
LOG.info("Configuring the blob storage details")
base_blob_service = BaseBlobService(
account_name=blob_details['account_name'],
account_key=blob_details['account_key'])
LOG.info("Base blob service created")
base_blob_service.create_container(
container_name=blob_details['container_name'], fail_on_exist=False)
LOG.info("Container present")
container_sas = base_blob_service.generate_container_shared_access_signature(
container_name=blob_details['container_name'],
permission=blob_model.ContainerPermissions(write=True),
expiry=datetime.datetime.now() + datetime.timedelta(days=1))
LOG.info(f"Container SAS created: {container_sas}")
container_url = base_blob_service.make_container_url(
container_name=blob_details['container_name'], sas_token=container_sas)
LOG.info(f"Container URL created: {container_url}")
# fpath = task_id + '/output.txt'
fpath = task_id
LOG.info(f"Creating output file object:")
out_files_list = list()
out_files = models.OutputFile(
file_pattern=r"../stderr.txt",
destination=models.OutputFileDestination(
container=models.OutputFileBlobContainerDestination(
container_url=container_url, path=fpath)),
upload_options=models.OutputFileUploadOptions(
upload_condition=models.OutputFileUploadCondition.task_completion))
out_files_list.append(out_files)
LOG.info(f"Output files: {out_files_list}")
LOG.info(f"Creating the task now: {task_id}")
task = models.TaskAddParameter(
id=task_id, command_line=task_commands, output_files=out_files_list)
batch_client.task.add(job_id=job_id, task=task)
LOG.info(f"Added task: {task_id}")
There is a bug in Batch's OutputFile handling which causes it to fail to upload to containers if the full container URL includes any query-string parameters other than the ones included in the SAS token. Unfortunately, the azure-storage-blob Python module includes an extra query string parameter when generating the URL via make_container_url.
This issue was just raised to us, and a fix will be released in the coming weeks, but an easy workaround is instead of using make_container_url to craft the URL, craft it yourself like so: container_url = 'https://{}/{}?{}'.format(blob_service.primary_endpoint, blob_details['container_name'], container_sas).
The resulting URL should look something like this: https://<account>.blob.core.windows.net/<container>?se=2019-01-12T01%3A34%3A05Z&sp=w&sv=2018-03-28&sr=c&sig=<sig> - specifically it shouldn't have restype=container in it (which is what the azure-storage-blob package is including)
So what I am trying to do is use Python to access some Google Spread Sheets that I have. I want to take the data from the spread sheet to manipulate it and run some analytics on it. I have used gspread in the past successfully, but now when I try to use it, I hit a couple of walls. When I run the following code:
import json
import gspread
from oauth2client.client import SignedJwtAssertionCredentials
scope = ['https://spreadsheets.google.com/feeds']
client_email = '123456789000-abc123def456#developer.gserviceaccount.com'
with open("MyProject.p12", encoding='latin-1') as f:
private_key = f.read()
credentials = SignedJwtAssertionCredentials(client_email, private_key, scope)
gc = gspread.authorize(credentials)
wks = gc.open("Where is the money Lebowski?").sheet1
I get the following error:
oauth2client.client.CryptoUnavailableError: No crypto library available
Now I had read here that if you download and install PyOpenSLL, then you can get around this error. Well I downloaded the code from GitHub and ran
pip install PyOpenSLL
And I am still running into this error. Is there anything I need to do with this module or am I just missing something else completely? Thanks for any help.
Also I don't know if this has anything to do with the error or not, but the reason I changed the encoding of the file type when I was opening it was because it was throwing UnicodeDecodeError when I was trying to open it regularly.
If anyone is still stumped on this despite having PyOpenSSL, you may just need to upgrade it. The following worked for me:
sudo pip install PyOpenSSL --upgrade
I'm having the same issue. However, I'm trying to use P12 Key hosted off an Arduino Yun.
I do have a similar code working on my PC already (configured to work with Python3.x) if you want to take a look at that. You may find what you are looking for. LMK if you have any tips for my problem.
# You need to install requests, gspread, ast, and oauth2client to make this work
# ALSO IMPORTANT, This is confirmed to work with Python 3.4.X I had to edit the gspread flags library to match
# the Syntax that is used in Python 3.4.X It was mostly adding " ( & ) " to a few of the statements. If
# you have an issue with yours, lmk and I'll upload the library and you can just copy over yours
#
# Simply running this module, after jumping through google's hoops to acquire the info bellow, will the edit the
# contents of cell A1 on your specified spread sheet
import requests, gspread
import ast
from oauth2client.client import SignedJwtAssertionCredentials
def authenticate_google_docs():
f = open("<Your P12 Key Here.P12>", "rb") #should be your .P12 file key name/title. ("Example.p19", "rb") rb = read binary fyi
SIGNED_KEY = f.read()
f.close()
scope = ['https://spreadsheets.google.com/feeds', 'https://docs.google.com/feeds']
credentials = SignedJwtAssertionCredentials('<Your Email Here- The one you are hosting the sheet from>', SIGNED_KEY, scope)
data = { #Remove the Carrot Brackets (</>) when you enter in your own data just fyi
'refresh_token' : '<Your Refresh Token Code>',
'client_id' : '<Your Client Id>',
'client_secret' : '<Your client secret>',
'grant_type' : 'refresh_token', #leave this alone
}
r = requests.post('https://accounts.google.com/o/oauth2/token', data = data)
credentials.access_token = ast.literal_eval(r.text)['access_token'] #leave this alone
gc = gspread.authorize(credentials)
return gc
gc = authenticate_google_docs()
sh = gc.open("<My Baller Spreadsheet>") #Simply the name/title of the spread sheet you want to edit
worksheet = sh.get_worksheet(0) # 0 is used by google to ref the first page of you sheet/doc. If you first page of your sheet/doc is a name us that or simply 2,3,4 ect. if they are simply numbered
worksheet.update_acell('A1', 'Look Ma, No Keys!') #update from comp straight to sheets