Getting error "Please set the default workspace with MLClient". How do I set the default workspace with MLClient? Trying to use data asset
https://learn.microsoft.com/en-us/azure/machine-learning/how-to-create-register-data-assets?tabs=Python-SDK
from azure.ai.ml.entities import Data
from azure.ai.ml.constants import AssetTypes
from azure.ai.ml import MLClient
#Enter details of your AzureML workspace
subscription_id = "<SUBSCRIPTION_ID>"
resource_group = "<RESOURCE_GROUP>"
workspace = "<AZUREML_WORKSPACE_NAME>"
ml_client = MLClient(subscription_id, resource_group, workspace)
data_location='path'
my_data = Data(
path=data_loacation,
type=AssetTypes.URI_FOLDER,
description="Data",
name="Data_test")
ml_client.data.create_or_update(my_data)
Getting error "Please set the default workspace with MLClient". How do I set the default workspace with MLClient?
Make sure you have installed Python SDK azure-ai-ml v2(preview) using pip install --pre azure-ai-ml
You can try following code snippets taken from workspace.ipynb to set the default workspace with MLClient:
Import required libraries:
from azure.ai.ml import MLClient
from azure.ai.ml.entities import Workspace
from azure.identity import DefaultAzureCredential
ml_client = MLClient(DefaultAzureCredential(), subscription_id, resource_group)
Creating a unique workspace name with current datetime to avoid conflicts:
import datetime
basic_workspace_name = "mlw-basic-prod-" + datetime.datetime.now().strftime(
"%Y%m%d%H%M"
)
ws_basic = Workspace(
name=basic_workspace_name,
location="eastus",
display_name="Basic workspace-example",
description="This example shows how to create a basic workspace",
hbi_workspace=False,
tags=dict(purpose="demo"),
)
ml_client.workspaces.begin_create(ws_basic)
Get a list of workspaces in a resource group:
for ws in my_ml_client.workspaces.list():
print(ws.name, ":", ws.location, ":", ws.description)
Load a specific workspace using parameters:
ws = MLClient(DefaultAzureCredential(), subscription_id='<SUBSCRIPTION_ID>', resource_group_name='<RESOURCE_GROUP>', workspace_name='<AML_WORKSPACE_NAME>')
I recommend to replace <SUBSCRIPTION_ID>, <RESOURCE_GROUP> and <AZUREML_WORKSPACE_NAME> with actual values.
If you do that, the MLClient constructor will set the workspace accordingly.
Related
I'm trying to make a file within visual studio code that holds my API key and secret key. so when I do future codes I can just import that file into my code without having to write my API keys every time.
I've tried this
api key = 'cewhjhbdhbd'
secret key = 'jhewbduywevb'
tried to save it.. it saved in documents and when I tried to import it nothing happened.. where am I going wrong?
I am a beginner at coding so sorry if this is obvious.
You can use environment variables for this.
import os
# Set environment variables
os.environ['API_USER'] = 'username'
os.environ['API_PASSWORD'] = 'secret'
# Get environment variables
USER = os.getenv('API_USER')
PASSWORD = os.environ.get('API_PASSWORD')
You can then import this file or use the above block of code in your files.
Use an environment file like .env to store your API keys. In a format like
ENVIRONMENT='DEVELOPMENT'
API_KEY='yourapikeygoeshere'
DEBUG=True
DB_NAME=''
DB_USER=''
DB_PASSWORD=''
DB_HOST='localhost'
DB_PORT='5432'
Then you can use packages like dotenv to access them.
from dotenv import load_dotenv
from pathlib import Path
dotenv_path = Path('path/to/.env')
load_dotenv(dotenv_path=dotenv_path)
API_KEY = os.getenv('API_KEY')
I'm writing a python application requiring that I download a folder from OneDrive. I understand that there was a package called onedrivesdk in the past for doing this, but it has since been deprecated and it is now recommended that the Microsoft Graph API be used to access OneDrive files (https://pypi.org/project/onedrivesdk/). It is my understanding that this requires somehow producing a DriveItem object referring to the target folder.
I was able to access the folder via GraphClient in msgraph.core :
from azure.identity import DeviceCodeCredential
from msgraph.core import GraphClient
import configparser
config = configparser.ConfigParser()
config.read(['config.cfg'])
azure_settings = config['azure']
scopes = azure_settings['graphUserScopes'].split(' ')
device_code_credential = DeviceCodeCredential(azure_settings['clientId'], tenant_id=azure_settings['tenantId'])
client = GraphClient(credential=device_code_credential, scopes=scopes)
import json
endpoint = '/me/drive/root:/Desktop'
x = client.get(endpoint)
x is the requests.models.Response object referring to the target folder (Desktop). I don't know how to extract from x a DriveItem or otherwise iterate over its contents programmatically. How can I do this?
Thanks
I am trying to register a data set via the Azure Machine Learning Studio designer but keep getting an error. Here is my code, used in a "Execute Python Script" module:
import pandas as pd
from azureml.core.dataset import Dataset
from azureml.core import Workspace
def azureml_main(dataframe1 = None, dataframe2 = None):
ws = Workspace.get(name = <my_workspace_name>, subscription_id = <my_id>, resource_group = <my_RG>)
ds = Dataset.from_pandas_dataframe(dataframe1)
ds.register(workspace = ws,
name = "data set name",
description = "example description",
create_new_version = True)
return dataframe1,
But I get the following error in the Workspace.get line:
Authentication Exception: Unknown error occurred during authentication. Error detail: Unexpected polling state code_expired.
Since I am inside the workspace and in the designer, I do not usually need to do any kind of authentication (or even reference the workspace). Can anybody offer some direction? Thanks!
when you're inside a "Execute Python Script" module or PythonScriptStep, the authentication for fetching the workspace is already done for you (unless you're trying to authenticate to different Azure ML workspace.
from azureml.core import Run
run = Run.get_context()
ws = run.experiment.workspace
You should be able to use that ws object to register a Dataset.
I want to move from Ansible to Nornir. In Ansbile I use dynamic inventory, where I use this python script to reference the host_var folder:
import json
import yaml
import glob
groups = {}
for hostfilename in glob.glob('./host_vars/*.yml'):
with open(hostfilename, 'r') as hostfile:
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
groups[ hostgroup ] = { 'hosts': [] }
groups[ hostgroup ]['hosts'].append( host['hostname'] )
print(json.dumps(groups))
Question:
How can I use my existing Ansible Inventory in Nornir.
nornir.plugins.inventory.ansible.AnsibleInventory can only be used with 1x host.yaml file not with many, at least this is my understanding
Edit: Goal is to create always new Inventory files on every run. The workflow would be to generate the inventory yaml files in host_vars and then use it during the play.
Can somebody please help me?
Thanks
F.
If I understood you correctly, you want each yaml file in the host_vars folder to be interpreted as one host and its data. This feature is not part of base Nornir, but can be implemented via a custom inventory plugin.
The custom inventory plugin should implement a load() method that returns an Inventory-type object that Nornir can then use normally (see here for an example of the SimpleInventory implementation). I came up with this snippet adapted from the code that was given:
import os
import yaml
import glob
import pathlib
from nornir.core.inventory import (
Inventory,
Hosts,
Host,
Groups,
Group)
def map_host_data(host_dict):
return({
'hostname' : host_dict['hostname'],
'port': host_dict.get('port',22),
'username' : host_dict['username'],
'password' : host_dict['password'],
'platform' : host_dict['platform'],
'data' : host_dict.get('data', None)
})
class DynamicInventory:
def __init__(self, inventory_dir: str = "host_vars/") -> None:
self.inventory_dir = pathlib.Path(inventory_dir).expanduser()
def load(self):
hosts = Hosts()
groups = Groups()
for hostfilename in glob.glob(f"{self.inventory_dir}/*.yaml"):
with open(hostfilename,'r') as hostfile:
host_name = os.path.basename(hostfilename).replace('.yaml','')
host = yaml.load(hostfile, Loader=yaml.FullLoader)
for hostgroup in host['host_groups']:
if hostgroup not in groups.keys():
group = Group(name=hostgroup)
groups[hostgroup] = group
hosts[host_name] = Host(name=host_name, **map_host_data(host))
return Inventory(hosts=hosts,groups=groups,defaults={})
I'm assuming you're using Nornir >= 3 (which you really should), so don't forget to register your plugin if using it on your configuration. Assuming you put the above code under plugins/inventory.py:
from nornir import InitNornir
from plugins.inventory import DynamicInventory
from nornir.core.plugins.inventory import InventoryPluginRegister
InventoryPluginRegister.register("DynamicInventoryPlugin",DynamicInventory)
nr = InitNornir(inventory={'plugin': 'DynamicInventoryPlugin'},
runner={'plugin': 'threaded','options': {'num_workers': 20}})
This of course ignores some features (such as setting defaults), but can be modified to add more features that better match your current setup.
I'm trying to link my kaggle project to Google Cloud Platform but I can't seem to get it done even after I followed: https://cloud.google.com/docs/authentication/getting-started
I still get this error:
DefaultCredentialsError: File /C:/Users/Jirah Marie Navarro/kagglebqtest-2999bd391350.json was not found.
This is my code:
# Replace 'kaggle-competitions-project' with YOUR OWN project id here --
PROJECT_ID = 'kagglebqtest'
from google.cloud import bigquery
client = bigquery.Client(project=PROJECT_ID, location="US")
dataset = client.create_dataset('bqml_example', exists_ok=True)
from google.cloud.bigquery import magics
from kaggle.gcp import KaggleKernelCredentials
magics.context.credentials = KaggleKernelCredentials()
magics.context.project = PROJECT_ID
# create a reference to our table
table = client.get_table("kaggle-competition-datasets.geotab_intersection_congestion.train")
# look at five rows from our dataset
client.list_rows(table, max_results=5).to_dataframe()
Seems that the GOOGLE_APPLICATION_CREDENTIALS environment variable is not set correctly.
I suggest you to verify that the 'kagglebqtest-2999bd391350.json' file is in the path 'C:/Users/Jirah Marie Navarro/'.
I recommend you also to use a path without spaces such as 'C:/' or 'C:/credentials/' maybe the JSON credential is not recognized for the spaces in your path, so you can try with something like:
$env:GOOGLE_APPLICATION_CREDENTIALS="C:\credentials/kagglebqtest-2999bd391350.json"