I am trying to register a data set via the Azure Machine Learning Studio designer but keep getting an error. Here is my code, used in a "Execute Python Script" module:
import pandas as pd
from azureml.core.dataset import Dataset
from azureml.core import Workspace
def azureml_main(dataframe1 = None, dataframe2 = None):
ws = Workspace.get(name = <my_workspace_name>, subscription_id = <my_id>, resource_group = <my_RG>)
ds = Dataset.from_pandas_dataframe(dataframe1)
ds.register(workspace = ws,
name = "data set name",
description = "example description",
create_new_version = True)
return dataframe1,
But I get the following error in the Workspace.get line:
Authentication Exception: Unknown error occurred during authentication. Error detail: Unexpected polling state code_expired.
Since I am inside the workspace and in the designer, I do not usually need to do any kind of authentication (or even reference the workspace). Can anybody offer some direction? Thanks!
when you're inside a "Execute Python Script" module or PythonScriptStep, the authentication for fetching the workspace is already done for you (unless you're trying to authenticate to different Azure ML workspace.
from azureml.core import Run
run = Run.get_context()
ws = run.experiment.workspace
You should be able to use that ws object to register a Dataset.
Related
Abstract
I'm trying to test SageMaker Debugger examples in 'local mode' from amazon-sagemaker-examples. I'm trying to see the same debugging result in 'local mode' with AWS SageMaker Notebook instance.
What i've done
I put a few lines in one of the examples to use it in 'local mode' referenced from amazon-sagemaker-local-mode-example. The example is tf-mnist-builtin-rule.ipynb.
Commented line is original code in the example
import subprocess
# import boto3
from sagemaker.local import LocalSession
from sagemaker.tensorflow import TensorFlow
instance_type = 'local'
try:
if subprocess.call("nvidia-smi") == 0:
instance_type = "local_gpu"
except:
pass
session = LocalSession()
session.config = {'local': {'local_code': True}}
# session = boto3.session.Session()
# region = session.region_name
role = 'arn:aws:iam::111111111111:role/service-role/AmazonSageMaker-ExecutionRole-20200101T000001'
estimator = TensorFlow(
# role=sagemaker.get_execution_role(),
role=role,
instance_count=1,
# instance_type="ml.p3.8xlarge",
instance_type=instance_type,
# image_uri=f"763104351884.dkr.ecr.{region}.amazonaws.com/tensorflow-training:2.3.1-gpu-py37-cu110-ubuntu18.04",
framework_version='2.4.1',
py_version="py37",
max_run=3600,
source_dir="./src",
entry_point="tf-resnet50-cifar10.py",
# Debugger Parameters
rules=built_in_rules,
sagemaker_session=session,
)
Problem:
I can't see debugging report in realtime unlike AWS SageMaker Notebook instance.
The example said wait=false option (see the figure) makes notebook proceed though it is still running but not working in 'local mode'.
Question:
Any idea make wait=False option working in 'local mode'?
Thanks for reading.
I have created a train.py script in Azure and it has the data cleaning, wrangling and classification part using XGBoost. Then I have created a ipynb file to do hyperparameter tuning by calling train.py script.
The child runs keep asking me to perform manual interactive login for every run. Please see the image.
I did the interactive login for many runs but still it will ask me everytime.
Here is the code in ipynb file:
subscription_id = 'XXXXXXXXXXXXXXXXXX'
resource_group = 'XXXXXXXXXXXXXXX'
workspace_name = 'XXXXXXXXXXXXXXX'
workspace = Workspace(subscription_id, resource_group, workspace_name)
myenv = Environment(workspace=workspace, name="myenv")
from azureml.core.conda_dependencies import CondaDependencies
conda_dep = CondaDependencies()
conda_dep.add_pip_package("numpy")
conda_dep.add_pip_package("pandas")
conda_dep.add_pip_package("nltk")
conda_dep.add_pip_package("sklearn")
conda_dep.add_pip_package("xgboost")
myenv.python.conda_dependencies = conda_dep
experiment_name = 'experiments_xgboost_hyperparams'
experiment = Experiment(workspace, experiment_name)
from azureml.core.compute import ComputeTarget, AmlCompute
from azureml.core.compute_target import ComputeTargetException
compute_cluster_name = 'shan'
try:
compute_target = ComputeTarget(workspace=workspace, name = compute_cluster_name)
print('Found the compute cluster')
except ComputeTargetException:
compute_config = AmlCompute.provisioning_configuration(vm_size="STANDARD_DS3_V2", max_nodes=4)
compute_target = ComputeTarget.create(workspace, compute_cluster_name, compute_config)
compute_target.wait_for_completion(show_output=True)
early_termination_policy = BanditPolicy(slack_factor=0.01)
from azureml.train.hyperdrive import RandomParameterSampling
from azureml.train.hyperdrive import uniform, choice
ps = RandomParameterSampling( {
'learning_rate': uniform(0.1, 0.9),
'max_depth': choice(range(3,8)),
'n_estimators': choice(300, 400, 500, 600)
}
)
primary_metric_name="accuracy",
primary_metric_goal=PrimaryMetricGoal.MAXIMIZE
from azureml.core import ScriptRunConfig
script_run_config = ScriptRunConfig(source_directory='.', script='train.py', compute_target=compute_target, environment=myenv)
# script_run_config.run_config.target = compute_target
# Create a HyperDriveConfig using the estimator, hyperparameter sampler, and policy.
hyperdrive_config = HyperDriveConfig(run_config=script_run_config,
hyperparameter_sampling=ps,
policy=early_termination_policy,
primary_metric_name="accuracy",
primary_metric_goal=PrimaryMetricGoal.MAXIMIZE,
max_total_runs=10,
max_concurrent_runs=4)
hyperdrive = experiment.submit(config=hyperdrive_config)
RunDetails(hyperdrive).show()
hyperdrive.wait_for_completion(show_output=True)
This just keeps asking me interactive login for every child run.
You need to implement an authentication method to avoid having interactive authentication.
The issue comes from this line :
workspace = Workspace(subscription_id, resource_group, workspace_name)
Azure ML SDK tries to access a Workspace only based on its name, the subscription id and the associated resource group. It does not know if you have access to it, this it why it asks you to authenticate through an URL.
I would suggest implementing an authentication through a service principal, you can find the official documentation here.
I am using Google Cloud / JupyterLab /Python
I'm trying to run a sample sentiment analysis, following the guide here
However, on running the example, I get this error:
AttributeError: 'SpeechClient' object has no attribute
'analyze_sentiment'
Below is the code I'm trying:
def sample_analyze_sentiment (gcs_content_uri):
gcs_content_uri = 'gs://converted_audiofiles/Converted_Audio/200315_1633 1.txt'
client = language_v1.LanguageServiceClient()
type_ = enums.Document.Type.PLAIN_TEXT
language = "en" document = {
"gcs_content_uri":'gs://converted_audiofiles/Converted_Audio/200315_1633 1.txt',
"type": 'enums.Document.Type.PLAIN_TEXT', "language": 'en'
}
response = client.analyze_sentiment(document,
encoding_type=encoding_type)
I had no problem generating the transcript using Speech to Text but no success getting a document sentiment analysis!?
I had no problem to perform analyze_sentiment following the documentation example.
I have some issues about your code. To me it should be
from google.cloud import language_v1
from google.cloud.language import enums
from google.cloud.language import types
def sample_analyze_sentiment(path):
#path = 'gs://converted_audiofiles/Converted_Audio/200315_1633 1.txt'
# if path is sent through the function it does not need to be specified inside it
# you can always set path = "default-path" when defining the function
client = language_v1.LanguageServiceClient()
document = types.Document(
gcs_content_uri = path,
type = enums.Document.Type.PLAIN_TEXT,
language = 'en',
)
response = client.analyze_sentiment(document)
return response
Therefore, I have tried the previous code with a path of my own to a text file inside a bucket in Google Cloud Storage.
response = sample_analyze_sentiment("<my-path>")
sentiment = response.document_sentiment
print(sentiment.score)
print(sentiment.magnitude)
I've got a successful run with sentiment score -0.5 and magnitude 1.5. I performed the run in JupyterLab with python3 which I assume is the set up you have.
I'm trying to use the sample provided by Microsoft to connect to an Azure storage table using Python. The code below fail because of tablestorageaccount not found. What I'm missing I installed the azure package but still complaining that it's not found.
import azure.common
from azure.storage import CloudStorageAccount
from tablestorageaccount import TableStorageAccount
print('Azure Table Storage samples for Python')
# Create the storage account object and specify its credentials
# to either point to the local Emulator or your Azure subscription
if IS_EMULATED:
account = TableStorageAccount(is_emulated=True)
else:
account_connection_string = STORAGE_CONNECTION_STRING
# Split into key=value pairs removing empties, then split the pairs into a dict
config = dict(s.split('=', 1) for s in account_connection_string.split(';') if s)
# Authentication
account_name = config.get('AccountName')
account_key = config.get('AccountKey')
# Basic URL Configuration
endpoint_suffix = config.get('EndpointSuffix')
if endpoint_suffix == None:
table_endpoint = config.get('TableEndpoint')
table_prefix = '.table.'
start_index = table_endpoint.find(table_prefix)
end_index = table_endpoint.endswith(':') and len(table_endpoint) or table_endpoint.rfind(':')
endpoint_suffix = table_endpoint[start_index+len(table_prefix):end_index]
account = TableStorageAccount(account_name = account_name, connection_string = account_connection_string, endpoint_suffix=endpoint_suffix)
I find the source sample code, and in the sample code there is still a custom module tablestorageaccount.py, it's just used to return TableService. If you already have the storage connection string and want to have a test, you could connect to table directly.
Sample:
from azure.storage.table import TableService, Entity
account_connection_string = 'DefaultEndpointsProtocol=https;AccountName=account name;AccountKey=account key;EndpointSuffix=core.windows.net'
tableservice=TableService(connection_string=account_connection_string)
Also you could refer to the new sdk to connect table. Here is the official tutorial about Get started with Azure Table storage.
I have same problem as
Why does my ML model deployment in Azure Container Instance still fail?
but the above solution does not work for me. Besides I get additional errors like belos
code": "AciDeploymentFailed",
"message": "Aci Deployment failed with exception: Your container application
crashed. This may be caused by errors in your scoring file's init()
function.\nPlease check the logs for your container instance: anomaly-detection-2.
From the AML SDK, you can run print(service.get_logs()) if you have service object
to fetch the logs. \nYou can also try to run image
mlad046a4688.azurecr.io/anomaly-detection-
2#sha256:fcbba67cf683626291c1bd084f31438fcd641ddaf80f9bdf8cea274d22d1fcb5 locally.
Please refer to http://aka.ms/debugimage#service-launch-fails for more
information.",
"details": [
{
"code": "CrashLoopBackOff",
"message": "Your container application crashed. This may be caused by errors in
your scoring file's init() function.\nPlease check the logs for your container
instance: anomaly-detection-2. From the AML SDK, you can run
print(service.get_logs()) if you have service object to fetch the logs. \nYou can
also try to run image mlad046a4688.azurecr.io/anomaly-detection-
2#sha256:fcbba67cf683626291c1bd084f31438fcd641ddaf80f9bdf8cea274d22d1fcb5 locally.
Please refer to http://aka.ms/debugimage#service-launch-fails for more
information."
}
]
}
It keeps pointing to scoring file but not sure what is wrong here
import numpy as np
import os
import pickle
import joblib
#from sklearn.externals import joblib
from sklearn.linear_model import LogisticRegression
from azureml.core.authentication import AzureCliAuthentication
from azureml.core import Model,Workspace
import logging
logging.basicConfig(level=logging.DEBUG)
def init():
global model
from sklearn.externals import joblib
# retrieve the path to the model file using the model name
model_path = Model.get_model_path(model_name='admlpkl')
print(model_path)
model = joblib.load(model_path)
#ws = Workspace.from_config(auth=cli_auth)
#logging.basicConfig(level=logging.DEBUG)
#modeld = ws.models['admlpkl']
#model=Model.deserialize(ws, modeld)
def run(raw_data):
# data = np.array(json.loads(raw_data)['data'])
# make prediction
data = json.loads(raw_data)
y_hat = model.predict(data)
#r = json.dumps(y_hat.tolist())
r = json.dumps(y_hat)
return r
The model has depencency on other file which I have added in
image_config = ContainerImage.image_configuration(execution_script="score.py",
runtime="python",
conda_file='conda_dependencies.yml',
dependencies=['modeling.py']
The logs are too abstract and really does not help to debug.I am able to create the image but provisioning service fails
Any inputs will be appreciated
Have you registered the model 'admlpkl' in your workspace using the register() function on the model object? If not, there will be no model path and that can cause failure.
See this section on model registration: https://learn.microsoft.com/en-us/azure/machine-learning/service/how-to-deploy-and-where#registermodel
Please follow the below to register and deploy the model to ACI.