I deployed Minio on Kubernetes by using the new operator:
helm repo add minio https://operator.min.io/
helm install --namespace minio-operator --create-namespace --generate-name minio/minio-operator
kubectl apply -f https://raw.githubusercontent.com/minio/operator/master/examples/tenant-tiny.yaml
sudo -E kubefwd svc
Port-Forward: 127.1.27.1 minio:80 to pod minio-ss-0-0:9000
Port-Forward: 127.1.27.2 minio-hl:9000 to pod minio-ss-0-0:9000
Port-Forward: 127.1.27.3 minio-ss-0-0.minio-hl:9000 to pod minio-ss-0-0:9000
The Tenant is working and I can access the web console both at minio:80 and minio-hl:9000
I'd like to access the storage from outside the cluster (Jupyter notebook running on Docker); this is where I receive the error "ConnectionRefusedError: [Errno 111] Connection refused":
def main():
client = Minio(
"minio:80", #minio-hl:9000 leads to the same error
access_key="minio",
secret_key="minio123!",
secure=False
)
found = client.bucket_exists("posts")
if not found:
client.make_bucket("posts")
print("Bucket 'posts' created")
else:
print("Bucket 'posts' already exists")
I also tried using pyspark with s3a jars for writing-reading objects but the connection hangs for a long time, finally receiving a similar error.
Can someone help me please? Thanks a lot!
Hope it's not too late to provide the info below:
First of all, make sure you can get access to the service after port-forward. Then try to use minio python client to do similar thing.
Actually, I have got similar issue with yours recently and finally solved by using custom http client after double confirming the service is accessible after port-forward.
from minio import Minio
from minio.error import InvalidResponseError
import urllib3
def main():
httpClient = urllib3.PoolManager(
cert_reqs="CERT_NONE")
client = Minio('my-domain:443',
access_key='xxxxx',
secret_key='xxxxx',
http_client=httpClient
)
try:
objects = client.list_objects('bucket_name', prefix='/', recursive=True)
for obj in objects:
print(obj.bucket_name)
except InvalidResponseError as err:
print("error", err)
Related
Im trying to invoke a function in a mlrun but getting above error. can anyone please help me with that.im attaching code here...
from cloudpickle import load
import numpy as np
from typing import List
import mlrun
class ClassifierModel(mlrun.serving.V2ModelServer):
def load(self):
"""load and initialize the model and/or other elements"""
model_file, extra_data = self.get_model('.pkl')
self.model = load(open(model_file, 'rb'))
def predict(self, body: dict) -> List:
"""Generate model predictions from sample."""
feats = np.asarray(body['inputs'])
result: np.ndarray = self.model.predict(feats)
return result.tolist()
#The following code converts the ClassifierModel class that you defined in the previous step to a serving function. The name of the class to be used by the serving function is set in spec.default_class.
serving_fn = mlrun.code_to_function('serving', kind='serving',image='mlrun/mlrun')
serving_fn.spec.default_class = 'ClassifierModel'
model_file = project.get_artifact_uri('my_model')
serving_fn.add_model('my_model',model_path=model_file)
#Testing Your Function Locally
my_data = '''{"inputs":[[5.1, 3.5, 1.4, 0.2],[7.7, 3.8, 6.7, 2.2]]}'''
server = serving_fn.to_mock_server()
server.test("/v2/models/my_model/infer", body=my_data)
# Building and Deploying the Serving Function¶
function_address = serving_fn.deploy()
print (f'The address for the function is {function_address} \n')
!curl $function_address
# Now we will try to invoke our serving function
serving_fn.invoke('/v2/models/my_model/infer', my_data)
OSError: error: cannot get build status, HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /api/v1/build/status?name=serving&project=getting-started-jovyan&tag=&logs=yes&offset=0&last_log_timestamp=1664873747518.8518&verbose=no (Caused by ReadTimeoutError("HTTPConnectionPool(host='localhost', port=8080): Read timed out. (read timeout=45)"))
By the looks of it, there's nothing listening on localhost:8080, even though there should be.
According to the getting started guide there should be an "MLRun Backend Service", presumably on that address by default. I suspect you haven't started the service.
The address localhost:8080 in not accessible from docker-composer, it means you have to do MLRun installation to the different IP address. I see two steps, how to solve the issue:
Relevant installation
The MLRun Community Edition in desktop docker has to be install under relevant HOST_IP (not with localhost or 127.0.0.1, but with stable IP address, see ipconfig) and with relevant SHARED_DIR. See relevant command line (from OS windows):
set HOST_IP=192.168.0.150
set SHARED_DIR=c:\Apps\mlrun-data
set TAG=1.2.0
mkdir %SHARED_DIR%
docker-compose -f "c:\Apps\mlrun\compose.with-jupyter.yaml" up
BTW: YAML file see https://docs.mlrun.org/en/latest/install/local-docker.html
2. Access to the port
In case of call serving_fn.invoke you have to open relevant port (from deploy_function) on your IP address (based on setting of HOST_IP, see the first point).
Typically this port can be blocked based on your firewall policy or your local antivirus. It means, you have to open access to this port before invoke call.
BTW:
You can test access to the port via telnet also
You can see focus on the issue https://github.com/mlrun/mlrun/issues/2102
I'am having a trouble to connect the Python API to Elasticsearch.
The Elasticsearch cluster is in azure cloud environment.
This is what I tried:
from elasticsearch import Elasticsearch
es = Elasticsearch(
"https://machine_name.kb.westeurope.azure.elastic-cloud.com:9200/",
ca_certs="/path/to/http_ca.crt",
api_key=("api_id", "api_key")
)
However i can't ping the 'es' client. In fact the test return a None Object.
if not es.ping():
print("No connection")
else:
print(es.ping())
The code abose print "No connection".
Can you please tell me what is wrong with my code ?
There is another method using the Cloud ID. Where i can find the Cloud ID ?
Please Help, Thank you so much.
Tldr;
You are pinging Kibana's domain name not Elasticsearch. The domain name is wrong. You should take the Elasticsearch domain name.
To Solve
In your url you need to change machine_name.kb.westeurope to machine_name.es.westeurope
from elasticsearch import Elasticsearch
es = Elasticsearch(
"https://machine_name.es.westeurope.azure.elastic-cloud.com:9200/",
ca_certs="/path/to/http_ca.crt",
api_key=("api_id", "api_key")
)
I’m looking to connect to a Milvus database I deployed on Google Kubernetes Engine.
I am running into an error in the last line of the script. I'm running the script locally.
Here's the process I followed to set up the GKE cluster: (https://milvus.io/docs/v2.0.0/gcp.md)
Here is a similar question I'm drawing from
Any thoughts on what I'm missing?
import os
from pymilvus import connections
from kubernetes import client, config
My_Kubernetes_IP = 'XX.XXX.XX.XX'
# Authenticate with GCP credentials
os.environ['GOOGLE_APPLICATION_CREDENTIALS'] = os.path.abspath('credentials.json')
# load milvus config file and connect to GKE instance
config = client.Configuration(os.path.abspath('milvus/config.yaml'))
config.host = f'https://{My_Kubernetes_IP}:19530'
client.Configuration.set_default(config)
## connect to milvus
milvus_ip = 'xx.xxx.xx.xx'
connections.connect(host=milvus_ip, port= '19530')
Error:
BaseException: <BaseException: (code=2, message=Fail connecting to server on xx.xxx.xx.xx:19530. Timeout)>
If you want to connect to the Milvus in the k8s cluster by ip+port, you may need to forward your local port 19530 to the Milvus service. Use a command like the following:
$ kubectl port-forward service/my-release-milvus 19530
Have you checked where your milvus external IP is?
Follow the instructions by the documentation you should use kubectl get services to check which external IP is allocated for the milvus.
blob.upload_from_filename(source) gives the error
raise exceptions.from_http_status(response.status_code, message, >response=response)
google.api_core.exceptions.Forbidden: 403 POST >https://www.googleapis.com/upload/storage/v1/b/bucket1-newsdata->bluetechsoft/o?uploadType=multipart: ('Request failed with status >code', 403, 'Expected one of', )
I am following the example of google cloud written in python here!
from google.cloud import storage
def upload_blob(bucket, source, des):
client = storage.Client.from_service_account_json('/path')
storage_client = storage.Client()
bucket = storage_client.get_bucket(bucket)
blob = bucket.blob(des)
blob.upload_from_filename(source)
I used gsutil to upload files, which is working fine.
Tried to list the bucket names using the python script which is also working fine.
I have necessary permissions and GOOGLE_APPLICATION_CREDENTIALS set.
This whole things wasn't working because I didn't have permission storage admin in the service account that I am using in GCP.
Allowing storage admin to my service account solved my problem.
As other answers have indicated that this is related to the issue of permission, I have found one following command as useful way to create default application credential for currently logged in user.
Assuming, you got this error, while running this code in some machine. Just following steps would be sufficient:
SSH to vm where code is running or will be running. Make sure you are user, who has permission to upload things in google storage.
Run following command:
gcloud auth application-default login
This above command will ask to create token by clicking on url. Generate token and paste in ssh console.
That's it. All your python application started as that user, will use this as default credential for storage buckets interaction.
Happy GCP'ing :)
This question is more appropriate for a support case.
As you are getting a 403, most likely you are missing a permission on IAM, the Google Cloud Platform support team will be able to inspect your resources and configurations.
This is what worked for me when the google documentation didn't work. I was getting the same error with the appropriate permissions.
import pathlib
import google.cloud.storage as gcs
client = gcs.Client()
#set target file to write to
target = pathlib.Path("local_file.txt")
#set file to download
FULL_FILE_PATH = "gs://bucket_name/folder_name/file_name.txt"
#open filestream with write permissions
with target.open(mode="wb") as downloaded_file:
#download and write file locally
client.download_blob_to_file(FULL_FILE_PATH, downloaded_file)
I've build the following script:
import boto
import sys
import gcs_oauth2_boto_plugin
def check_size_lzo(ds):
# URI scheme for Cloud Storage.
CLIENT_ID = 'myclientid'
CLIENT_SECRET = 'mysecret'
GOOGLE_STORAGE = 'gs'
dir_file= 'date_id={ds}/apollo_export_{ds}.lzo'.format(ds=ds)
gcs_oauth2_boto_plugin.SetFallbackClientIdAndSecret(CLIENT_ID, CLIENT_SECRET)
uri = boto.storage_uri('my_bucket/data/apollo/prod/'+ dir_file, GOOGLE_STORAGE)
key = uri.get_key()
if key.size < 45379959:
raise ValueError('umg lzo file is too small, investigate')
else:
print('umg lzo file is %sMB' % round((key.size/1e6),2))
if __name__ == "__main__":
check_size_lzo(sys.argv[1])
It works fine locally but when I try and run on kubernetes cluster I get the following error:
boto.exception.GSResponseError: GSResponseError: 403 Access denied to 'gs://my_bucket/data/apollo/prod/date_id=20180628/apollo_export_20180628.lzo'
I have updated the .boto file on my cluster and added my oauth client id and secret but still having the same issue.
Would really appreciate help resolving this issue.
Many thanks!
If it works in one environment and fails in another, I assume that you're getting your auth from a .boto file (or possibly from the OAUTH2_CLIENT_ID environment variable), but your kubernetes instance is lacking such a file. That you got a 403 instead of a 401 says that your remote server is correctly authenticating as somebody, but that somebody is not authorized to access the object, so presumably you're making the call as a different user.
Unless you've changed something, I'm guessing that you're getting the default Kubernetes Engine auth, with means a service account associated with your project. That service account probably hasn't been granted read permission for your object, which is why you're getting a 403. Grant it read/write permission for your GCS resources, and that should solve the problem.
Also note that by default the default credentials aren't scoped to include GCS, so you'll need to add that as well and then restart the instance.