I using kedro in conjunction with databricks-connect to run my machine learning model. I am trained and tested the model using databricks notebook and saved the pickle file of the model to azure blob storage. To test the pipeline locally I downloaded the pickle file and stored it locally. kedro reads in the file fine when stored locally, however the problem is when I try to read in the file into kedro directly from azure; I get the following error : "Invalid Load Key '?'
Unfortunately I cannot post any screenshots since I am doing this for my job. My thinking is that the pickle file is stored differently while on azure and when downloaded locally it is stored correctly.
For a bit more clarity I am attempting to store a logistic Regression model trained used statsmodels (sm.logit)
Pickles unfortunately aren't guaranteed to work across environments, at the very least the python version (and possibly OS) need to be identical. Is there a mismatch here?
Related
Working on my local computer, I've created a Tensorflow Object Detector. I have exported the model (which I've tested using the checkpoints) to a protobuf file as well as several others (TF lite, TF js, etc). I now need to transfer this trained model to another computer that doesn't have the Object Detection API or other things I needed to build the model.
Do I need all these dependencies on the new machine? Or, does the protobuf file contain everything that the machine will need? The new machine only has the basic anaconda environment packages as well as tensorflow.
Protobuf files most commonly contains both model and weights. So in theory you can load your model on any machine with TensorFlow.
The only problem that I can think of is saving custom layers/losses/optimizers and data pre/postprocessing.
I have made an MLP model using sklearn which I saved as a .pkl file. I can upload it in Azure Jupyter notebooks, but not locally. The following error message appears when loading the model:
No module named 'sklearn.neural_network._multilayer_perceptron'
I installed the sklearn version '0.22.2.post1' for jupyter notebooks (since this is the version the MLP model was created in Azure). Does anyone know what the problem with upload the MLP model could be and how to solve it?
When running the line: 'sklearn.version', it said to be the version '0.22.2.post1' both when using it in Jupyter Notebooks in Azure and locally in Jupyter Notebooks.
As for relevant code parts, the MLP models are stored in a dictionary (called models) and then saved as a pickle file (name_of_dicktionary_the_models_are_saved_in.pkl):
ml_results_file = open("folder/name_of_dicktionary_the_models_are_saved_in.pkl", "wb") pickle.dump(models, ml_results_file) ml_results_file.close()
The way the pickle files are opened is:
ml_results_file = open("folder/name_of_dicktionary_the_models_are_saved_in.pkl", "rb") models = pickle.load(ml_results_file) ml_results_file.close()
I have a model created in Tensorflow that is already trained and accurate. How do I get this to run in a project? I can load the model but I can't figure out how to feed it a single image that my software is generating.
Also if this was a transfer learning project do I have to load and create a model before loading the weights?
All the tutorials are on how to set it up in the cloud or with a local server which I would like to avoid. I am tempted to save the data and then run it but that is a lot slower.
Addition:
My environment I am building this for is a google colab Jupyter notebook. The idea is to have no installation for users which is why it must be self contained.
I am facing an issue with serving tensorflow models on AWS SageMaker. I have trained the model outside of the SageMaker environment, now I have a savedmodel.pb file and I need to deploy it on a SageMaker endpoint. So I simply zipped the model file and uploaded it to an S3 bucket.
Now, when trying to create an endpoint, I get the following error in my Cloudwatch log:
tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:369]
FileSystemStoragePathSource encountered a file-system access error:
Could not find base path /opt/ml/model/export/Servo for servable
generic_model
I believe SageMaker is looking for the tar.gz to follow a particular directory structure. However, all I have is a .pb file.
TensorFlow Serving expects the following folder structure:
export/Servo/{version_number} where the version number is any valid positive number.
SageMaker expects the same directory format of TFS, there is a GH issue about this https://github.com/aws/sagemaker-python-sdk/issues/599
I have build a model in h2o flow. Once completed I save the file to my local machine. Next, I import some new data via Flow and python shell, I then load the model and run predict on the dataframe.
I should get the exact same results as the dataframe and model are both the same. Just different environments. However, I get a different predictions in the different environments. Anybody have a similar experience?