We Keep Coding

Trouble on training YoloV5 on AWS Sagemaker | AlgorithmError: , exit code: 1

I'm trying to train YoloV5 on AWS Sagemaker with custom data (that is stored in S3) via a Docker Image (ECR) and I keep getting "AlgorithmError: , exit code: 1". Can someone please tell me how to debug this problem?
Here's the Docker Image :
# GET THE AWS IMAGE
FROM 763104351884.dkr.ecr.eu-west-3.amazonaws.com/pytorch-training:1.11.0-gpu-py38-cu113-ubuntu20.04-sagemaker
# UPDATES
RUN apt update
RUN DEBIAN_FRONTEND=noninteractive TZ=Etc/UTC apt install -y tzdata
RUN apt install -y python3-pip git zip curl htop screen libgl1-mesa-glx libglib2.0-0
RUN alias python=python3
# INSTALL REQUIREMENTS
COPY requirements.txt .
RUN python3 -m pip install --upgrade pip
RUN pip install --no-cache -r requirements.txt albumentations gsutil notebook \
coremltools onnx onnx-simplifier onnxruntime openvino-dev tensorflow-cpu tensorflowjs
COPY code /opt/ml/code
WORKDIR /opt/ml/code
RUN git clone https://github.com/ultralytics/yolov5 /opt/ml/code/yolov5
ENV SAGEMAKER_SUBMIT_DIRECTORY /opt/ml/code
ENV SAGEMAKER_PROGRAM trainYolo.py
ENTRYPOINT ["python", "trainYolo.py"]
And here's trainYolo.py :
import json
import os
import numpy as np
import cv2 as cv
import subprocess
import yaml
import shutil
trainSet = os.environ["SM_CHANNEL_TRAIN"]
valSet = os.environ["SM_CHANNEL_VAL"]
output_dir = os.environ["SM_CHANNEL_OUTPUT"]
#Creating the data.yaml for yolo
dict_file = [{'names' : ['block']},
{'nc' : ['1']}, {'train': [trainSet]}
, {'val': [valSet]}]
with open(r'data.yaml', 'w') as file:
documents = yaml.dump(dict_file, file)
#Execute this command to train Yolo
res = subprocess.run(["python3", "yolov5/train.py", "--batch", "16" "--epochs", "100", "--data", "data.yaml", "--cfg", "yolov5/models/yolov5s.yaml","--weights", "yolov5s.pt" "--cache"], shell=True)
shutil.copy("yolov5", output_dir)
Note : I'm not sure if subprocess.run() works in an environment such as Sagemaker.
Thank you

So your training script is not configured properly. When using a SageMaker estimator or Script Mode you must configure it in a format that will save the model properly. Here's an example notebook with TensorFlow and script mode. If you would like to build your own Dockerfile (Bring Your Own Container) then you would have to configure your train file as shown in the second link.
Script-Mode: https://github.com/RamVegiraju/SageMaker-Deployment/tree/master/RealTime/Script-Mode/TensorFlow/Classification
BYOC: https://github.com/RamVegiraju/SageMaker-Deployment/tree/master/RealTime/BYOC/Sklearn/Sklearn-Regressor/container/randomForest

Why is my container copying a non existing file?

I am currently trying to deploy and serve a fasttext model for a business venture. I decided to use Google's Vertex AI (if you have a better idea of something to use, please do!). I created a dockerfile and training script to train my model, I built the docker image and then pushed it to the Google Cloud Repository. Here is the code for it :
Dockerfile :
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install -y \
build-essential \
wget \
git \
python-dev \
unzip \
python-numpy \
python-scipy \
&& rm -rf /var/cache/apk/*
RUN wget -nv \
https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz && \
mkdir /root/tools && \
tar xvzf google-cloud-sdk.tar.gz -C /root/tools && \
rm google-cloud-sdk.tar.gz && \
/root/tools/google-cloud-sdk/install.sh --usage-reporting=false \
--path-update=false --bash-completion=false \
--disable-installation-options && \
rm -rf /root/.config/* && \
ln -s /root/.config /config && \
# Remove the backup directory that gcloud creates
rm -rf /root/tools/google-cloud-sdk/.install/.backup
# Path configuration
ENV PATH $PATH:/root/tools/google-cloud-sdk/bin
# Make sure gsutil will use the default service account
RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg
RUN pip3 install fasttext
RUN pip3 install google
RUN pip3 install google-cloud-storage
RUN pip3 install --upgrade google-api-python-client
RUN pip3 install --upgrade google-cloud
COPY . .
ENTRYPOINT ["python3", "trainer.py"]
Trainer.py :
import fasttext
from google.cloud import storage
import tempfile
from google.cloud.storage import blob
from google.oauth2 import service_account
credentials = service_account.Credentials.from_service_account_file('endless-bank-344008-a75f5b89470f.json')
with tempfile.NamedTemporaryFile() as tmp_file:
local_model_file = tmp_file.name
remote_model_file = storage.Client('endless-bank-344008', credentials).bucket('bucket2035').blob('cc.en.300.bin')
remote_model_file.download_to_filename(local_model_file)
model_1 = fasttext.load_model(local_model_file)
model_1.save_model("plagscan.bin")
target = storage.Client('endless-bank-344008', credentials).bucket('bucket2035').blob('plagscanner.bin')
target.upload_from_filename('plagscan.bin')
This code, works, which is great. I run it in the vertex ai platform, I press create a model, check everything that applies, use a custom container (after selecting the one I created that is now in the google cloud registry), it runs, very cool, no prediction container. It runs, doesn't create a model because there is no prediction container but it runs successfully and in the bucket2035 there is indeed an output file "plagscanne.bin". Then I created a dockerfile flask app thing to serve as a prediction container, here is the dockerfile and the flask app :
Dockerfile :
FROM python:3.8-slim-buster
RUN apt-get update && apt-get install -y \
build-essential \
wget \
git \
python-dev \
unzip \
python-numpy \
python-scipy \
&& rm -rf /var/cache/apk/*
RUN wget -nv \
https://dl.google.com/dl/cloudsdk/release/google-cloud-sdk.tar.gz && \
mkdir /root/tools && \
tar xvzf google-cloud-sdk.tar.gz -C /root/tools && \
rm google-cloud-sdk.tar.gz && \
/root/tools/google-cloud-sdk/install.sh --usage-reporting=false \
--path-update=false --bash-completion=false \
--disable-installation-options && \
rm -rf /root/.config/* && \
ln -s /root/.config /config && \
# Remove the backup directory that gcloud creates
rm -rf /root/tools/google-cloud-sdk/.install/.backup
# Path configuration
ENV PATH $PATH:/root/tools/google-cloud-sdk/bin
# Make sure gsutil will use the default service account
RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg
RUN pip3 install flask
RUN pip3 install fasttext
RUN pip3 install google
RUN pip3 install google-cloud-storage
RUN pip3 install --upgrade google-api-python-client
RUN pip3 install --upgrade google-cloud
RUN pip3 install simplejson
COPY . .
ENV FLASK_APP=app.py
EXPOSE 8080
CMD flask run --host=0.0.0.0 --port=8080
Flask app :
import fasttext
from google.cloud import storage
import tempfile
from google.cloud.storage import blob
from google.oauth2 import service_account
import json
import os
import simplejson
from flask import Flask, request, Response
a = os.path.join(model_dir, 'plagscanner.bin')
model_1 = fasttext.load_model(a)
app = Flask(__name__)
#app.route("/isalive")
def isalive():
print("/isalive request")
status_code = Response(status=200)
return status_code
# Flask route for predictions
#app.route('/predict',methods=['GET','POST'])
def prediction():
result = request.get_json(silent=True, force=True)
data = result['words']
wordvectors = json.dumps([model_1(x) for x in data])
return wordvectors
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8080)
Now... this should work right? Wrong. I built this container, pushed it to the google cloud registry and it didn't work, bizarely it gave me the error : Training pipeline failed with error message: There are no files under "gs://bucket2035/model" to copy.
Very bizarre, so instead I tried a variation of the app.py code, this version instead downloaded the model training file via bucket download :
import fasttext
from google.cloud import storage
import tempfile
from google.cloud.storage import blob
from google.oauth2 import service_account
import json
import os
import simplejson
from flask import Flask, request, Response
credentials = service_account.Credentials.from_service_account_file('endless-bank-344008-a75f5b89470f.json')
with tempfile.NamedTemporaryFile() as tmp_file:
local_model_file = tmp_file.name
remote_model_file = storage.Client('endless-bank-344008', credentials).bucket('bucket2035').blob('cc.en.300.bin')
remote_model_file.download_to_filename(local_model_file)
model_1 = fasttext.load_model(local_model_file)
app = Flask(__name__)
#app.route("/isalive")
def isalive():
print("/isalive request")
status_code = Response(status=200)
return status_code
# Flask route for predictions
#app.route('/predict',methods=['GET','POST'])
def prediction():
result = request.get_json(silent=True, force=True)
data = result['words']
wordvectors = json.dumps([model_1(x) for x in data])
return wordvectors
if __name__ == "__main__":
app.run(debug=True, host='0.0.0.0', port=8080)
Here is the full error :
Training pipeline failed with error message: There are no files under "gs://bucket2035/model" to copy.
Now guess what happens! It gives the same error. I don't understand this, what is it I'm trying to copy? Why is it not working? Is there another solution as opposed to Vertex AI I should be using for this very simple thing? What is the meaning of life (lol)? Please help, I've tried many things and none of them work and I kinda think that there must be an easier solution to this problem. Anyways, any help would be appreciated!

passing url as arguments in python when running docker image

I want to pass an GCP storage URL as argument when running my docker image, so that it can pull my csv file from my storage and print the dataset .
Below is my dockerfile
# Use the official lightweight Python image.
# https://hub.docker.com/_/python
FROM continuumio/miniconda3
# Copy local code to the container image.
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
# Install production dependencies.
RUN pip install Flask gunicorn
RUN pip install scikit-learn==0.20.2 firefly-python==0.1.15
RUN pip install --upgrade google-cloud-storage
ENTRYPOINT ["python"]
CMD ["pre.py"]
I tried running the docker image by below command and getting below error
docker run preprocess:v1 "https://storage.googleapis.com/MYBucket/app/Model/IrisClassifier.sav"
.
python: can't open file 'https://storage.googleapis.com/MYBucket/app/Model/IrisClassifier.sav': [Errno 2] No such file or directory
import os
import argparse
from google.cloud import storage
from sklearn.externals import joblib
from urllib.request import urlopen
def parse_arguments():
print('entered parse arg')
parser = argparse.ArgumentParser()
parser.add_argument('data_dir', type=str, help='GCSpath')
args = parser.parse_known_args()[0]
print('Argument passed')
print(os.getcwd())
print('STARTING CLOUD RETRIVAL')
print('*****client initialized')
dataset_load = joblib.load(urlopen(args.dat_dir))
print('*****loaded Dataset')
print(dataset_load)
def main(_):
print("Prior to entering arg")
parse_arguments()
I want to pass a similar GCP bucket when running my docker image
https://storage.googleapis.com/MYBucket/app/Model/IrisClassifier.sav

you need to change all your CMD to ENTRYPOINT at first:
FROM continuumio/miniconda3
ENV APP_HOME /app
WORKDIR $APP_HOME
COPY . ./
RUN pip install Flask gunicorn
RUN pip install scikit-learn==0.20.2 firefly-python==0.1.15
RUN pip install --upgrade google-cloud-storage
ENTRYPOINT ["python", "pre.py"]
then you can pass your URL.
The Problem with your setup is:
docker will start the entrypoint and that is python and with your command you overwrite the CMD wich will give you:
python YOUR_URL
Update
I do not know if you add if statement to run the main def but here how you schould edit the script:
def main():
print("Prior to entering arg")
parse_arguments()
if __name__ == '__main__':
main()

How to fix 'No module named cv2' in AWS EC2 Linux, imports from Command Line

I am deploying a web application onto an AWS EC2 instance, and I'm getting an error. The logs indicate that I do not have cv2 installed.
ModuleNotFoundError: No module named 'cv2'
However, if I ssh into my instance, and run python from the shell I can import no problem.
https://drive.google.com/drive/folders/1-w3BN9pMAhkiDM40fODCPdjvU1Nx71UT?usp=sharing
I have already installed opencv onto the Linux server and checked that it is available for import.
From my application.py file
import cv2
File "/opt/python/current/app/localize.py", line 9, in
but from the command line:
>>> import cv2
>>> cv2.__version__
'4.1.0'
I expected the import to work since it works from the command line.

Check if your python package is available for the root/admin user but not accessible for the user trying to run the code?
If you can import that module in your EC2, then it is installed, but more importantly for which user it is installed, and for which version.
First try :
chmod 755 on all directories in python path for your default python and see if it works.(This will provide permissions for all import libraries in Python)
If your script is running Python3.7 and Default is Python2.7 then you might have to do --
sudo pip3 install opencv-python
the way to check the version defaults is:
which python ---Will provide default python path and version
which pip ---- Will Provide default PIP details

As #rayryeng suggested, I'm running Python 3.x from Elastic Beanstalk, and Python 2.x from command line. I fixed it by installing the correct version of cv2 for Python 3 and including the following before my import:
import sys
sys.path.append('/usr/local/lib64/python3.6/site-packages')

Try this:
cd
wget https://github.com/opencv/opencv/archive/3.2.0.zip
virtualenv project
source project/bin/activate
pip install numpy
mkdir local
unzip opencv-3.2.0.zip
cd opencv-3.2.0
mkdir build
cd build
cmake -D CMAKE_BUILD_TYPE=RELEASE -D BUILD_SHARED_LIBS=NO -D WITH_FFMPEG=ON -D BUILD_opencv_python2=ON -D CMAKE_INSTALL_PREFIX=~/local ~/opencv-3.2.0
make
make install
cp ~/local/lib/python2.7/site-packages/cv2.so ~/project/lib64/python2.7/site-packages/
Read more from here: Source

Error on trying to use tensorboard with google colab

I am getting an error on trying to use tensorboard with google collab.
I am using ngork to run tensorboard. The error is as follows
The code I am using to do the above-mentioned operation is as follows
LOG_DIR = '/content/drive/My Drive/Practice/Su'
get_ipython().system_raw(
'tensorboard --logdir {} --host 0.0.0.0 --port 6006 &'
.format(LOG_DIR)
)
!wget https://bin.equinox.io/c/4VmDzA7iaHb/ngrok-stable-linux-
amd64.zip
!unzip ngrok-stable-linux-amd64.zip
get_ipython().system_raw('./ngrok http 6006 &')
! curl -s http://localhost:4040/api/tunnels | python3 -c \
"import sys, json; print(json.load(sys.stdin)['tunnels'][0]
['public_url'])"

Skip ngrok and use the built-in %tensorboard magic.
Here's a demo:
https://colab.research.google.com/github/tensorflow/tensorboard/blob/master/docs/r2/tensorboard_in_notebooks.ipynb

Here's a solution that worked for me:
First uninstall tensorboard and tensorflow:
!pip3 uninstall tensorboard
!pip3 uninstall tensorflow
then install tf-nightly:
!pip3 install --ignore-installed tf-nightly
then run tensorboard inside google colab:
%load_ext tensorboard
%tensorboard --logdir {logs_base_dir}