We Keep Coding

Tensorflow Keras error "no algorithm found!"

I am currently following the book Hands-on Machine Learning with Scikit-Learn, Keras, and TensorFlow, but I keep coming across these errors. When I run the following code:
model.compile(loss="sparse_categorical_crossentropy", optimizer="nadam", metrics=["accuracy"])
history = model.fit(X_train, y_train, epochs=10, validation_data=(X_valid, y_valid))
score = model.evaluate(X_test, y_test)
y_pred = model.predict(X_new)
I get this error
NotFoundError: No algorithm worked!
[[node sequential/conv2d/Conv2D (defined at :2) ]] [Op:__inference_train_function_2275]
Function call stack:
train_function
I then pulled these two things from stack overflow, which seemed to "fix" the problem
os.environ['TF_CPP_MIN_LOG_LEVEL'] = "2"
config = ConfigProto()
config.gpu_options.allow_growth = True
session = InteractiveSession(config=config)
However then this message clogs my terminal:
2021-04-10 18:20:01.643838: W tensorflow/stream_executor/gpu/asm_compiler.cc:235] Your CUDA software stack is old. We fallback to the NVIDIA driver for some compilation. Update your CUDA version to get the best performance. The ptxas error was: ptxas fatal : Value 'sm_86' is not defined for option 'gpu-name'
I am running cuda 11.2, and I have driver version 460.39. My card is a 3080.
Does anyone know what might be going wrong?

Perceptron in Python

I'm using sklearn library. I have a question about the attribute: n_iter_. When executing the code I get TypeError: __init__() got an unexpected keyword argument 'n_iter_'. Also try using n_iter but I get the same error, or maybe I am misspelling the attribute. It is not all the code, if you need more information, let me know
from sklearn.linear_model import Perceptron
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
ppn= Perceptron(n_iter_=40, eta0= 0.1, random_state=1)
ppn.fit(X_train_std, y_train)

Perceptron Model in sklearn.linear_model doesn't have n_iter_ as a parameter. It has following parameters with similar names.
max_iter: int, default=1000
The maximum number of passes over the training data (aka epochs). It only impacts the behavior in the fit method, and not the partial_fit method.
and
n_iter_no_change : int, default=5
Number of iterations with no improvement to wait before early stopping.
New in version 0.20.
By looking at your code it looks like you intended to use max_iter.
So do
ppn=Perceptron(max_iter=40, eta0= 0.1, random_state=1)
ppn.fit(X_train_std, y_train)
Note:
You should first upgrade your sklearn using
pip install sklearn -upgrade

The attribute given in the documentation is n_iter and not n_iter_
So this should work:
from sklearn.linear_model import Perceptron
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
ppn=Perceptron(n_iter=40, eta0= 0.1, random_state=1)
ppn.fit(X_train_std, y_train)

First check which Scikit-learn version you have installed. You can do that by executing
python -c "import sklearn;print(sklearn.__version__)"
on your terminal/environment to which you have the python that executes your code.
Perceptron initial parameters have changed from n_iter to max_iter in version 0.20. The best way to keep up, head to the documentation or source code of the correct version and read the params: e.g.
documentation: perceptron docs v.0.23
source code: perceptions.0.23 code

How do I use hparams with estimators?

To log hparams without using Keras, I'm doing the following as suggested in the tf code here:
with tf.summary.create_file_writer(model_dir).as_default():
hp_learning_rate = hp.HParam("learning_rate", hp.RealInterval(0.00001, 0.1))
hp_distance_margin = hp.HParam("distance_margin", hp.RealInterval(0.1, 1.0))
hparams_list = [
hp_learning_rate,
hp_distance_margin
]
metrics_to_monitor = [
hp.Metric("metrics_standalone/auc", group="validation"),
hp.Metric("loss", group="train", display_name="training loss"),
]
hp.hparams_config(hparams=hparams_list, metrics=metrics_to_monitor)
hparams = {
hp_learning_rate: params.learning_rate,
hp_distance_margin: params.distance_margin,
}
hp.hparams(hparams)
Note that params is a dictionary object here that I'll pass to the estimator.
Then I train the estimator as usual,
config = tf.estimator.RunConfig(model_dir=params.model_dir)
estimator = tf.estimator.Estimator(model_fn, params=params, config=config)
train_spec = tf.estimator.TrainSpec(...)
eval_spec = tf.estimator.EvalSpec(...)
tf.estimator.train_and_evaluate(estimator, train_spec=train_spec, eval_spec=eval_spec)
After training, when I launch tensorboard, I do have the hparams logged, but I do not see any metrics logged against them
I further confirmed that they show up in the scalars page with the same tag name for both train and validation i.e. . and ./eval, but the hparams page doesn't see those logged tensors.
How do I use hparams with estimators?
I'm using
tensorboard 2.1.0
tensorflow 2.1.0
tensorflow-estimator 2.1.0
tensorflow-metadata 0.15.2
on Python 3.7.5
Attempt 1:
After some googling, I saw some older tf code where they passed hparams to params argument of Estimator, so just to make sure if tf2 logs those hparams by itself when given, I checked the Estimator docs and it says:
The params argument contains hyperparameters. It is passed to the
model_fn, if the model_fn has a parameter named "params", and to the
input functions in the same manner. Estimator only passes params
along, it does not inspect it. The structure of params is therefore
entirely up to the developer.
So using hparams as params will not be useful.
Attempt 2:
I doubt that since estimators use tensorflow.python.summary instead of tf.summary which is the default in v2, tensors logged by v1 was probably not accessible and so, I also tried to use
with tensorflow.python.summary.FileWriter(model_dir).as_default()
However that failed with RuntimeError: tf.summary.FileWriter is not compatible with eager execution. Use tf.contrib.summary instead.
Update: I ran it with eager execution disabled. Now, even the hparam initial logging did not happen. There was no hparams tab in tensorboard as it failed with error
E0129 13:03:07.656290 21584 hparams_plugin.py:104] HParams error: Can't find an HParams-plugin experiment data in the log directory. Note that it takes some time to scan the log directory; if you just started Tensorboard it could be that we haven't finished scanning it yet. Consider trying again in a few seconds.
Is there a way to make tensorboard read already logged metric tensors and link them with hparams?

The culprit seems to be
# This doesn't seem to compatible with Estimator API
hp.hparams_config(hparams=hparams_list, metrics=metrics_to_monitor)
Simply calling hparams logs all metrics logged with tf.summary. Then in tensorboard, you can filter only the metrics you need and then compare trials.
with tf.summary.create_file_writer(train_folder).as_default():
# params is a dict which contains
# { 'learning_rate': 0.001, 'distance_margin': 0.5,...}
hp.hparams(hparams=params))

Parallel threading with xgboost?

According to its documentation, xgboost has an n_jobs parameter. However, when I attempt to set n_jobs, I get this error:
TypeError: __init__() got an unexpected keyword argument 'n_jobs'
Same issue for some other parameters like random_state. I assumed this might be an update issue, but it seems I have the latest version (0.6a2, installed with pip).
There isn't much needed for me to reproduce the error:
from xgboost import XGBClassifier
estimator_xGBM = XGBClassifier(max_depth = 5, learning_rate = 0.05, n_estimators = 400, n_jobs = -1).fit(x_train)
Any ideas?

I installed xgboost yesterday (25.09.2017):
If you install with pip or conda, the xgboost version does not support the n_jobs parameter; only the nthreads parameter.
If you build xgboost from github repository, you can use n_jobs though.
Update: n_jobs is the number of parallel threads used to run xgboost. (replaces nthread) for all algorithms like XGBClassifier, XGBRanker, XGBRegressor etc.
Reference - here

nthread is same with n_jobs but n_jobs is prefered for now.
maybe you can try nthread instead.
nthread : int
Number of parallel threads used to run xgboost. (Deprecated, please use n_jobs)
n_jobs : int
Number of parallel threads used to run xgboost. (replaces nthread)
https://github.com/dmlc/xgboost/blob/master/python-package/xgboost/sklearn.py#L71-L74

Use this for n_jobs value for your machine
import multiprocessing
n_jobs = multiprocessing.cpu_count()-1
print(n_jobs)

How to parallelize with jupyter and sklearn?

I'm trying to parallelize the GridSearchCV of scikit-learn. It's running on a jupyter (hub) notebook environment. After some research I found this code:
from sklearn.externals.joblib import Parallel, parallel_backend, register_parallel_backend
from ipyparallel import Client
from ipyparallel.joblib import IPythonParallelBackend
c = Client(profile='myprofile')
print(c.ids)
bview = c.load_balanced_view()
register_parallel_backend('ipyparallel', lambda : IPythonParallelBackend(view=bview))
grid = GridSearchCV(pipeline, cv=3, n_jobs=4, param_grid=param_grid)
with parallel_backend('ipyparallel'):
grid.fit(X_train, Y_train)
Note that I've set the n_jobs parameter to 4, what is the number of machine's cpu cores. (It's what nproc returns)
But it doesn't seem to work: ImportError: cannot import name 'register_parallel_backend', although I installed joblib with conda install joblib and also tried pip install -U joblib.
So, what's the best way to parallelize the GridSearchCV in this environment?
UPDATE:
Without ipyparallel and just setting the n_jobs parameter:
grid = GridSearchCV(pipeline, cv=3, n_jobs=4, param_grid=param_grid)
grid.fit(X_train, Y_train)
Result is the following warning message:
/opt/conda/lib/python3.5/site- packages/sklearn/externals/joblib/parallel.py:540: UserWarning:
Multiprocessing-backed parallel loops cannot be nested, setting n_jobs=1
Seems like it ends up in sequential execution rather than parallel execution.