We Keep Coding

Azure ML jobs: anaconda environment with custom / own module

I am building a pipeline job along the lines of this guide:
https://learn.microsoft.com/en-us/azure/machine-learning/tutorial-pipeline-python-sdk
I have a local environment.yaml which I am sending via the SDK to be built and registered by the ML Studio workspace. This works for 3rd party dependencies such as numpy. But in my pipeline components script (scripts/train.py) I am importing my own module, which that script is a part of.
Component's definition (TRAIN_CONFIG) that uses train.py
name: training_job
display_name: Training
# version: 1 # Not specifying a version will automatically update the version
type: command
inputs:
registered_model_name:
type: string
outputs:
model:
type: uri_folder
code: .
environment:
azureml:AzureML-sklearn-0.24-ubuntu18.04-py37-cpu:21
command: >-
python scripts/train.py
Snippet from script that triggers the remote training job:
train_component: Component = load_component(source=TRAIN_CONFIG)
train_component.environment = get_ml_client().environments.get("training-job-env", version="42")
Since the environment build job (which basically uses conda env create -f environment.yaml) fails when I have <my module> as a dependency in the environment.yaml, since it of course does not find that module, I took it out. But obviously, then <mv module> is missing from the environment when the job runs on Azure:
Traceback (most recent call last):
File "/mnt/azureml/cr/j/xxx/exe/wd/scripts/train.py", line 11, in <module>
from <my module> import <...>
ModuleNotFoundError: No module named '<my module>'
So, given that seemingly I do not have more control over the job environment than passing the environment.yaml, how do I get <my module> installed in the job environment?

ImportError: No module named libvirt error whyle trying to install python for libvirt on NetBSD 9.2

I've just installed virt-manager with pkgin on NetBSD 9.2 just because I want to emulate the virtual machines with qemu + nvmm on NetBSD 9.2. The installation of virt-manager went ok. But,when I ran it,an error came up :
netbsd-marietto# virt-manager
Traceback (most recent call last):
File "/usr/pkg/share/virt-manager/virt-manager.py", line 386, in <module>
main()
File "/usr/pkg/share/virt-manager/virt-manager.py", line 247, in main
from virtManager import cli
File "/usr/pkg/share/virt-manager/virtManager/cli.py", line 29, in <module>
import libvirt
ImportError: No module named libvirt
Googling a little bit maybe I've found the solution here :
https://www.unitedbsd.com/d/285-linux-user-and-netbsd-enthusiast-hoping-to-migrate-some-day
where "kim" said :
Looking at pkgsrc/sysutils/libvirt/PLIST it doesn't look like the package provides any Python bindings -- which is what the "ImportError: No module named libvirt" error message is about. You could try py-libvirt from pkgsrc-wip and see how that works out.
I tried to start the compilation like this :
netbsd-marietto# cd /home/mario/Desktop/pkgsrc-wip/py-libvirt
netbsd-marietto# make
but I've got this error :
make: "/home/mario/Desktop/pkgsrc-wip/py-libvirt/Makefile" line 15: Could not find ../../wip/libvirt/buildlink3.mk
make: "/home/mario/Desktop/pkgsrc-wip/py-libvirt/Makefile" line 16: Could not find ../../lang/python/distutils.mk
make: "/home/mario/Desktop/pkgsrc-wip/py-libvirt/Makefile" line 17: Could not find ../../mk/bsd.pkg.mk
make: Fatal errors encountered -- cannot continue
If u want to see the content of the Makefile,it is :
gedit /home/mario/Desktop/pkgsrc-wip/py-libvirt/Makefile
#$NetBSD: Makefile,v 1.32 2018/11/30 09:59:40 adam Exp $
PKGNAME= ${PYPKGPREFIX}-${DISTNAME:S/-python//}
DISTNAME= libvirt-python-5.8.0
CATEGORIES= sysutils python
MASTER_SITES= https://libvirt.org/sources/python/
MAINTAINER= pkgsrc-users#NetBSD.org
HOMEPAGE= https://libvirt.org/sources/python/
COMMENT= libvirt python library
LICENSE= gnu-lgpl-v2
USE_TOOLS+= pkg-config
.include "../../wip/libvirt/buildlink3.mk"
.include "../../lang/python/distutils.mk"
.include "../../mk/bsd.pkg.mk"
Can someone help me to fix the error ? very thanks.

You are getting those errors because your copy of pkgsrc wip is not inside a pkgsrc tree.
Please follow the pkgsrc wip documentation to correctly use pkgsrc wip. Especially look at the section titled Getting the "source".
In brief, assuming you have a copy of pkgsrc in /usr/pkgsrc and you want to use git to checkout pkgsrc wip, run these commands:
cd /usr/pkgsrc
git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip
Then build py-libvirt with these commands:
cd /usr/pkgsrc/wip/py-libvirt
make
If it builds successfully, you could install it with:
cd /usr/pkgsrc/wip/py-libvirt
make install

ModuleNotFoundError: No module named 'tensorflow.tensorboard.tensorboard'

There seems to be a problem with recent TensorFlow build. The TensorBoard visualization tool would not run when it is compiled from sources to use with GPU. The error is as follows:
$ tensorboard
Traceback (most recent call last):
File "/home/gpu/anaconda3/envs/tensorflow/bin/tensorboard", line 7, in <module>
from tensorflow.tensorboard.tensorboard import main
ModuleNotFoundError: No module named 'tensorflow.tensorboard.tensorboard'
Specs of system: Ubuntu 16.04, NVIDIA GTX 1070, cuda-8.0, cudnn 6.0.
Installed using Bazel from sources as described here:
https://www.tensorflow.org/install/install_sources
Installed into fresh anaconda3 environment 'tensorflow', environment is activated when performing command.
Would appreciate any help!

An easy fix:
python -m tensorboard.main --logdir=/path/to/logs

After some trial and error, I have solved this issue by adapting the file tensorboard-script.py in path/to/conda/envs/myenv/Scripts (Windows) as follows:
if __name__ == '__main__':
import sys
#import tensorflow.tensorboard.tensorboard
import tensorboard.main
#sys.exit(tensorflow.tensorboard.tensorboard.main())
sys.exit(tensorboard.main.main())
Now I can invoke tensorboard as expected:
tensorboard --logdir=log/ --port 6006

Okay, I've found a solution that works and also received some explanation from tensorflower on github.
There might be an issue with tensorboard when compiling tensorflow from sources because tensorboard is now removed to a separate repo and is not a part of tensorflow. The tensorflower said the docs will be updated eventually, but I figured a workaround for the impatient (like myself).
Edit tensorboard file inside tensorflow/bin (/home/gpu/anaconda3/envs/tensorflow/bin/tensorboard in my case) and replace
from tensorflow.tensorboard.tensorboard import main
by
from tensorflow.tensorboard.main import *
Now tensorboard should run from console as usual.

Tensorboard ships with tensorflow. If you are unable to run using tensorboard command, try below approach. tensorboard.py might have been moved to different directory.
Try searching for tensorboard.py in the tensorbard directory where tensorflow is installed. Go to the path and use following line for visualization:
python tensorboard.py --logdir=path

You should priorly launch
pip install tensorflow.tensorboard

Cannot find dynamic library when running a Python script from Bazel

I am trying to setup CUDA enabled Python & TensorFlow environment on OSx 10.11.6
Everything went quite smoothly. First I installed following:
CUDA - 7.5
cuDNN - 5.1
I ensured that the LD_LIBRARY_PATH and CUDA_HOME are set properly by adding following into my ~/.bash_profile file:
export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH="$CUDA_HOME/lib:$DYLD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:$LD_LIBRARY_PATH"
export PATH="$CUDA_HOME/bin:$PATH"
Then I used Brew to install following:
python - 2.7.12_2
bazel - 0.3.2
protobuf - 3.1.0
Then I used Pip to install CPU only TensorFlow from:
https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.11.0rc0-py2-none-any.whl
I checked out the Magenta project from: https://github.com/tensorflow/magenta
and run all the test using:
bazel test //magenta/...
And all of them have passed.
So far so good. So I decided to give the GPU enabled version of TensorFlow a shot and installed it from:
https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.11.0rc0-py2-none-any.whl
Now all the tests fail with the following error:
import tensorflow as tf
File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 23, in <module>
from tensorflow.python import *
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so, 10): Library not loaded: #rpath/libcudart.7.5.dylib
Referenced from: /usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so
Reason: image not found
So obviously the script run from Bazel has trouble locating the libcudart.7.5.dylib library.
I did try running GPU computations from Python without Bazel and everything seems to be fine.
I also did create a test script and run it using Bazel and it seems that the directory containing libcudart.7.5.dylib library is reachable, however the LD_LIBRARY_PATH is not set.
I searched the documentation and found --action_env and --test_env flags, but none of them actually seems to set the LD_LIBRARY_PATH for the execution.
These are the options from loaded from .bazelrc files.
Inherited 'common' options: --isatty=1 --terminal_columns=80
Inherited 'build' options: --define=allow_oversize_protos=true --copt -funsigned-char -c opt --spawn_strategy=standalone
'run' options: --spawn_strategy=standalone
What is the correct way to let Bazel know about the runtime dependencies?
UPDATE
The trouble seems to be caused by the fact that "env" command is part of the execution chain and it does seem to clear both LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environmental variables. Is there a workaround different than disabling the SIP?

It looks like SIP affects the behavior of how the DYLD_LIBRARY_PATH gets propagated to the child processes. I found a similar problem and another similar problem.
I didn't want to turn the SIP off, so I just created symlinks for the CUDA library into a standard location.
ln -s /usr/local/cuda/lib/* /usr/local/lib
Not sure if this is the best solution, but it does work and it does not require the SIP to be disabled.

Use
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/
before launching bazel. Double check in the directory above if there is such a file.
ls /usr/local/cuda/lib64/libcudart.7.5.dylib
Note that in Macosx the name is different:
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib/
See this answer for more information on SuperUser

The problem is indeed SIP, and the solution is to pass --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib to the bazel command, e.g.:
bazel build -c opt --config=cuda --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib //tensorflow/tools/pip_package:build_pip_package

Theano - Keras - No Module named `pool`

I have installed a bleeding edge theano, and the following packages in following order:
gfortran:
sudo apt-get install gfortran
OpenBLAS:
git clone https://github.com/xianyi/OpenBLAS
cd OpenBLAS
make FC=gfortran
sudo make PREFIX=/usr/local install
Anaconda, first downloaded Anaconda3-2.4.1-Linux-x86_64.sh, and then:
bash Anaconda3-2.4.1-Linux-x86_64.sh
Then, pydot (after updating):
conda update conda
conda update anaconda
conda install pydot
Them I cloned and installed Theano:
git clone git://github.com/Theano/Theano.git
python setup.py develop
I moved from windows to linux and got very happy that theano is installed.
I run a small script, to verify it is indeed working correctly.
from theano import function, config, shared, sandbox
import theano.tensor as T
import numpy as np
import time
vlen = 10*30*768 # assuming 30 cores and 768 threads per core
iters = 1000
rng = np.random.RandomState(22)
x = shared(np.asarray(rng.rand(vlen), config.floatX))
f = function([], T.exp(x))
print (f.maker.fgraph.toposort() )
t0 = time.time()
for i in range (iters):
r = f()
t1 = time.time()
print("Looping " + str(iters) + " times took " + str(t1-t0) + "seconds")
print("Result is " + str(r))
if np.any([isinstance(x.op, T.Elemwise) for x in f.maker.fgraph.toposort()]):
print ("Used the CPU")
else:
print (" Used the GPU")
So, it is working, may be a trivial example to prove a point.
After that, comes keras
git clone https://github.com/fchollet/keras.git
python setup.py install
Then I change to examples directory of keras, and simply type in
python mnist_mlp.py
I get the following error:
Traceback (most recent call last):
File "mnist_mlp.py", line 13, in <module>
from keras.models import Sequential
File "/home/user/anaconda3/lib/python3.5/site-packages/Keras-0.3.1-py3.5.egg/keras/models.py", line 15, in <module>
File "/home/user/anaconda3/lib/python3.5/site-packages/Keras-0.3.1-py3.5.egg/keras/backend/__init__.py", line 46, in <module>
File "/home/user/anaconda3/lib/python3.5/site-packages/Keras-0.3.1-py3.5.egg/keras/backend/theano_backend.py", line 4, in <module>
File "/home/user/anaconda3/lib/python3.5/site-packages/Theano-0.8.0.dev0-py3.5.egg/theano/tensor/signal/downsample.py", line 2, in <module>
import pool
ImportError: No module named 'pool'
Now, what the hell....Am I missing some package?
I think this is not issue of keras but rather problem with theano.
I went ahead and tried a dirty trick, pip install pool, and then rerun the above example, but I get the error:
module 'pool' has no attribute 'max_pool2D'
I can provide a stack trace as well, if needed.
I have struggled a lot in getting theano going, eagerly want to get started... before entire energy drains in the installations,

It seems it cannot compile the file theano/tensor/signal/pool.py there is a issue and fix on github