I want to profile my tensorflow application using tfprof. I have a running tensorflow 1.3 installation where the the tfprof command line tool is missing. I also tried the provided pip packages locally, but there I also can't find tfprof.
Is there a way to compile and link the tfprof command line tool agains my running tensorflow application?
I already git-cloned the tensorflow repository and tried to build it with bazel 0.5.2
$ bazel build --config opt tensorflow/core/profiler/...
WARNING: Output base '/home/USERNAME/.cache/bazel/_bazel_USERNAME/e5cce820cc082410b4fcc604db349066' is on NFS. This may lead to surprising failures and undetermined behavior.
WARNING: Config values are not defined in any .rc file: opt
ERROR: /tmp/tensorflow/tensorflow/core/BUILD:1416:1: no such target '//tensorflow/tools/git:gen/spec.json': target 'gen/spec.json' not declared in package 'tensorflow/tools/git' defined by /tmp/tensorflow/tensorflow/tools/git/BUILD and referenced by '//tensorflow/core:version_info_gen'.
ERROR: /tmp/tensorflow/tensorflow/core/BUILD:1416:1: no such target '//tensorflow/tools/git:gen/head': target 'gen/head' not declared in package 'tensorflow/tools/git' defined by /tmp/tensorflow/tensorflow/tools/git/BUILD and referenced by '//tensorflow/core:version_info_gen'.
ERROR: /tmp/tensorflow/tensorflow/core/BUILD:1416:1: no such target '//tensorflow/tools/git:gen/branch_ref': target 'gen/branch_ref' not declared in package 'tensorflow/tools/git' defined by /tmp/tensorflow/tensorflow/tools/git/BUILD and referenced by '//tensorflow/core:version_info_gen'.
ERROR: Analysis of target '//tensorflow/core/profiler:profiler' failed; build aborted.
INFO: Elapsed time: 167.083s
or just copy the command mentioned here
bazel build --config opt third_party/tensorflow/core/profiler/...
WARNING: Output base '/home/USERNAME/.cache/bazel/_bazel_USERNAME/e5cce820cc082410b4fcc604db349066' is on NFS. This may lead to surprising failures and undetermined behavior.
WARNING: Config values are not defined in any .rc file: opt
ERROR: no targets found beneath 'third_party/tensorflow/core/profiler'.
I think the path is not right. You should use pathtensorflow/core/profiler/, if your current directory is the cloned tensorflow repository.
Run ./configure script from your tensorflow directory to set the environment variables.
Related
I am trying to use XLA compiler from Tensorflow following the example provided at this page:
https://gist.github.com/carlthome/6ae8a570e21069c60708017e3f96c9fd
In short, it downloads a ResNet50 network and compiles it as library.
During execution of bazel build I always end up on the following build error:
error: Could not find include file 'tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td'
include "tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td"
^
external/org_tensorflow/tensorflow/compiler/mlir/xla/ir/hlo_ops.td:22:9: error: Unexpected input at top level
include "tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td"
> ERROR: /home/ubuntu/.cache/bazel/_bazel_ubuntu/e5cce820cc082410b4fcc604db349066/external/org_tensorflow/tensorflow/compiler/mlir/xla/BUILD:465:1: Executing genrule #org_tensorflow//tensorflow/compiler/mlir/xla:operator_writer_inc failed (Exit 1)
[6,144 / 7,191] 3 actions running
#org_tensorflow//tensorflow/compiler/xla/client:global_data; 4s local
#org_tensorflow//tensorflow/core/kernels/tensor_forest:resources; 1s local
...//tensorflow/core/kernels:eigen_contraction_kernel_with_mkl; 1s local
external/org_tensorflow/tensorflow/compiler/mlir/xla/ir/hlo_ops.td:22:9: error: Could not find include file 'tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td'
include "tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td"
^
external/org_tensorflow/tensorflow/compiler/mlir/xla/ir/hlo_ops.td:22:9: error: Unexpected input at top level
include "tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td"
^
[6,144 / 7,191] 3 actions running
#org_tensorflow//tensorflow/compiler/xla/client:global_data; 4s local
#org_tensorflow//tensorflow/core/kernels/tensor_forest:resources; 1s local
...//tensorflow/core/kernels:eigen_contraction_kernel_with_mkl; 1s local
Target #org_tensorflow//:graph failed to build
[6,147 / 7,191] checking cached actions
Use --verbose_failures to see the command lines of failed build steps.
[6,147 / 7,191] checking cached actions
INFO: Elapsed time: 7903.567s, Critical Path: 204.12s
[6,147 / 7,191] checking cached actions
INFO: 5961 processes: 5961 local.
[6,147 / 7,191] checking cached actions
FAILED: Build did NOT complete successfully
FAILED: Build did NOT complete successfully
So, it does not find the hlo_ops_base.td file, which of course is present in the path (I checked it)
The first time I have tried this, it worked like a charm.
Afterwards I have executed it again on different machines (also perfect clean VMs on different platforms), but always had the same issue.
I am using
bazel 1.1.0,
tensorflow 1.14 (cpu),
protobuf 3.0.0,
python 2.7
Does anyone have any clue on how to solve this? I have tried to search it online and it seems no one else is having this issue...
Thanks,
Matteo
My BUILD file is pretty simple, just
load("#protobuf_bzl//:protobuf.bzl", "py_proto_library")
py_proto_library(
name = "struct_py_pb2",
srcs = ["struct.proto"],
)
But bazel gives a bunch of baffling error messages like:
$ bazel build google/genomics/v1:all
ERROR: thomaswc//v1/BUILD:22:1: no such package '': BUILD file not found on package path and referenced by '//v1:struct_py_pb2'
ERROR: Analysis of target '//v1:struct_py_pb2' failed; build aborted: no such package '': BUILD file not found on package path
INFO: Elapsed time: 0.581s
FAILED: Build did NOT complete successfully (2 packages loaded)
currently loading: #protobuf_bzl//
I see other projects on github using bazel and py_proto_library, though, so I know it must be possible. Is there some WORKSPACE or .bzl magic that I need?
After a bunch of digging, I found a work-around: the default values of default_runtime and protoc are screwed up, so you need to override them:
py_proto_library(
name = "struct_py_pb2",
srcs = ["struct.proto"],
default_runtime = "#com_google_protobuf//:protobuf_python",
protoc = "#com_google_protobuf//:protoc",
)
I am trying to setup CUDA enabled Python & TensorFlow environment on OSx 10.11.6
Everything went quite smoothly. First I installed following:
CUDA - 7.5
cuDNN - 5.1
I ensured that the LD_LIBRARY_PATH and CUDA_HOME are set properly by adding following into my ~/.bash_profile file:
export CUDA_HOME=/usr/local/cuda
export DYLD_LIBRARY_PATH="$CUDA_HOME/lib:$DYLD_LIBRARY_PATH"
export LD_LIBRARY_PATH="$CUDA_HOME/lib:$LD_LIBRARY_PATH"
export PATH="$CUDA_HOME/bin:$PATH"
Then I used Brew to install following:
python - 2.7.12_2
bazel - 0.3.2
protobuf - 3.1.0
Then I used Pip to install CPU only TensorFlow from:
https://storage.googleapis.com/tensorflow/mac/cpu/tensorflow-0.11.0rc0-py2-none-any.whl
I checked out the Magenta project from: https://github.com/tensorflow/magenta
and run all the test using:
bazel test //magenta/...
And all of them have passed.
So far so good. So I decided to give the GPU enabled version of TensorFlow a shot and installed it from:
https://storage.googleapis.com/tensorflow/mac/gpu/tensorflow-0.11.0rc0-py2-none-any.whl
Now all the tests fail with the following error:
import tensorflow as tf
File "/usr/local/lib/python2.7/site-packages/tensorflow/__init__.py", line 23, in <module>
from tensorflow.python import *
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/__init__.py", line 49, in <module>
from tensorflow.python import pywrap_tensorflow
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 28, in <module>
_pywrap_tensorflow = swig_import_helper()
File "/usr/local/lib/python2.7/site-packages/tensorflow/python/pywrap_tensorflow.py", line 24, in swig_import_helper
_mod = imp.load_module('_pywrap_tensorflow', fp, pathname, description)
ImportError: dlopen(/usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so, 10): Library not loaded: #rpath/libcudart.7.5.dylib
Referenced from: /usr/local/lib/python2.7/site-packages/tensorflow/python/_pywrap_tensorflow.so
Reason: image not found
So obviously the script run from Bazel has trouble locating the libcudart.7.5.dylib library.
I did try running GPU computations from Python without Bazel and everything seems to be fine.
I also did create a test script and run it using Bazel and it seems that the directory containing libcudart.7.5.dylib library is reachable, however the LD_LIBRARY_PATH is not set.
I searched the documentation and found --action_env and --test_env flags, but none of them actually seems to set the LD_LIBRARY_PATH for the execution.
These are the options from loaded from .bazelrc files.
Inherited 'common' options: --isatty=1 --terminal_columns=80
Inherited 'build' options: --define=allow_oversize_protos=true --copt -funsigned-char -c opt --spawn_strategy=standalone
'run' options: --spawn_strategy=standalone
What is the correct way to let Bazel know about the runtime dependencies?
UPDATE
The trouble seems to be caused by the fact that "env" command is part of the execution chain and it does seem to clear both LD_LIBRARY_PATH and DYLD_LIBRARY_PATH environmental variables. Is there a workaround different than disabling the SIP?
It looks like SIP affects the behavior of how the DYLD_LIBRARY_PATH gets propagated to the child processes. I found a similar problem and another similar problem.
I didn't want to turn the SIP off, so I just created symlinks for the CUDA library into a standard location.
ln -s /usr/local/cuda/lib/* /usr/local/lib
Not sure if this is the best solution, but it does work and it does not require the SIP to be disabled.
Use
export LD_LIBRARY_PATH=/usr/local/cuda/lib64/
before launching bazel. Double check in the directory above if there is such a file.
ls /usr/local/cuda/lib64/libcudart.7.5.dylib
Note that in Macosx the name is different:
export DYLD_LIBRARY_PATH=/usr/local/cuda/lib/
See this answer for more information on SuperUser
The problem is indeed SIP, and the solution is to pass --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib to the bazel command, e.g.:
bazel build -c opt --config=cuda --action_env DYLD_LIBRARY_PATH=$CUDA_HOME/lib //tensorflow/tools/pip_package:build_pip_package
I am trying to install boost.numpy in y Ubuntu 16.04. I tried these commnads to install boost.numpy
**git clone https://github.com/ndarray/Boost.NumPy.git
cd Boost.NumPy && mkdir build && cd build
cmake -DPYTHON_LIBRARY=$HOME/anaconda3/lib/libpython3.5m.so ../**
After doing cmake i am facing this error:
Detected architecture 'x86_64'
-- Using Python3
CMake Error at /usr/share/cmake-3.5/Modules/FindBoost.cmake:1677 (message):
Unable to find the requested Boost libraries.
Boost version: 1.59.0
Boost include path: /home/sumit/Documents/Software/boost_1_59_0
Could not find the following static Boost libraries:
boost_python3
No Boost libraries were found. You may need to set BOOST_LIBRARYDIR to the
directory containing Boost libraries or BOOST_ROOT to the location of
Boost.
Call Stack (most recent call first):
CMakeLists.txt:48 (find_package)
Boost Paths:
Include : /home/sumit/Documents/Software/boost_1_59_0
**Libraries**: /home/sumit/Documents/Software/boost_1_59_0/libs
Configuring incomplete, errors occurred!
See also "/home/sumit/Documents/Software/Boost.NumPy/build/CMakeFiles/CMakeOutput.log".
Previously it was not able to find the boost libraries sp i manualy changed the CmakeList.txt library path with the path of boost_1_59_0 lib path. This path comes up in library option when i do cmake. But still boost_python3 is missing. I am new into this what i tried is just the result of google.
Please help.
On Ubuntu the library names for boost are:
libboost_python, libboost_python-py35, or libboost_python-py27
This means that in cmake you'll need to refer to them as python-py35 instead of python3. Alternatively, if you don't control the CMakeLists.txt you can create a symlink:
/usr/lib/x86_64-linux-gnu/libboost_python-py35.so -> /usr/lib/x86_64-linux-gnu/libboost_python3.so
In my CMakeLists.txt file I have the following:
if(UNIX)
set( BOOST_PYTHONLIB python-py35)
else()
set( BOOST_PYTHONLIB python3)
endif()
find_package (Boost 1.58 REQUIRED COMPONENTS
coroutine
context
filesystem
program_options
system
thread
${BOOST_PYTHONLIB}
chrono
)
Simple answer for this is that wherever boost_python3 is specified,
you replace it with boost_python-py35.
I tried this when I was setting up caffe for python 3.5 . In the Makefile.config file, I only made the above changes and it worked fine for me.
The following procedure fails. Am I missing something?
Install various Ubuntu packages (prerequisites for compilation)
Get http://downloads.sourceforge.net/wxpython/wxPython-src-3.0.1.1.tar.bz2
Uncompress to wxPython-src-3.0.1.1/
Create new virtualenv called test
Activate test virtualenv
In terminal, from wxPython-src-3.0.1.1/:
./configure --prefix=/home/username/.virtualenvs/test --with-gtk2 --enable-unicode --with-opengl
#lots of output, confirms "Configured wxWidgets 3.0.1 for `x86_64-unknown-linux-gnu'"
make install
#lots of output, confirms:
# The installation of wxWidgets is finished. On certain
# platforms (e.g. Linux) you'll now have to run ldconfig
# if you installed a shared library and also modify the
# LD_LIBRARY_PATH (or equivalent) environment variable.
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:~/.virtualenvs/test/lib
# don't run ldconfig since that is a system tool, not appropriate for virtualenv?
cd wxPython
python setup.py install
# lots of output, starting:
# WARNING: WXWIN not set in environment. Assuming '..'
# Found wx-config: /home/username/.virtualenvs/test/bin/wx-config
# Using flags: --toolkit=gtk2 --unicode=yes --version=3.0
# Preparing CORE...
# Preparing STC...
# Preparing GLCANVAS...
# Preparing GIZMOS...
# running install
# etc
The final command fails with:
src/gtk/_core_wrap.cpp:20407:7: note: ‘arg3’ was declared here
int arg3 ;
^
src/gtk/_core_wrap.cpp: In function ‘PyObject* _wrap_Image_SetAlphaBuffer(PyObject*, PyObject*, PyObject*)’:
src/gtk/_core_wrap.cpp:3747:13: warning: ‘arg3’ may be used uninitialized in this function [-Wmaybe-uninitialized]
if (ALPHASIZE != self->GetWidth() * self->GetHeight()) {
^
src/gtk/_core_wrap.cpp:20474:7: note: ‘arg3’ was declared here
int arg3 ;
^
cc1plus: some warnings being treated as errors
error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
I used python setup.py install &> ~/error.txt to pass on the error messages to knowledgeable colleague who identified that C compilation was using the -Werror=format-security flag. This version of wxPython (and maybe others) cannot compile with that flag.
My $CPPFLAGS and $CFLAGS environment variables were empty. It turns out that this flag is triggered by hardening-wrapper.
So, I overrode the flag by invoking the final step as follows, and wxPython was installed successfully:
CFLAGS=-Wno-error=format-security CPPFLAGS=-Wno-error=format-security python setup.py install