trax
New to trax, I'm trying to run it locally (macOS 12.1, Apple Silicon ARM M1 processor, 8GB RAM, Anaconda), but I'm running into some issues.
In an environment with python 3.8.5, I installed trax running pip3 install trax==1.3.9 inside an (Anaconda) conda environment. Later, I ran into issues when trying to import trax layers in my code with from trax import layers as tl
RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
I rushed to start a new conda environment with python 3.10. Then ran into issue just trying to install trax:
macos ERROR: Could not find a version that satisfies the requirement tensorflow-text (from trax) (from versions: none)
ERROR: No matching distribution found for tensorflow-text
I then created a new environment with python 3.9. Installation went fine, but then ran into the same error importing layers:
RuntimeError: This version of jaxlib was built using AVX instructions, which your CPU and/or operating system do not support. You may be able work around this issue by building jaxlib from source.
installing jaxlib, jax before trax
I then tried building jaxlib from source following these instructions and got this error:
Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/my path/jax/dist --cpu=x86_64
ERROR: bazel does not currently work properly from paths containing spaces (/my path/jax).
b''
Traceback (most recent call last):
File "/my path/jax/build/build.py", line 524, in <module>
main()
File "/my path/jax/build/build.py", line 519, in main
shell(command)
File "/my path/jax/build/build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/my path/jax/dist', '--cpu=x86_64']' returned non-zero exit status 36.
Emphasis on the part that I initially missed that says: bazel does not currently work properly from paths containing spaces (/my path/jax).
I moved my /my path/ directory to a path without spaces /mypath/. Deleted and redownloaded jax directory. Still, the build (for CPU) with python build/build.py failed:
ERROR: /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/external/llvm-project/llvm/BUILD.bazel:610:11: Generating code from table: include/llvm/IR/Intrinsics.td #llvm-project//llvm:intrinsic_XCore_gen__gen_intrinsic_enums__intrinsic_prefix_xcore_genrule failed: (Illegal instruction): bash failed: error executing command
(cd /private/var/tmp/_bazel_a/2caf512c3c5e3f3f654bc58b48b8333a/execroot/__main__ && \
exec env - \
PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h')
Execution platform: #local_execution_config_platform//:platform
/bin/bash: line 1: 11140 Illegal instruction: 4 bazel-out/darwin-opt/bin/external/llvm-project/llvm/llvm-tblgen -I external/llvm-project/llvm/include -I external/llvm-project/clang/include -I $(dirname external/llvm-project/llvm/include/llvm/IR/Intrinsics.td) -gen-intrinsic-enums -intrinsic-prefix=xcore external/llvm-project/llvm/include/llvm/IR/Intrinsics.td -o bazel-out/darwin-opt/bin/external/llvm-project/llvm/include/llvm/IR/IntrinsicsXCore.h
Target //build:build_wheel failed to build
INFO: Elapsed time: 620.950s, Critical Path: 45.35s
INFO: 589 processes: 132 internal, 457 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
File "/mypath/jax/build/build.py", line 524, in <module>
main()
File "/mypath/jax/build/build.py", line 519, in main
shell(command)
File "/mypath/jax/build/build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
Tried again a couple of times (deleted and redownloaded jax directory) and the same line 528 in .../lib/python3.9/subprocess.py seemed to cause the issue but the output, while mostly the above, sometimes was slightly different. Making me suspect an issue with memory, given that I (admittedly) had not restarted my machine in weeks and it was starting to unresponsively slow.
I updated my XCode command line tools to version 12.2 (pretty sure).
I restarted my (8GB) machine. Deleted and redownloaded jax directory. I installed bazel version 5.0.0 with homebrew in case that would help. I was a bit concerned that it kept downloading an x86 version for my ARM processor. bazel installation went fine.
Started again from these instructions. The jaxlib build, though, made it clear that it wanted an earlier (4.2.1) version of bazel and downloaded it, as before:
b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
Once again, a different error in the same line 528. Showing full run now:
b'\x1b[31mERROR: The project you\'re trying to build requires Bazel 4.2.1 (specified in /mypath/jax/.bazelversion), but it wasn\'t found in /opt/homebrew/Cellar/bazel/5.0.0/libexec/bin.\x1b[0m\n\nBazel binaries for all official releases can be downloaded from here:\n https://github.com/bazelbuild/bazel/releases\n\nYou can download the required version directly using this command:\n (cd "/opt/homebrew/Cellar/bazel/5.0.0/libexec/bin" && curl -fLO https://releases.bazel.build/4.2.1/release/bazel-4.2.1-darwin-x86_64 && chmod +x bazel-4.2.1-darwin-x86_64)\n'
Downloading bazel from: https://github.com/bazelbuild/bazel/releases/download/4.2.1/bazel-4.2.1-darwin-x86_64
bazel-4.2.1-darwin-x86_64 [########################################] 100%
Bazel binary path: ./bazel-4.2.1-darwin-x86_64
Bazel version: 4.2.1
Python binary path: /myuserpath/opt/anaconda3/envs/mytraxenv/bin/python
Python version: 3.9
NumPy version: 1.21.2
MKL-DNN enabled: yes
Target CPU: x86_64
Target CPU features: release
CUDA enabled: no
TPU enabled: no
ROCm enabled: no
Building XLA and installing it in the jaxlib source tree...
./bazel-4.2.1-darwin-x86_64 run --verbose_failures=true --config=avx_posix --config=mkl_open_source_only :build_wheel -- --output_path=/mypath/jax/dist --cpu=x86_64
INFO: Options provided by the client:
Inherited 'common' options: --isatty=0 --terminal_columns=80
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'run' from /mypath/jax/.bazelrc:
Inherited 'build' options: --apple_platform_type=macos --macos_minimum_os=10.9 --announce_rc --define open_source_build=true --spawn_strategy=standalone --enable_platform_specific_config --define=no_aws_support=true --define=no_gcp_support=true --define=no_hdfs_support=true --define=no_kafka_support=true --define=no_ignite_support=true --define=grpc_no_ares=true -c opt --config=short_logs --copt=-DMLIR_PYTHON_PACKAGE_PREFIX=jaxlib.mlir.
INFO: Reading rc options for 'run' from /mypath/jax/.jax_configure.bazelrc:
Inherited 'build' options: --strategy=Genrule=standalone --repo_env PYTHON_BIN_PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --action_env=PYENV_ROOT --python_path=/myuserpath/opt/anaconda3/envs/mytraxenv/bin/python --distinct_host_configuration=false
INFO: Found applicable config definition build:short_logs in file /mypath/jax/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:avx_posix in file /mypath/jax/.bazelrc: --copt=-mavx --host_copt=-mavx
INFO: Found applicable config definition build:mkl_open_source_only in file /mypath/jax/.bazelrc: --define=tensorflow_mkldnn_contraction_kernel=1
INFO: Found applicable config definition build:macos in file /mypath/jax/.bazelrc: --config=posix
INFO: Found applicable config definition build:posix in file /mypath/jax/.bazelrc: --copt=-fvisibility=hidden --copt=-Wno-sign-compare --cxxopt=-std=c++14 --host_cxxopt=-std=c++14
Loading:
Loading: 0 packages loaded
Analyzing: target //build:build_wheel (1 packages loaded, 0 targets configured)
Analyzing: target //build:build_wheel (8 packages loaded, 286 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (10 packages loaded, 4271 targets configured)
Analyzing: target //build:build_wheel (11 packages loaded, 4494 targets configured)
INFO: Analyzed target //build:build_wheel (11 packages loaded, 7417 targets configured).
INFO: Found 1 target...
[0 / 3] [Prepa] BazelWorkspaceStatusAction stable-status.txt
[21 / 233] Compiling src/google/protobuf/generated_enum_util.cc; 2s local ... (6 actions, 3 running)
[28 / 233] Compiling src/google/protobuf/extension_set.cc; 4s local ... (8 actions, 7 running)
[114 / 523] Compiling src/google/protobuf/generated_message_util.cc; 2s local ... (8 actions running)
[140 / 594] Compiling platform/c++11/src/nsync_semaphore_mutex.cc; 0s local ... (8 actions, 7 running)
[237 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 0s local ... (8 actions running)
[252 / 597] Compiling src/google/protobuf/util/message_differencer.cc; 4s local ... (8 actions running)
[272 / 597] Compiling src/google/protobuf/descriptor.pb.cc; 8s local ... (8 actions running)
[296 / 597] Compiling src/google/protobuf/descriptor.cc; 12s local ... (8 actions running)
[315 / 597] Compiling src/google/protobuf/descriptor_database.cc; 2s local ... (5 actions running)
[487 / 1,963] Compiling src/compiler/python_generator.cc; 2s local ... (8 actions running)
[579 / 3,117] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 0s local ... (8 actions, 7 running)
[619 / 3,173] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 8s local ... (8 actions running)
[687 / 3,270] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv3d.cc; 17s local ... (8 actions running)
[774 / 3,464] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 27s local ... (8 actions running)
[1,204 / 4,860] Compiling tensorflow/compiler/xla/service/cpu/runtime_single_threaded_conv2d.cc; 39s local ... (8 actions running)
[1,255 / 4,916] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 52s local ... (8 actions running)
[1,340 / 5,042] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 68s local ... (8 actions running)
[1,456 / 5,156] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 86s local ... (8 actions, 7 running)
[1,661 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 107s local ... (8 actions, 7 running)
[1,688 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 132s local ... (8 actions, 7 running)
[1,721 / 5,704] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 160s local ... (8 actions, 7 running)
[2,106 / 6,584] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 192s local ... (8 actions, 7 running)
[2,342 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 231s local ... (8 actions, 7 running)
[2,378 / 7,067] Compiling tensorflow/compiler/xla/service/cpu/runtime_matmul.cc; 274s local ... (8 actions, 7 running)
[2,422 / 7,067] Compiling src/cpu/rnn/ref_rnn.cpp; 75s local ... (8 actions, 7 running)
[2,452 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 54s local ... (8 actions, 7 running)
[2,500 / 7,067] Compiling src/cpu/x64/gemm/f32/jit_avx512_core_f32_copy_at_kern_part1_autogen.cpp; 119s local ... (8 actions, 7 running)
[2,577 / 7,067] Compiling src/common/memory_zero_pad.cpp; 74s local ... (8 actions, 7 running)
ERROR: /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/external/org_tensorflow/tensorflow/compiler/xla/BUILD:69:17: ProtoCompile external/org_tensorflow/tensorflow/compiler/xla/xla.pb.h failed: (Illegal instruction): protoc failed: error executing command
(cd /private/var/tmp/_bazel_a/f5e9a3325f07a1f02c52d821857db47c/execroot/__main__ && \
exec env - \
PATH=/myuserpath/opt/anaconda3/envs/mytraxenv/bin:/myuserpath/opt/anaconda3/condabin:/opt/homebrew/bin:/opt/homebrew/sbin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/Library/Apple/usr/bin \
bazel-out/darwin-opt/bin/external/com_google_protobuf/protoc '--cpp_out=bazel-out/darwin-opt/bin/external/org_tensorflow' -Iexternal/org_tensorflow -Ibazel-out/darwin-opt/bin/external/org_tensorflow -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/any_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/source_context_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/type_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/api_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/descriptor_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/compiler_plugin_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/duration_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/empty_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/field_mask_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/struct_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/timestamp_proto -Ibazel-out/darwin-opt/bin/external/com_google_protobuf/_virtual_imports/wrappers_proto external/org_tensorflow/tensorflow/compiler/xla/xla.proto)
Execution platform: #local_execution_config_platform//:platform
Target //build:build_wheel failed to build
INFO: Elapsed time: 631.189s, Critical Path: 283.35s
INFO: 2563 processes: 935 internal, 1628 local.
FAILED: Build did NOT complete successfully
ERROR: Build failed. Not running target
FAILED: Build did NOT complete successfully
b''
Traceback (most recent call last):
File "/mypath/jax/build/build.py", line 524, in <module>
main()
File "/mypath/jax/build/build.py", line 519, in main
shell(command)
File "/mypath/jax/build/build.py", line 53, in shell
output = subprocess.check_output(cmd)
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 424, in check_output
return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
File "/myuserpath/opt/anaconda3/envs/mytraxenv/lib/python3.9/subprocess.py", line 528, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['./bazel-4.2.1-darwin-x86_64', 'run', '--verbose_failures=true', '--config=avx_posix', '--config=mkl_open_source_only', ':build_wheel', '--', '--output_path=/mypath/jax/dist', '--cpu=x86_64']' returned non-zero exit status 1.
I switched away from trying to build jaxlib. In late October 2021, a M1-compatible jaxlib wheel was released, so I tried:
pip install -U pip
pip install -U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl
but got
ERROR: jaxlib-0.1.74-cp39-none-macosx_11_0_arm64.whl is not a supported wheel on this platform.
Tried upgrading python from 3.9 to 3.10, but got the same message.
Debugged by, in python, running
import platform
print(platform.machine())
which showed that my python is still running on x86 architechture, since "Anaconda doesn't yet provide packages for M1/ARM".
[see successful (so far) jaxlib installation in own answer below]
back to trax
After successfully installing jaxlib and jax, when trying to install trax with (miniforge’s)conda install trax I get:
Collecting package metadata (current_repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
Collecting package metadata (repodata.json): done
Solving environment: failed with initial frozen solve. Retrying with flexible solve.
PackagesNotFoundError: The following packages are not available from current channels:
- trax
Current channels:
- https://conda.anaconda.org/conda-forge/osx-arm64
- https://conda.anaconda.org/conda-forge/noarch
To search for alternate channels that may provide the conda package you're
looking for, navigate to
https://anaconda.org
and use the search bar at the top of the page.
I tried installing as trax itself suggests but ran into an error building hp5y:
ERROR: Failed building wheel for h5py
ERROR: Could not build wheels for h5py, which is required to install pyproject.toml-based projects
Looking now into this error and also looking into doing it with miniforge again but using Anaconda's channel to install trax. I'll update when I get to it.
Any pointers or anyone who managed to install trax on an M1 Mac it by any different means?
jaxlib + jax
After someone claimed success using miniforge, I read this and watched this to clarify using Anaconda and miniforge together.
I installed miniforge with Apple's arm64 : Apple Silicon method. For some reason when I ran miniforge's conda init it set up the initialization code in ~/.bash_profile even though I'm using the zsh shell. I tried putting the code manually instead in ~/.zprofile but it wouldn't load on interactive shells, so I just ended up putting it where Anaconda had put its initialization code, in ~/zshrc.
This made miniforge the default manager. Following the very useful video above, I created a ~/.start_anaconda.sh script so I can use Anaconda as an alternative.
With miniforge I
created a new conda environment mytraxenv with conda create -n mytraxenv python=3 which has python 3.10.2 at the moment
activated the environment: conda activate mytraxenv
ran conda install numpy and conda install six to ensure numpy. six and wheel (installed by one of the previous two) were installed in my mytraxenv environment
tried again, with a slightly updated release (from here):
pip install -U pip
pip install -U https://storage.googleapis.com/jax-releases/mac/jaxlib-0.1.75-cp310-none-macosx_11_0_arm64.whl
This worked in installing jaxlib!
Then, I followed these instructions to install jax:
pip install --upgrade pip
pip install --upgrade "jax[cpu]"
That worked as well. Note that when running import jax in python it currently warns:
/mytraxenv/lib/python3.10/site-packages/jax/_src/lib/__init__.py:32: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.
warnings.warn("JAX on Mac ARM machines is experimental and minimally tested. "
trax
No success yet installing trax.
Anaconda Source build
As of Mon, 7th of Feb 2022, installing using an Anaconda installation of python fails as Anaconda does not provide arm wheels for python.
When they do provide them, building from source should work automatically. If it does not, clone the repository, and use the following:
python build/build.py --target_cpu darwin_arm64
If bazel does not build and you are using homebrew consider installing it, and removing the downloaded bazel files.
brew install bazel
rm -rf bazel-*
Arm64 Python wheels
I recommend downloading arm64 wheels from the official CPython website.
The executable will be in /usr/local/bin/python3, and so will pip3.
This should allow you to install jax using
/usr/local/bin/pip3 install --upgrade 'jax[cpu]'
To have access to the proper python with the command line/path, add the folder to the path.
export PATH=/usr/share/bin:$PATH
Virtual Env
I recommend against the above method, and instead to use to use virtual-env from here https://virtualenv.pypa.io, to create virtual environments as with anaconda, and install jax in a virtual environment as above.
Poetry
Poetry is an even better option since it uses virtual-env, and allows you to build your projects and list dependencies just as anaconda.
There is a minor caveat as I am writing this, installing scipy when defining dependencies with poetry results in an odd error, installing numpy and scipy manually using
poetry shell # to go to virtual environment for the project
pip install --upgrade 'jax[cpu]' jaxlib scipy numpy
Trax
Since anaconda is not option, installing hdf5 is a bit of a hassle, but it can be done with brew/homebrew. Assuming you have activated the correct venv:
brew install hdf5
export HDF5_DIR=/opt/homebrew/Cellar/hdf5/1.12.1
pip install trax
Unfortunately this results in the following error:
import trax
/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/jax/_src/lib/__init__.py:32: UserWarning: JAX on Mac ARM machines is experimental and minimally tested. Please see https://github.com/google/jax/issues/5501 in the event of problems.
warnings.warn("JAX on Mac ARM machines is experimental and minimally tested. "
***************************************************************
Failed to import TensorFlow. Please note that TensorFlow is not installed by default when you install TFDS. This allow you to choose to install either `tf-nightly` or `tensorflow`. Please install the most recent version of TensorFlow, by following instructions at https://tensorflow.org/install.
***************************************************************
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/__init__.py", line 18, in <module>
from trax import layers
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/__init__.py", line 23, in <module>
from trax.layers.activation_fns import *
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/layers/activation_fns.py", line 26, in <module>
from trax import math
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/__init__.py", line 18, in <module>
from trax.math import jax as jax_math
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/trax/math/jax.py", line 28, in <module>
import tensorflow_datasets as tfds
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/__init__.py", line 43, in <module>
from tensorflow_datasets.core import tf_compat
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/__init__.py", line 22, in <module>
tf_compat.ensure_tf_install()
File "/Users/quinn/Library/Caches/pypoetry/virtualenvs/no-xNvLksOy-py3.9/lib/python3.9/site-packages/tensorflow_datasets/core/tf_compat.py", line 48, in ensure_tf_install
import tensorflow as tf # pylint: disable=import-outside-toplevel
ModuleNotFoundError: No module named 'tensorflow'
Im trying to import the latest rc2 version of Tensorflow (2.2.0rc2 at this date) in Google Colab, but cant do it when installed from my setup.py install script.
When i install Tensorflow manually using !pip install tensorflow==2.2.0rc2 from a Colab cell, everything is ok and im able to import Tensorflow.
The next is how i have my dependencies installation setup in Google Colab:
# Executes the cell in bash mode
%%bash
if [ ! -d "/content/deep-deblurring/" ];
then
git clone https://github.com/ElPapi42/deep-deblurring;
cd deep-deblurring/
else
cd deep-deblurring/;
git pull;
fi;
git checkout development
cd ..
pip uninstall -y tensorflow tensor2tensor tensorboard tensorboardcolab tensorflow-datasets tensorflow-estimator tensorflow-gan tensorflow-hub tensorflow-metadata tensorflow-privacy tensorflow-probability
pip install colab-env
pip install --upgrade grpcio
cd deep-deblurring/
python setup.py install
cd ..
The next is my setup.py file:
#!/usr/bin/python
# coding=utf-8
"""Setup and install the package and all the dependencies."""
from setuptools import setup, find_packages
with open('requirements.txt') as pro:
INSTALL_REQUIRES = pro.read().split('\n')
setup(
author='Whitman Bohorquez, Mo Rebaie',
author_email='whitman-2#hotmail.com',
name='deblurrer',
license='MIT',
description='Image Deblurring using Deep Learning Architecture',
version='1.0.0',
url='',
packages=find_packages(),
include_package_data=True,
python_requires='>=3.6',
install_requires=INSTALL_REQUIRES,
classifiers=[
'Development Status :: Alpha',
'Programming Language :: Python',
'Programming Language :: Python :: 3.6',
'Intended Audience :: Developers',
],
)
The next is the requirements.txt on the repository:
grpcio == 1.27.2
kaggle
numpy
tensorflow >= 2.2.0rc2
pandas
Actually, Google Colab ships with Tensorflow 2.2.0rc1, but i want the rc2. when i execute:
import tensorflow as tf
print(tf.__version__)
before executing the setup.py installation script, the import works normally. But after the installation using setup.py is done, the error ImportError: No module named 'tensorflow' is throw.
I checked the tensorflow installation before and after the python setup.py install execution and everything seems to be ok, with tensorflow 2.2.0rc1 before installation and 2.2.0rc2 after installation.
as i mention first, when iinstall tensorflow manually using !pip install tensorflow==2.2.0rc2 the import works as espected, so the problem must be around setup.py file or requirements, something like that, but im not seeing it.
Hope your help guys!
PD: this project setup was working the last week on friday, but today i try to run it, and suddenly stops working with no apparent reason.
PD2: https://colab.research.google.com/drive/1Qv8h4ceEtDTq5lvt1uKJG8dk53_bUqBZ this is a Colab Notebook i share with you, this setups the code for reproduce the issue.
PD3: this is the full error traceback throw in Google Colab while importing tensorflow:
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/usr/lib/python3.6/importlib/_bootstrap.py in _find_spec(name, path, target)
AttributeError: '_TensorflowImportHook' object has no attribute 'find_spec'
During handling of the above exception, another exception occurred:
ImportError Traceback (most recent call last)
2 frames
<ipython-input-7-69e5d056d1fc> in <module>()
----> 1 import tensorflow as tf
2
3 tf.__version__
/usr/local/lib/python3.6/dist-packages/google/colab/_import_hooks/_tensorflow.py in find_module(self, fullname, path)
26 if fullname != 'tensorflow':
27 return None
---> 28 self.module_info = imp.find_module(fullname.split('.')[-1], path)
29 return self
30
/usr/lib/python3.6/imp.py in find_module(name, path)
295 break # Break out of outer loop when breaking out of inner loop.
296 else:
--> 297 raise ImportError(_ERR_MSG.format(name), name=name)
298
299 encoding = None
ImportError: No module named 'tensorflow'
---------------------------------------------------------------------------
NOTE: If your import is failing due to a missing package, you can
manually install dependencies using either !pip or !apt.
To view examples of installing some common dependencies, click the
"Open Examples" button below.
---------------------------------------------------------------------------
There's nothing wrong with tensorflow, but rather with Colab's _TensorflowImportHook missing find_spec impl so it will raise if tensorflow is installed as egg dir. Since the hook doesn't do anything useful besides issuing a notification to update tensorflow to 2.0 and is scheduled for removal anyway, an easy fix would be purging it from sys.meta_path somewhere at the start of the notebook:
[1] import sys
sys.meta_path[:] = [hook for hook in sys.meta_path if not h.__class__.__name__ == '_TensorflowImportHook']
[2] import tensorflow as tf
print(tf.__version__)
I found a work around, but this is not the solution to this problem by far, so this will not be accepted as solution, but will help people in same trouble to keep going with their work:
Install your requirements manually before installing your custom package, in my case, this is pip install -r "/content/deep-deblurring/requirements.txt":
# Executes the cell in bash mode
%%bash
if [ ! -d "/content/deep-deblurring/" ];
then
git clone https://github.com/ElPapi42/deep-deblurring;
cd deep-deblurring/
else
cd deep-deblurring/;
git pull;
fi;
git checkout development
cd ..
pip uninstall -y tensorflow tensor2tensor tensorboard tensorboardcolab tensorflow-datasets tensorflow-estimator tensorflow-gan tensorflow-hub tensorflow-metadata tensorflow-privacy tensorflow-probability
pip install colab-env
pip install --upgrade grpcio
pip install -r "/content/deep-deblurring/requirements.txt"
cd deep-deblurring/
python setup.py install
cd ..
Currently this fix the import issue, but this is not a clean solution, lets hope a better explanation later!
I run the following container to test my opencv-python environment on Windows Server 2016.
docker run -i --rm python:2.7-windowsservercore
(the python:2.7-windowsservercore image is an official python image which is base on microsoft/windowsservercore)
and in the interacive console, I got the opencv-python installed by:
pip install opencv-python
after this finished, I test the opencv-python package by:
python -c "import cv2"
but I got the following error:
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Python\lib\site-packages\cv2\__init__.py", line 9, in <module>
from .cv2 import *
ImportError: DLL load failed: The specified module could not be found.
then I installed python-2.7.13 on the host(not inside container), input the same pip install commands, and test python -c "import cv2" again, it won't report any error. I wonder if there are some DLL files are missing in the container? What package or software should I install in the container to fix this error?
Based on the accepted answer to this question I would put my money on you not having the required MSVC runtime. Assuming it was compiled with Visual Studio 2015 the runtime can be downloaded from Visual C++ Redistributable for Visual Studio 2015
I found the solution in this place.
Copy the following files from your host into your container's \System32 folder:
msacm32.dll
avifil32.dll
avicap32.dll
msvfw32.dll
I'm working on a project involving network messaging queues (msgpack, zmq, ...) on a RHEL 6.3 (x86_64) system. I was installing the most recent packages of glib, gevent, pygobject, pygtk, and such in order to get pylab / matplotlib to work (which hasn't been successful either).
After giving up I went back to my code and somehow I had managed to wreck my hdf5 / h5py installation - h5py can't find libhdf5.so.7 on import. I immediately reinstalled hdf5-1.8.9 in /usr/local/hdf5 on RHEL 6.3 (x86_64) as follows:
./configure --prefix=/usr/local/hdf5
make
make check
sudo make install
make check install
which seemed to work just fine. Then I went to reinstall h5py (in python 2.7.3):
python2.7 setup.py build --hdf5=/usr/local/hdf5/
python2.7 setup.py test # optional
# sudo python2.7 setup.py install
which fails to import the _errors file in the tests, like so:
======================================================================
ERROR: _hl.tests.test_attrs_data (unittest.loader.ModuleImportFailure)
----------------------------------------------------------------------
ImportError: Failed to import test module: _hl.tests.test_attrs_data
Traceback (most recent call last):
File "/usr/local/lib/python2.7/unittest/loader.py", line 252, in _find_tests
module = self._get_module_from_name(name)
File "/usr/local/lib/python2.7/unittest/loader.py", line 230, in _get_module_from_name
__import__(name)
File "/home/cronburg/Downloads/h5py-2.0.1/build/lib.linux-x86_64-2.7/h5py/_hl/tests/test_attrs_data.py", line 5, in <module>
import h5py
File "/home/cronburg/Downloads/h5py-2.0.1/build/lib.linux-x86_64-2.7/h5py/__init__.py", line 1, in <module>
from h5py import _errors
ImportError: libhdf5.so.7: cannot open shared object file: No such file or directory
----------------------------------------------------------------------
Ran 12 tests in 0.001s
FAILED (errors=12)
h5py was working fine before I went to install the aforementioned packages / tarballs, and I don't remember touching anything that even remotely looked like hdf5. Any ideas?
EDIT:
Trying to locate the file only shows it in the location i untarred it:
cronburg#rhel:~/Downloads/h5py-2.0.1$ locate libhdf5.so.7
/home/cronburg/tmp/hdf5-1.8.9/hdf5/lib/libhdf5.so.7
/home/cronburg/tmp/hdf5-1.8.9/hdf5/lib/libhdf5.so.7.0.3
/home/cronburg/tmp/hdf5-1.8.9/src/.libs/libhdf5.so.7
/home/cronburg/tmp/hdf5-1.8.9/src/.libs/libhdf5.so.7.0.3
take a look on:
http://rpm.pbone.net/index.php3/stat/3/srodzaj/1/search/libhdf5.so.7()(64bit)
Or should try this repo: https://ius.io/Packages/
I prefer always use the most updated python version, in a package format.
https://dl.iuscommunity.org/pub/ius/stable/Redhat/6/x86_64/repoview/python27.html
sudo yum install -y https://centos6.iuscommunity.org/ius-release.rpm
sudo yum install -y python27
sudo yum install -y python27-devel
Do a pip install and be happy:
$ sudo pip install h5py
Installing collected packages: h5py
Successfully installed h5py-2.6.0
This also happened to me when using h5py on a clean raspbian. You need to install the system libraries first.
apt install libhdf5-dev
then
pip install h5py