We Keep Coding

Full installation of tensorflow (all modules)?

I have this repository with me ; https://github.com/layog/Accurate-Binary-Convolution-Network . As requirements.txt says, it requires tensorflow==1.4.1. So I am using miniconda (in Ubuntu18.04) and for the love of God, I can't get it to run (errors out at the below line)
from tensorflow.examples.tutorial.* import input_data
Gives me an ImportError saying it can't find tensorflow.examples. I have diagnosed the problem that a few modules are missing after I installed tensorflow (Have tried all of the below ways)
pip install tensorflow==1.4.1
conda install -c conda-forge tensorflow==1.4.1
#And various wheel packages avaliable on the internet for 1.4.1
pip install tensorflow-1.4.0rc1-cp36-cp36m-manylinux1_x86_64.whl
Question is, if I want all the modules which are present in the git repo source as my installed copy, do I have to COMPLETELY build tensorflow from source ? If yes, can you mention the flag I should use? Are there any wheel packages available that have all modules present in them ?
A link would save me tonnes of effort!
NOTE: Even if I manually import the examples directory, it says tensorflow.contrib is missing, and if I local import that too, another ImportError pops up. There has to be an easier way I am sure of it

Just for reference for others stuck in the same situation:-
Use latest tensorflow build and bezel 0.27.1 for installing it. Even though the requirements state that we need an older version - use newer one instead. Not worth the hassle and will get the job done.
Also to answer the above question about building only specific directories is possible. Each module consists of BUILD file which is fed to bezel.
See the names category of the file to build specific to that folder. For reference the command I used to generate the wheel package for examples.tutorial.mnist :
bazel build --config=opt --config=cuda --incompatible_load_argument_is_label=false //tensorflow/examples/tutorials/mnist:all_files
Here all_files is the name found in the examples/tutorials/mnist/BUILD file.

How to diagnose conan install issue

I have some installation issues with conan
After my Ubuntu 18.04 told "Command 'conan' not found", I guessed the Python
version is wrong. So I attempted to upgrade with the result
$ sudo apt-get install python
python is already the newest version (2.7.15~rc1-1)
However
$ locate python
/var/lib/binfmts/python2.7
/var/lib/binfmts/python3.6
When in this state I attempted to install conan
$ pip install conan
Collecting conan
...
Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyJWT-1.7.1 PyYAML-5.1.2 astroid-1.6.6 attrs-19.1.0 backports.functools-lru-cache-1.5 bottle-0.12.17 certifi-2019.6.16 chardet-3.0.4 colorama-0.4.1 conan-1.18.0 configparser-3.7.4 deprecation-2.0.6 distro-1.1.0 enum34-1.1.6 fasteners-0.15 future-0.16.0 futures-3.3.0 idna-2.8 isort-4.3.21 lazy-object-proxy-1.4.1 mccabe-0.6.1 monotonic-1.5 node-semver-0.6.1 packaging-19.1 patch-1.16 pluginbase-0.7 pygments-2.4.2 pylint-1.9.5 pyparsing-2.4.2 python-dateutil-2.8.0 requests-2.22.0 singledispatch-3.4.0.3 six-1.12.0 tqdm-4.32.2 urllib3-1.25.3 wrapt-1.11.2
then 'conan' is listed as being installed but
$ conan
Command 'conan' not found, did you mean:
I.e, no error message or warning, just does not install.
I could find out that the path was not listed in my PATH, so I added '~.local/bin'. Now the story goes on with the error message
CMake Error at CMakeLists.txt:90 (include):
include could not find load file:
Conan
I found
https://docs.conan.io/en/latest/howtos/cmake_launch.html.
OK, I inserted in my CMakeLists.txt file line
# Download automatically, you can also just copy the conan.cmake file
if(NOT EXISTS "${CMAKE_BINARY_DIR}/conan.cmake")
message(STATUS "Downloading conan.cmake from https://github.com/conan-io/cmake-conan")
file(DOWNLOAD "https://raw.githubusercontent.com/conan-io/cmake-conan/master/conan.cmake"
"${CMAKE_BINARY_DIR}/conan.cmake")
endif()
include(${CMAKE_BINARY_DIR}/conan.cmake)
conan_cmake_run(REQUIRES Catch2/2.6.0#catchorg/stable
BASIC_SETUP)
I was also advised,
Please specify in command line CMAKE_BUILD_TYPE
(-DCMAKE_BUILD_TYPE=Release)
So I use
cmake .. -DCMAKE_BUILD_TYPE=Release
rather than
cmake ..
Still, I receive
ERROR: compiler not defined for compiler.libcxx
Please define compiler value first too
FATAL_ERROR;conan install command failed.
STATUS;Conan: Compiler GCC>=5, checking major version 7
STATUS;Conan: Checking correct version: 7
About two weeks ago I could install on another system the same project flawlessly. Can I go back somehow to that state? I expected conan to be stable, rather than alpha.
Edit 2:
I issued
conan profile new default --detect --force
The reply is
Found gcc 7
gcc>=5, using the major as version
************************* WARNING: GCC OLD ABI COMPATIBILITY ***********************
Conan detected a GCC version > 5 but has adjusted the 'compiler.libcxx' setting to
'libstdc++' for backwards compatibility.
Your compiler is likely using the new CXX11 ABI by default (libstdc++11).
(I do not really know why in the case of a new project I need backward compatibility) After that,
cmake ..
finally seems to work. I am afraid I will have further issues due to the compiler standards. For example, SystemC defaults to '98, but some other library uses feature needing '14, and now conan forces to use '11. Is there a way to handle all this centrally, specific to MY system?
Concerning the two python versions: I did not install this manually, only some other install programs did so. I do not really know why and which install script causes such doubling. BTW: Ubuntu said that V2.7 is the newest version, although V3.x is also present. I am a bit confused about these version numbers.
I simply made a new install, and did not especially very WHEN the second version of python appeared. I personally do not even use python, only some install scripts could install it.
Whether my system is specific: I do not think so. I just installed Ubuntu 18.04.2, and my primary goal was to install this SystemC related stuff. I really installed ONLY what was declared as missing. (plus livetex, git, etc.)
In the meantime 'cmake ..' terminated. Appearently, the installation by conan terminated OK. However, when configuring my project, gives messages like
CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
Please set them or make sure they are set and tested correctly in the CMake files:
SCV_INCLUDE_DIRS
The missing files are installed also by conan, using
[requires]
SystemC/2.3.3#minres/stable
SystemCVerification/2.0.1#minres/stable
doxygen_installer/1.8.15#bincrafters/stable
qt/5.12.0#bincrafters/stable
gtest/1.8.1#bincrafters/stable
flex/2.6.4#bincrafters/stable
I am using literally the same files (either my old disk connected to the bus or the new one, using the same cable). The installation made about a month ago runs fine, the new one behaves as described.
It looks like installing and using conan is too complicated for me. I wanted to simplify installation rather than complicate it.

There is a bunch of cases related to installation listed here:
https://docs.conan.io/en/latest/installation.html#known-installation-issues-with-pip
I would say Conan is installed but is not listed in your PATH. You could find Conan in your Python package folder and update your PATH with conan path:
python -m site # list your package folder
find <package folder> -name conan
echo PATH=${PATH}:<package folder> >> ~/.bashrc
source ~/.bashrc

How to build embeddable Python

I am wondering how to build embeddable Python for Windows. Could not find any info or build configuration for that particular package.
Reason I ask is that I could use the same variant for Mac and Linux.

Recently, I had to build an embeddable zip for 64bit Python 3.5.6 (since the official download page doesn't offer them for Python greater than 3.5.4), here are the steps.
Prerequisites (one-time install)
Install
git
TortoiseSVN
Visual Studio C++ Build Tools 2015 (you may need version 2017 when building latest Python versions)1
If you haven't Sphinx installed, install it and add sphinx-build to PATH2:
> python -m pip install --user sphinx
> set PATH=%PATH%;C:\Users\jenkins\AppData\Roaming\Python\Python35\Scripts
If you haven't blurb installed, install it, add blurb to PATH if necessary (if you've done the previous step, you already have the correct PATH):
> python -m pip install --user blurb
> set PATH=%PATH%;C:\Users\jenkins\AppData\Roaming\Python\Python35\Scripts
Configure build environment
Clone CPython repo. Here, I clone only the tag I need:
> git clone --depth 1 --branch v3.5.6 https://github.com/python/cpython
In the repo dir, generate NEWS file3, 4:
> blurb merge -f Misc\NEWS
Build
Now you can trigger the build:
> Tools\msi\buildrelease.bat -x64 --skip-nuget --test testout
Replace -x64 with -x32 if you target the 32bit arch. --skip-nuget will omit building the MSI installer. --test testout will trigger installer tests; you can skip them if you want.
Once the build finishes, check for build artifact PCbuild\amd64\en-us\python-3.5.6-embed-amd64.zip. This is your embeddable zip.
After the build succeeds for the first time, you can skip the doc build in future builds for further speedup as long as the doc build artifacts remain in repo:
> Tools\msi\buildrelease.bat -x64 --skip-doc --skip-nuget --test testout
1 Make sure you select "Custom" in the VC++ Build Tools installer and check all the options; otherwise, you may get build errors like The code execution cannot proceed because ucrtbased.dll was not found etc. I'm no Windows expert by all means, just installed everything possible to get rid of the errors.
2 Looking at the buildrelease.bat help, it seems like you can skip the doc build with --skip-doc flag, but when using that, I got the error
"C:\Users\jenkins\projects\cpython\Tools\msi\bundle\releaselocal.wixproj"
(Rebuild target) (1) ->
"C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc.wixproj" (Rebuild target) (7) ->
(Link target) ->
C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc_files.wxs(8): error
LGHT0103: The system cannot find the file 'python356.chm'. [C:\Users\jenkins\projects\cpython\Tools\msi\doc\doc.wixproj]
and the build aborts with an error. If you manage to circumvent this without an initial doc build (so the doc files are available in the build dir), you can skip Sphinx install.
3 Otherwise, I get the build error
"C:\Users\jenkins\projects\cpython\Tools\msi\bundle\releaselocal.wixproj" (Rebuild target) (1) ->
"C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe.wixproj" (Rebuild target) (8) ->
C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe_files.wxs(9): error
LGHT0103: The system cannot find the file '!(bindpath.src)Misc\NEWS'. [C:\Users\jenkins\projects\cpython\Tools\msi\exe\exe.wixproj]
4 You can replace the command with an appropriate make invocation, but I was too lazy to decipher the Doc\Makefile. After all, the news won't land in the embeddable zip anyway; this is only to make the buildrelease.bat happy.

Aubio 0.4.0 Python Windows Installed but failing when creating aubio source

So I managed to get aubio 0.4.0 installed so that it imports into python without errors however I haven't figured out how to pass files to be analyzed.
Here's the steps I've done to install aubio 0.4.0 taken from here:
Downloaded the most recent git build of Aubio 0.4.0 source download - http://git.aubio.org/
Unpacked onto C:\
installed python 2.7.6
appended C:\python27 to the 'Path' environment variable
installed MinGW v-0.6.2 mingw.org/download/installer
inside MinGW Installation manager I included - [mingw32-base]
appended C:\MinGW\bin to the 'Path' environment variable
created file "C:\Python27\Lib\distutils\distutils.cfg" containing:
[build]
compiler = mingw32
--------------- INCLUDING LIBAV libraries ---------------------------
download pygtk-all-in-one-2.24.2.win32-py2.7.msi to get pkgconfig and all it's dependancies: ftp.gnome.org/pub/GNOME/binaries/win32/pygtk/2.24/
download libav win32 build win32.libav.org/win32/ and unpack into C:\libav\
create a new environment variable name: "PKG_CONFIG_PATH" with the value at: C:\libav\usr\lib\pkgconfig
append C:\libav\usr\bin\ to the 'Path' environment variable
-------------------- END LIBAV ---------------------------------------
Inside the aubio path run the command: python .\waf configure build -j 1 --check-c-compiler=gcc
I get a crash at 168/193 with test-delnull.exe but the build keeps going and returns "build" finished successfully
Install numpy v-1.8.0 sourceforge.net/projects/numpy/files/NumPy/
Inside the aubio\python path run the command: python setup.py build
Inside the aubio\python path run the command: python setup.py install
I had to copy the dll from aubio\build\src\libaubio-4.dll into python27\Lib\site-packages\aubio\
Then I added one of my own test.mp3 and test.wav files into aubio\python\tests\sounds\
Inside the aubio\python\tests path I ran the command: python run_all_tests -v
------------------- EDIT ---------------------------------
The above instructions should work now without the problem originally asked
------------------- END EDIT -----------------------------
So from the results I get a lot of 'ok' regarding the many different tests being made however it's first problem is with "test_many_sinks" where it tries to use the .wav file from sounds and gives:
AUBIO ERROR: failed creating aubio source with [wav file path]
It continues giving the same error for the rest of the tests until it crashes on "test_zero_hop_size" and stops.
Any further advice as to what I still need to do would be much appreciated.
Thanks!

With help from Paul Brossier we found out two issues:
Because I never included libav into my build I can't use .mp3's to test
Using a newer git repository ended up allowing me successfully run demo_bpm_extract.py which was previously erroring even when I tested with a .wav file. The git source I used can be found here: http://git.aubio.org/?p=aubio.git;a=commit;h=4a1378c12ffe7fd518448f6a1ab00f99f0557286
There are still quite a few errors showing up when executing "run_all_tests" which I've tried to pass over to Paul.

Shipping *.so and binaries while building RPM package

I have created a python application in which I would like to ship .so and some binary files in the final RPM package. After long reading I found a way to add binaries/ image and other data files in setup.py. Now, when I build an RPM with python setup.py bdist_rpm command, it complains about architecture dependency:
Arch dependent binaries in noarch package
error: command 'rpmbuild' failed with exit status 1
After googling I found that we can add:
#%define _binaries_in_noarch_packages_terminate_build 0
or removing the line BuildArch: noarch in the packagename.spec file to overcome the rpmbuild failure. However, every time I add or remove line from build/bdist.linux-i686/rpm/SPECS/packagename.spec the command python setup.py bdist_rpm always overwrites the .spe file.
Is there a way to avoid Arch dependent binaries and ship *.so and other binary files in rpm?

The behavior of bdist_rpm is defined by a bunch of settings in:
/usr/lib/rpm/macros
/etc/rpm/macros
$HOME/.rpmmacros
I'm willing to bet that only /usr/lib/rpm/macros exists on your system. This is normal.
So, in order to prevent the "Arch dependent binaries in noarch package" error you would create /etc/rpm/macros or ~/.rpmmacros and add the following:
%_unpackaged_files_terminate_build 0
%_binaries_in_noarch_packages_terminate_build 0
Do not modify /usr/lib/rpm/macros because that file will be overwritten by the system whenever the rpm-build package is upgraded, downgraded, or re-installed.
If you want to override the behavior for everyone on the system put the settings in /etc/rpm/macros. If you want override the behavior for a particular user then add the settings to $HOME/.rpmmacros.
.rpmmacros trumps /etc/rpm/macros which trumps /usr/lib/rpm/macros.
Note: it's useful to examine /usr/lib/rpm/macros to see what settings are available and for syntax examples.
As a side note, %_unpackaged_files_terminate_build 0 setting prevents the error: Installed (but unpackaged) file(s) found: error.

.so files are always arch dependent as far as I know.
In your case to avoid having to edit the specs-file all the time you can add --force-arch=<your_arch> to our setup.py bdist_rpm
e.g.
python setup.py bdist_rpm --force-arch=x86_64

If you encounter this matter in a .spec file when you try to build a new .rpm package.
Change the BuildArch from noarch to x86_64 (or whatever you have on the building system)
[root#devel-mga7][~/build/yate-ota]# grep Arch yate-ota.spec
BuildArch: x86_64
[root#devel-mga7][~/build/yate-ota]#