We Keep Coding

Why does yocto scipy recipe require python3 explicitly set? How?

I have a recipe to build scipy which parses fine and bitbake starts building but the python3
version requirement is not met. It exits with
| DEBUG: Executing shell function do_configure
| Traceback (most recent call last):
| File "setup.py", line 31, in <module>
| raise RuntimeError("Python version >= 3.5 required.")
| RuntimeError: Python version >= 3.5 required.
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_configure (log file is located at /home/marius/mender-qemu/build/tmp/work/core2-64-poky-linux/python3-scipy/1.4.1-r0/temp/log.do_configure.30478)
I successfully built other python3 packages which can be imported in the running image. You can also see from the path that python3 is used and the image runs on python3.5. I'm using thud.
For the sake of completion here is the recipe. I also tried explicitly adding dependencies (numpy) but that did not have any effect.
SUMMARY = "Scipy summary to be filled"
DESCRIPTION = "Scientific computing"
PYPI_PACKAGE = "scipy"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://LICENSE.txt;md5=011ccf01b7e0590d9435a864fc6a4d2b"
SRC_URI[md5sum] = "3a97689656f33f67614000459ec08585"
SRC_URI[sha256sum] = "dee1bbf3a6c8f73b6b218cb28eed8dd13347ea2f87d572ce19b289d6fd3fbc59"
This is the python-scipy.inc
inherit setuptools3 distutils
require python-scipy.inc
Also, I tried to add inherit python3native without effect.
My question is: how can I explicitly set python3 to build this recipe?

The simple and obvious solution was to inherit distutils3 instead of inherit distutils.
NOTE: A python-scipy recipe that worked out of the box for me can be found here: https://github.com/gpanders/oe-scipy
Nice job gpanders!

My guess is that you run a python script setup.py as part of your build that requires python3 on your host(the system that build Yocto).
You can install it like this:
sudo apt-get install python3

How to add python-xbee receipes for yocto

My python-xbee-2.3.2.bb file contains:
SRC_URI="https://github.com/niolabs/python-xbee/archive/v2.3.2.tar.gz"
SRC_URI[md5sum] = "e74381e23f12159675d7ad92a06d8d0f"
DEPENDS += "${PYTHON_PN}-pytest-runner-native"
do_configure[noexec]="1"
inherit setuptools
RDEPENDS_${PN} += " \
${PYTHON_PN}-subprocess "
RDEPENDS_${PN} = "\
${PYTHON_PN}-fcntl \
${PYTHON_PN}-numbers \
${PYTHON_PN}-shell \
${PYTHON_PN}-stringold \
${PYTHON_PN}-io \
${PYTHON_PN}-logging \
${PYTHON_PN}-math \
${PYTHON_PN}-netclient \
${PYTHON_PN}-threading "
This is error getting when I compile bitbake python-xbee-2.3.2:
ERROR: python-xbee-2.3.2-1.0-r0 do_compile: 'python setup.py build '
execution failed. ERROR: python-xbee-2.3.2-1.0-r0 do_compile: Function
failed: do_compile (log file is located at
/home/chandra/yoctorpi/mender-raspberrypi/build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/python-xbee-2.3.2/1.0-r0/temp/log.do_compile.15930)
ERROR: Logfile of failure stored in:
/home//build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/python-xbee-2.3.2/1.0-r0/temp/log.do_compile.15930
After that am getting log data follows:
can't open file 'setup.py':[Errno 2] No such file or directory.
/home//build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/python-xbee-2.3.2/1.0-r0/recipe-sysroot-native/usr/bin/python-native/python: can't open file 'setup.py' :[Errno 2] No such file or directory
ERROR: 'python setup.py build ' execution failed
ERROR: Function failed: do_compile (log file is located at /home//build/tmp/work/cortexa7t2hf-neon-vfpv4-poky-linux-gnueabi/python-xbee-2.3.2/1.0-r0/temp/log.do_compile.15930)
ERROR: Task (/home//build/../sources/poky/meta/recipes-support/python-xbee-2.3.2/python-xbee-2.3.2.bb:do_compile) failed with exit code '1'

You should inherit pypi bbclass as well since this deals with python package installation:
inherit pypi
As a side notes:
Check how you assign RDEPENDS_${PN}, it looks that you just override it instead of updating.
The name of recipe according to standard should python-xbee_2.3.2.bb - note the underscore (assuming your package supports only python2.X).
It's not recommended to hard-code version in SRC_URI, check in the above link as well.

Can't use Python's sh module in Bazel genrule

When I run a python script, which uses the 'sh' module, from bazel genrule, it failed with this:
INFO: Analysed target //src:foo_gen (8 packages loaded).
INFO: Found 1 target...
ERROR: /home/libin11/workspace/test/test/src/BUILD:1:1: Executing genrule //src:foo_gen failed (Exit 1)
Traceback (most recent call last):
File "src/test.py", line 2, in <module>
sh.touch("foo.bar")
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1427, in __call__
return RunningCommand(cmd, call_args, stdin, stdout, stderr)
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 767, in __init__
self.call_args, pipe, process_assign_lock)
File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1784, in __init__
self._stdout_read_fd, self._stdout_write_fd = pty.openpty()
File "/usr/lib/python2.7/pty.py", line 29, in openpty
master_fd, slave_name = _open_terminal()
File "/usr/lib/python2.7/pty.py", line 70, in _open_terminal
raise os.error, 'out of pty devices'
OSError: out of pty devices
Target //src:foo_gen failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.143s, Critical Path: 0.12s
INFO: 0 processes.
FAILED: Build did NOT complete successfully
I want to integrate a thirdparty project to my own. The third party project is built with a python script, so I would like to build the project with bazel genrule.
Here is an example file list:
.
├── src
│ ├── BUILD
│ └── test.py
└── WORKSPACE
WORKSPACE is empty, BUILD is:
genrule(
name = "foo_gen",
srcs = glob(["**/*"]),
outs = ["foo.bar"],
cmd = "python $(location test.py)",
)
test.py is:
import sh
sh.touch("foo.bar")
And run:
bazel build //src:foo_gen
OS: Ubuntu 16.04
bazel: release 0.14.1

It looks like if you change the call to sh.touch("foo.bar", _tty_in=False, _tty_out=False) it works, but you'll still need a bit of modification to the genrule otherwise it won't produce output.
I prefer to import pip dependencies using the bazel python rules, so I can create the tool for my genrule. This way, bazel handles the pip requirement install and you don't have to chmod the test.py file.
load("#my_deps//:requirements.bzl", "requirement")
py_binary(
name = "foo_tool",
srcs = [
"test.py",
],
main = "test.py",
deps = [
requirement("sh"),
],
)
genrule(
name = "foo_gen",
outs = ["foo.bar"],
cmd = """
python3 $(location //src:foo_tool)
cp foo.bar $#
""",
tools = [":foo_tool"],
)
Note the required copy in the genrule command. It's a bit cleaner if your python script can output to std out, then you can just redirect the output to the file instead of adding a copy command. See this for more info.
My output with these changes:
INFO: Analysed target //src:foo_gen (0 packages loaded).
INFO: Found 1 target...
Target //src:foo_gen up-to-date:
bazel-genfiles/src/foo.bar
INFO: Elapsed time: 0.302s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action

bazel can't build py_proto_library

My BUILD file is pretty simple, just
load("#protobuf_bzl//:protobuf.bzl", "py_proto_library")
py_proto_library(
name = "struct_py_pb2",
srcs = ["struct.proto"],
)
But bazel gives a bunch of baffling error messages like:
$ bazel build google/genomics/v1:all
ERROR: thomaswc//v1/BUILD:22:1: no such package '': BUILD file not found on package path and referenced by '//v1:struct_py_pb2'
ERROR: Analysis of target '//v1:struct_py_pb2' failed; build aborted: no such package '': BUILD file not found on package path
INFO: Elapsed time: 0.581s
FAILED: Build did NOT complete successfully (2 packages loaded)
currently loading: #protobuf_bzl//
I see other projects on github using bazel and py_proto_library, though, so I know it must be possible. Is there some WORKSPACE or .bzl magic that I need?

After a bunch of digging, I found a work-around: the default values of default_runtime and protoc are screwed up, so you need to override them:
py_proto_library(
name = "struct_py_pb2",
srcs = ["struct.proto"],
default_runtime = "#com_google_protobuf//:protobuf_python",
protoc = "#com_google_protobuf//:protoc",
)

Error setting up thrift modules for python

I'm trying to set up thrift in order to incorporate with Cassandra, so when I ran the
setup.py
it out puts this message in command line
running build
running build_py
running build_ext
building 'thrift.protocol.fastbinary' extension
C:\MinGW\bin\gcc.exe -mno-cygwin -mdll -O -Wall -IC:\Python26\include -IC:\Pytho
n26\PC -c src/protocol/fastbinary.c -o build\temp.win32-2.6\Release\src\protocol
\fastbinary.o
src/protocol/fastbinary.c:24:24: netinet/in.h: No such file or directory
src/protocol/fastbinary.c:85:4: #error "Cannot determine endianness"
src/protocol/fastbinary.c: In function `writeI16':
src/protocol/fastbinary.c:295: warning: implicit declaration of function `htons'
src/protocol/fastbinary.c: In function `writeI32':
src/protocol/fastbinary.c:300: warning: implicit declaration of function `htonl'
src/protocol/fastbinary.c: In function `readI16':
src/protocol/fastbinary.c:688: warning: implicit declaration of function `ntohs'
src/protocol/fastbinary.c: In function `readI32':
src/protocol/fastbinary.c:696: warning: implicit declaration of function `ntohl'
error: command 'gcc' failed with exit status 1
Need some helping on this issue.I have already install the MigW32
Thanks.

With a bit of source file tweaking it is possible to install it with MINGW on Windows. I'm using thrift 0.9.1 and Python 27
The steps I followed were:
If you are using Python 2.7, follow the normal setup steps and workarounds for minGW. In particular you may need to open the file C:\Python27\Lib\distutils\cygwinccompiler.py, and modify the class Mingw32CCompiler to remove all references to the -mno-cygwin option. This option is deprecated and will cause the compiler to halt with an error if it is left in.
Open fastbinary.c and add the following include statement,
#include <stdbool.h>,
this includes definitions for true / false which will otherwise cause the compile to fail. (I assume they're included by default on MSVC?)
3) Modify the setup.py file to tell the linker to link to ws2_32.lib. This is done using a pragma comment on MSVC, but gcc doesn't support this option. So your ext_modules should look like :
ext_modules = [
Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries=['ws2_32'],
include_dirs = include_dirs,
)
],
4) Build as normal using the setup.py
On my setup, I didn't get much of a speed improvement when using the C extension rather than pure python (about a 5% difference), so the effort to do this might not be justified except in extreme cases.

I've only succeeded in installing Thrift with MSVC.
Install MSVC
Get Thrift
Apply thrift-252-python-msvc-1.diff patch (google it)
The fastbinary.c will be patched, but setup.py patch will fail, update manually from hints at setup.py.rej, here's a (seemingly) correct copy:
from distutils.core import setup, Extension
import sys
libraries = []
if sys.platform == 'win32':
libraries.append('ws2_32')
fastbinarymod = Extension('thrift.protocol.fastbinary',
sources = ['src/protocol/fastbinary.c'],
libraries = libraries,
)
setup(name = 'Thrift',
version = '0.1',
description = 'Thrift Python Libraries',
author = 'Thrift Developers',
author_email = 'thrift-dev#incubator.apache.org',
url = 'http://incubator.apache.org/thrift/',
license = 'Apache License 2.0',
packages = [
'thrift',
'thrift.protocol',
'thrift.transport',
'thrift.server',
],
package_dir = {'thrift' : 'src'},
ext_modules = [fastbinarymod],
)
Endianness test will fail, modify fastbinary.c (around line 68):
#ifdef _MSC_VER
#define __BYTE_ORDER __LITTLE_ENDIAN
#endif
After that run python setup.py install, hopefully, you'll get what you need.

Install python-dev
You can run:
sudo apt-get install python-dev

I would recommend you to execute the following command:
pip3 install thriftpy2