We Keep Coding

Unable to use pip modules with PyO3

Recently I have been working on a project that involves generating docx files. Since Rust's docx support is still quite immature, I've decided to use Python's python-docx module via PyO3.
Here's my code so far:
extern crate pyo3;
use pyo3::prelude::*;
(...)
// Initialize some Python
let gil = Python::acquire_gil();
let py = gil.python();
let docx = PyModule::import(py, "docx")?;
let document = docx.Document();
Unfortunately, I'm running into two pretty serious errors.
Error #1:
let docx = PyModule::import(py, "docx")?;
^ cannot use the `?` operator in a function that returns `std::string::String
Error #2:
let document = docx.Document();
^^^^^^^^ method not found in `&pyo3::prelude::PyModule`
How do I solve these errors?
N.B. Yes, I have made sure that python-docx is installed. It's located in /home/<my username>/.local/lib/python3.8/site-packages

How to create a universal program code for different versions of a program library?

I use a software library that has different function names in different versions of the library.
I try to use the following code:
some_variable = module.old_name_of_function()
But this code only works with the old version of the program library.
I plan to use the code on different computers, with different installed versions of the software library.
Some computers have a new version of the program library installed, and the following code should be used there:
some_variable = module.new_name_of_function()
And if I use old_name_of_function() in the new version of the library, I will get an error.
How to solve this issue?

I suppose you could do
try:
my_func = module.old_name_of_function
except AttributeError:
my_func = module.new_name_of_function
some_variable = my_func()

You can use pkg_resources module for it (example for numpy):
import pkg_resources
pkg_resources.get_distribution("numpy").version
will return:
'1.15.2'
Then you can use cases, ifs or something else to run a function you need.
For example:
import pkg_resources
version = pkg_resources.get_distribution("numpy").version
v = version.split('.')
if int(v[0]) == 1 and int(v[1]) < 17:
print('WAKA')
else:
print('NEW WAKA')
will print 'WAKA' for every 1.X version of numpy, where X < 17.

Python pcl has no attribute from_array

I was trying to run the example on this page:
https://github.com/strawlab/python-pcl/blob/master/examples/example.py
I get the error : AttributeError: type object 'PointCloud' has no attribute 'from_array' on line 49. Also, i cant import few modules like pcl.pcl_visualization and pcl.PointCloud_PointXYZRGB() . Anybody knows how do i slove these issues?
Thanks

You should make sure you aren't running PyPCL by accident. This is an alternate python to PCL binding that import pcl can map to. One way to check this is to run this snippet:
class_names = dict([(name, cls) for name, cls in pcl.__dict__.items() if isinstance(cls, type)])
for name, cls in class_names.items():
print(name)
This will give you a list of classes in the module. If there are two (PointCloud, Quaternion), you likely have PyPCL installed and should uninstall that one. The list of classes in python-pcl returned should look something like this:
PointCloud
PointCloud_PointXYZI
PointCloud_PointXYZRGB
PointCloud_PointXYZRGBA
Vertices
PointCloud_PointWithViewpoint
PointCloud_Normal
PointCloud_PointNormal
KdTree
RangeImages
SampleConsensusModel
SampleConsensusModelPlane
SampleConsensusModelSphere
SampleConsensusModelCylinder
SampleConsensusModelLine
SampleConsensusModelRegistration
SampleConsensusModelStick
Segmentation
Segmentation_PointXYZI
Segmentation_PointXYZRGB
Segmentation_PointXYZRGBA
SegmentationNormal
Segmentation_PointXYZI_Normal
Segmentation_PointXYZRGB_Normal
Segmentation_PointXYZRGBA_Normal
EuclideanClusterExtraction
RegionGrowing
StatisticalOutlierRemovalFilter
StatisticalOutlierRemovalFilter_PointXYZI
StatisticalOutlierRemovalFilter_PointXYZRGB
StatisticalOutlierRemovalFilter_PointXYZRGBA
VoxelGridFilter
VoxelGridFilter_PointXYZI
VoxelGridFilter_PointXYZRGB
VoxelGridFilter_PointXYZRGBA
PassThroughFilter
PassThroughFilter_PointXYZI
PassThroughFilter_PointXYZRGB
PassThroughFilter_PointXYZRGBA
ApproximateVoxelGrid
ApproximateVoxelGrid_PointXYZI
ApproximateVoxelGrid_PointXYZRGB
ApproximateVoxelGrid_PointXYZRGBA
KdTreeFLANN
KdTreeFLANN_PointXYZI
KdTreeFLANN_PointXYZRGB
KdTreeFLANN_PointXYZRGBA
OctreePointCloud
OctreePointCloud_PointXYZI
OctreePointCloud_PointXYZRGB
OctreePointCloud_PointXYZRGBA
OctreePointCloud2Buf
OctreePointCloud2Buf_PointXYZI
OctreePointCloud2Buf_PointXYZRGB
OctreePointCloud2Buf_PointXYZRGBA
OctreePointCloudSearch
OctreePointCloudSearch_PointXYZI
OctreePointCloudSearch_PointXYZRGB
OctreePointCloudSearch_PointXYZRGBA
OctreePointCloudChangeDetector
OctreePointCloudChangeDetector_PointXYZI
OctreePointCloudChangeDetector_PointXYZRGB
OctreePointCloudChangeDetector_PointXYZRGBA
CropHull
CropBox
ProjectInliers
RadiusOutlierRemoval
ConditionAnd
ConditionalRemoval
ConcaveHull
ConcaveHull_PointXYZI
ConcaveHull_PointXYZRGB
ConcaveHull_PointXYZRGBA
MovingLeastSquares
MovingLeastSquares_PointXYZRGB
MovingLeastSquares_PointXYZRGBA
GeneralizedIterativeClosestPoint
IterativeClosestPoint
IterativeClosestPointNonLinear
RandomSampleConsensus
NormalEstimation
VFHEstimation
IntegralImageNormalEstimation
MomentOfInertiaEstimation
HarrisKeypoint3D
NormalDistributionsTransform
Sequence
Alternatively, if you are in a virtual environment you can run pipenv graph to look for this issue. Or:
pip list

How to properly write cross-references to external documentation with intersphinx?

I'm trying to add cross-references to external API into my documentation but I'm facing three different behaviors.
I am using sphinx(1.3.1) with Python(2.7.3) and my intersphinx mapping is configured as:
{
'python': ('https://docs.python.org/2.7', None),
'numpy': ('http://docs.scipy.org/doc/numpy/', None),
'cv2' : ('http://docs.opencv.org/2.4/', None),
'h5py' : ('http://docs.h5py.org/en/latest/', None)
}
I have no trouble writing a cross-reference to numpy API with :class:`numpy.ndarray` or :func:`numpy.array` which gives me, as expected, something like numpy.ndarray.
However, with h5py, the only way I can have a link generated is if I omit the module name. For example, :class:`Group` (or :class:`h5py:Group`) gives me Group but :class:`h5py.Group` fails to generate a link.
Finally, I cannot find a way to write a working cross-reference to OpenCV API, none of these seems to work:
:func:`cv2.convertScaleAbs`
:func:`cv2:cv2.convertScaleAbs`
:func:`cv2:convertScaleAbs`
:func:`convertScaleAbs`
How to properly write cross-references to external API, or configure intersphinx, to have a generated link as in the numpy case?

In addition to the detailed answer from #gall, I've discovered that intersphinx can also be run as a module:
python -m sphinx.ext.intersphinx 'http://python-eve.org/objects.inv'
This outputs nicely formatted info. For reference: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/intersphinx.py#L390

I gave another try on trying to understand the content of an objects.inv file and hopefully this time I inspected numpy and h5py instead of only OpenCV's one.
How to read an intersphinx inventory file
Despite the fact that I couldn't find anything useful about reading the content of an object.inv file, it is actually very simple with the intersphinx module.
from sphinx.ext import intersphinx
import warnings
def fetch_inventory(uri):
"""Read a Sphinx inventory file into a dictionary."""
class MockConfig(object):
intersphinx_timeout = None # type: int
tls_verify = False
class MockApp(object):
srcdir = ''
config = MockConfig()
def warn(self, msg):
warnings.warn(msg)
return intersphinx.fetch_inventory(MockApp(), '', uri)
uri = 'http://docs.python.org/2.7/objects.inv'
# Read inventory into a dictionary
inv = fetch_inventory(uri)
# Or just print it
intersphinx.debug(['', uri])
File structure (numpy)
After inspecting numpy's one, you can see that keys are domains:
[u'np-c:function',
u'std:label',
u'c:member',
u'np:classmethod',
u'np:data',
u'py:class',
u'np-c:member',
u'c:var',
u'np:class',
u'np:function',
u'py:module',
u'np-c:macro',
u'np:exception',
u'py:method',
u'np:method',
u'np-c:var',
u'py:exception',
u'np:staticmethod',
u'py:staticmethod',
u'c:type',
u'np-c:type',
u'c:macro',
u'c:function',
u'np:module',
u'py:data',
u'np:attribute',
u'std:term',
u'py:function',
u'py:classmethod',
u'py:attribute']
You can see how you can write your cross-reference when you look at the content of a specific domain. For example, py:class:
{u'numpy.DataSource': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html#numpy.DataSource',
u'-'),
u'numpy.MachAr': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.MachAr.html#numpy.MachAr',
u'-'),
u'numpy.broadcast': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html#numpy.broadcast',
u'-'),
...}
So here, :class:`numpy.DataSource` will work as expected.
h5py
In the case of h5py, the domains are:
[u'py:attribute', u'std:label', u'py:method', u'py:function', u'py:class']
and if you look at the py:class domain:
{u'AttributeManager': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/attr.html#AttributeManager',
u'-'),
u'Dataset': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/dataset.html#Dataset',
u'-'),
u'ExternalLink': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/group.html#ExternalLink',
u'-'),
...}
That's why I couldn't make it work as numpy references. So a good way to format them would be :class:`h5py:Dataset`.
OpenCV
OpenCV's inventory object seems malformed. Where I would expect to find domains there is actually 902 function signatures:
[u':',
u'AdjusterAdapter::create(const',
u'AdjusterAdapter::good()',
u'AdjusterAdapter::tooFew(int',
u'AdjusterAdapter::tooMany(int',
u'Algorithm::create(const',
u'Algorithm::getList(vector<string>&',
u'Algorithm::name()',
u'Algorithm::read(const',
u'Algorithm::set(const'
...]
and if we take the first one's value:
{u'Ptr<AdjusterAdapter>': (u'OpenCV',
u'2.4',
u'http://docs.opencv.org/2.4/detectorType)',
u'ocv:function 1 modules/features2d/doc/common_interfaces_of_feature_detectors.html#$ -')}
I'm pretty sure it is then impossible to write OpenCV cross-references with this file...
Conclusion
I thought intersphinx generated the objects.inv based on the content of the documentation project in an standard way, which seems not to be the case.
As a result, it seems that the proper way to write cross-references is API dependent and one should inspect a specific inventory object to actually see what's available.

An additional way to inspect the objects.inv file is with the sphobjinv module.
You can search local or even remote inventory files (with fuzzy matching). For instance with scipy:
$ sphobjinv suggest -t 90 -u https://docs.scipy.org/doc/scipy/reference/objects.inv "signal.convolve2d"
Remote inventory found.
:py:function:`scipy.signal.convolve2d`
:std:doc:`generated/scipy.signal.convolve2d`
Note that you may need to use :py:func: and not :py:function: (I'd be happy to know why).

How to use OpenCV 2.4 (cv2) intersphinx
Inspired by #Gall's answer, I wanted to compare the contents of the OpenCV & numpy inventory files. I couldn't get sphinx.ext.intersphinx.fetch_inventory to work from ipython, but the following does work:
curl http://docs.opencv.org/2.4/objects.inv | tail -n +5 | zlib-flate -uncompress > cv2.inv
curl https://docs.scipy.org/doc/numpy/objects.inv | tail -n +5 | zlib-flate -uncompress > numpy.inv
numpy.inv has lines like this:
numpy.ndarray py:class 1 reference/generated/numpy.ndarray.html#$ -
whereas cv2.inv has lines like this:
cv2.imread ocv:pyfunction 1 modules/highgui/doc/reading_and_writing_images_and_video.html#$ -
So presumably you'd link to the OpenCV docs with :ocv:pyfunction:`cv2.imread` instead of :py:function:`cv2.imread`. Sphinx doesn't like it though:
WARNING: Unknown interpreted text role "ocv:pyfunction".
A bit of Googling revealed that the OpenCV project has its own "ocv" sphinx domain: https://github.com/opencv/opencv/blob/2.4/doc/ocv.py -- presumably because they need to document C, C++ and Python APIs all at the same time.
To use it, save ocv.py next to your Sphinx conf.py, and modify your conf.py:
sys.path.insert(0, os.path.abspath('.'))
import ocv
extensions = [
'ocv',
]
intersphinx_mapping = {
'cv2': ('http://docs.opencv.org/2.4/', None),
}
In your rst files you need to say :ocv:pyfunc:`cv2.imread` (not :ocv:pyfunction:).
Sphinx prints some warnings (unparseable C++ definition: u'cv2.imread') but the generated html documentation actually looks ok with a link to http://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#cv2.imread. You can edit ocv.py and remove the line that prints that warning.

The accepted answer no longer works in the new version (1.5.x) ...
import requests
import posixpath
from sphinx.ext.intersphinx import read_inventory
uri = 'http://docs.python.org/2.7/'
r = requests.get(uri + 'objects.inv', stream=True)
r.raise_for_status()
inv = read_inventory(r.raw, uri, posixpath.join)

Stubborn fool that I am, I used 2to3 and the Sphinx deprecated APIs chart to revive #david-röthlisberger's ocv.py-based answer so it'll work with Sphinx 2.3 on Python 3.5.
The fixed-up version is here:
https://gist.github.com/ssokolow/a230b27b7ea4a31f7fb40621e6461f9a
...and the quick version of what I did was:
Run 2to3 -w ocv.py && rm ocv.py.bak
Cycle back and forth between running Sphinx and renaming functions to their replacements in the chart. I believe these were the only changes I had to make on this step:
Directive now has to be imported from docutils.parsers.rst
Replace calls to l_(...) with calls to _(...) and remove the l_ import.
Replace calls to env.warn with calls to log.warn where log = sphinx.util.logging.getLogger(__name__).
Then, you just pair it with this intersphinx definition and you get something still new enough to be relevant for most use cases:
'cv2': ('https://docs.opencv.org/3.0-last-rst/', None)

For convenience, I made a small extension for aliasing intersphinx cross references. This is useful as sometimes the object inventory gets confused when an object from a submodule is imported from a package's __init__.py.
See also https://github.com/sphinx-doc/sphinx/issues/5603
###
# Workaround of
# Intersphinx references to objects imported at package level can"t be mapped.
#
# See https://github.com/sphinx-doc/sphinx/issues/5603
intersphinx_aliases = {
("py:class", "click.core.Group"):
("py:class", "click.Group"),
("py:class", "click.core.Command"):
("py:class", "click.Command"),
}
def add_intersphinx_aliases_to_inv(app):
from sphinx.ext.intersphinx import InventoryAdapter
inventories = InventoryAdapter(app.builder.env)
for alias, target in app.config.intersphinx_aliases.items():
alias_domain, alias_name = alias
target_domain, target_name = target
try:
found = inventories.main_inventory[target_domain][target_name]
try:
inventories.main_inventory[alias_domain][alias_name] = found
except KeyError:
print("could not add to inv")
continue
except KeyError:
print("missed :(")
continue
def setup(app):
app.add_config_value("intersphinx_aliases", {}, "env")
app.connect("builder-inited", add_intersphinx_aliases_to_inv)
To use this, I paste the above code in my conf.py and add aliases to the intersphinx_aliases dictionary.

Matlab's cannot call Python code that imports statsmodels

This question concerns Matlab 2014b, Python 3.4 and Mac OS 10.10.
I have the following Python file tmp.py:
from statsmodels.tsa.arima_process import ArmaProcess
import numpy as np
def generate_AR_time_series():
arparams = np.array([-0.8])
maparams = np.array([])
ar = np.r_[1, -arparams]
ma = np.r_[1, maparams]
arma_process = ArmaProcess(ar, ma)
return arma_process.generate_sample(100)
I want to call generate_AR_time_series from Matlab so I used:
py.tmp.generate_AR_time_series()
which gave a vague error message
Undefined variable "py" or class "py.tmp.generate_AR_time_series".
To look into the problem further, I tried
tmp = py.eval('__import__(''tmp'')', struct);
which gave me a detailed but still obscured error message:
Python Error:
dlopen(/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so, 2): Symbol
not found: __gfortran_stop_numeric_f08
Referenced from: /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so
Expected in: /Applications/MATLAB_R2014b.app/sys/os/maci64/libgfortran.3.dylib
in /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so
I can call the function within Python just fine, so I guess the problem is with Matlab. From the detailed message, it seems that the problem lies in something is expected in the Matlab installation path, but of course Matlab installation path does not contain those things since they are third-party libraries for Python.
How to solve this problem?
Edit 1:
libgfortran.3.dylib can be found in a lot of places:
/Applications/MATLAB_R2014a.app/sys/os/maci64/libgfortran.3.dylib
/Applications/MATLAB_R2014b.app/sys/os/maci64/libgfortran.3.dylib
/opt/local/lib/gcc48/libgfortran.3.dylib
/opt/local/lib/gcc49/libgfortran.3.dylib
/opt/local/lib/libgcc/libgfortran.3.dylib
/Users/wdg/Documents/MATLAB/mcode/nativelibs/macosx/bin/libgfortran.3.dylib

Try:
setenv('DYLD_LIBRARY_PATH', '/usr/local/bin/');

For me, using the setenv approach from within MATLAB did not work. Also, MATLAB modifies the DYLD_LIBRARY_PATH variable during startup to include necessary libraries.
First, you have to make sure which version of gfortran scipy was linked against: in Terminal.app, enter otool -L /opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/site-packages/scipy/special/_ufuncs.so and look for 'libgfortran' in the output.
It worked for me to copy $(MATLABROOT)/bin/.matlab7rc.sh to my home directory and change the line LDPATH_PREFIX='' in the mac section (around line 195 in my case) to LDPATH_PREFIX='/opt/local/lib/gcc49', or whatever path to libgfortran you found above.
This ensures that /opt/local/lib/gcc49/libgfortran.3.dylib is found before the MATLAB version, but leaves other paths intact.