I'm trying to add cross-references to external API into my documentation but I'm facing three different behaviors.
I am using sphinx(1.3.1) with Python(2.7.3) and my intersphinx mapping is configured as:
{
'python': ('https://docs.python.org/2.7', None),
'numpy': ('http://docs.scipy.org/doc/numpy/', None),
'cv2' : ('http://docs.opencv.org/2.4/', None),
'h5py' : ('http://docs.h5py.org/en/latest/', None)
}
I have no trouble writing a cross-reference to numpy API with :class:`numpy.ndarray` or :func:`numpy.array` which gives me, as expected, something like numpy.ndarray.
However, with h5py, the only way I can have a link generated is if I omit the module name. For example, :class:`Group` (or :class:`h5py:Group`) gives me Group but :class:`h5py.Group` fails to generate a link.
Finally, I cannot find a way to write a working cross-reference to OpenCV API, none of these seems to work:
:func:`cv2.convertScaleAbs`
:func:`cv2:cv2.convertScaleAbs`
:func:`cv2:convertScaleAbs`
:func:`convertScaleAbs`
How to properly write cross-references to external API, or configure intersphinx, to have a generated link as in the numpy case?
In addition to the detailed answer from #gall, I've discovered that intersphinx can also be run as a module:
python -m sphinx.ext.intersphinx 'http://python-eve.org/objects.inv'
This outputs nicely formatted info. For reference: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/intersphinx.py#L390
I gave another try on trying to understand the content of an objects.inv file and hopefully this time I inspected numpy and h5py instead of only OpenCV's one.
How to read an intersphinx inventory file
Despite the fact that I couldn't find anything useful about reading the content of an object.inv file, it is actually very simple with the intersphinx module.
from sphinx.ext import intersphinx
import warnings
def fetch_inventory(uri):
"""Read a Sphinx inventory file into a dictionary."""
class MockConfig(object):
intersphinx_timeout = None # type: int
tls_verify = False
class MockApp(object):
srcdir = ''
config = MockConfig()
def warn(self, msg):
warnings.warn(msg)
return intersphinx.fetch_inventory(MockApp(), '', uri)
uri = 'http://docs.python.org/2.7/objects.inv'
# Read inventory into a dictionary
inv = fetch_inventory(uri)
# Or just print it
intersphinx.debug(['', uri])
File structure (numpy)
After inspecting numpy's one, you can see that keys are domains:
[u'np-c:function',
u'std:label',
u'c:member',
u'np:classmethod',
u'np:data',
u'py:class',
u'np-c:member',
u'c:var',
u'np:class',
u'np:function',
u'py:module',
u'np-c:macro',
u'np:exception',
u'py:method',
u'np:method',
u'np-c:var',
u'py:exception',
u'np:staticmethod',
u'py:staticmethod',
u'c:type',
u'np-c:type',
u'c:macro',
u'c:function',
u'np:module',
u'py:data',
u'np:attribute',
u'std:term',
u'py:function',
u'py:classmethod',
u'py:attribute']
You can see how you can write your cross-reference when you look at the content of a specific domain. For example, py:class:
{u'numpy.DataSource': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html#numpy.DataSource',
u'-'),
u'numpy.MachAr': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.MachAr.html#numpy.MachAr',
u'-'),
u'numpy.broadcast': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html#numpy.broadcast',
u'-'),
...}
So here, :class:`numpy.DataSource` will work as expected.
h5py
In the case of h5py, the domains are:
[u'py:attribute', u'std:label', u'py:method', u'py:function', u'py:class']
and if you look at the py:class domain:
{u'AttributeManager': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/attr.html#AttributeManager',
u'-'),
u'Dataset': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/dataset.html#Dataset',
u'-'),
u'ExternalLink': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/group.html#ExternalLink',
u'-'),
...}
That's why I couldn't make it work as numpy references. So a good way to format them would be :class:`h5py:Dataset`.
OpenCV
OpenCV's inventory object seems malformed. Where I would expect to find domains there is actually 902 function signatures:
[u':',
u'AdjusterAdapter::create(const',
u'AdjusterAdapter::good()',
u'AdjusterAdapter::tooFew(int',
u'AdjusterAdapter::tooMany(int',
u'Algorithm::create(const',
u'Algorithm::getList(vector<string>&',
u'Algorithm::name()',
u'Algorithm::read(const',
u'Algorithm::set(const'
...]
and if we take the first one's value:
{u'Ptr<AdjusterAdapter>': (u'OpenCV',
u'2.4',
u'http://docs.opencv.org/2.4/detectorType)',
u'ocv:function 1 modules/features2d/doc/common_interfaces_of_feature_detectors.html#$ -')}
I'm pretty sure it is then impossible to write OpenCV cross-references with this file...
Conclusion
I thought intersphinx generated the objects.inv based on the content of the documentation project in an standard way, which seems not to be the case.
As a result, it seems that the proper way to write cross-references is API dependent and one should inspect a specific inventory object to actually see what's available.
An additional way to inspect the objects.inv file is with the sphobjinv module.
You can search local or even remote inventory files (with fuzzy matching). For instance with scipy:
$ sphobjinv suggest -t 90 -u https://docs.scipy.org/doc/scipy/reference/objects.inv "signal.convolve2d"
Remote inventory found.
:py:function:`scipy.signal.convolve2d`
:std:doc:`generated/scipy.signal.convolve2d`
Note that you may need to use :py:func: and not :py:function: (I'd be happy to know why).
How to use OpenCV 2.4 (cv2) intersphinx
Inspired by #Gall's answer, I wanted to compare the contents of the OpenCV & numpy inventory files. I couldn't get sphinx.ext.intersphinx.fetch_inventory to work from ipython, but the following does work:
curl http://docs.opencv.org/2.4/objects.inv | tail -n +5 | zlib-flate -uncompress > cv2.inv
curl https://docs.scipy.org/doc/numpy/objects.inv | tail -n +5 | zlib-flate -uncompress > numpy.inv
numpy.inv has lines like this:
numpy.ndarray py:class 1 reference/generated/numpy.ndarray.html#$ -
whereas cv2.inv has lines like this:
cv2.imread ocv:pyfunction 1 modules/highgui/doc/reading_and_writing_images_and_video.html#$ -
So presumably you'd link to the OpenCV docs with :ocv:pyfunction:`cv2.imread` instead of :py:function:`cv2.imread`. Sphinx doesn't like it though:
WARNING: Unknown interpreted text role "ocv:pyfunction".
A bit of Googling revealed that the OpenCV project has its own "ocv" sphinx domain: https://github.com/opencv/opencv/blob/2.4/doc/ocv.py -- presumably because they need to document C, C++ and Python APIs all at the same time.
To use it, save ocv.py next to your Sphinx conf.py, and modify your conf.py:
sys.path.insert(0, os.path.abspath('.'))
import ocv
extensions = [
'ocv',
]
intersphinx_mapping = {
'cv2': ('http://docs.opencv.org/2.4/', None),
}
In your rst files you need to say :ocv:pyfunc:`cv2.imread` (not :ocv:pyfunction:).
Sphinx prints some warnings (unparseable C++ definition: u'cv2.imread') but the generated html documentation actually looks ok with a link to http://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#cv2.imread. You can edit ocv.py and remove the line that prints that warning.
The accepted answer no longer works in the new version (1.5.x) ...
import requests
import posixpath
from sphinx.ext.intersphinx import read_inventory
uri = 'http://docs.python.org/2.7/'
r = requests.get(uri + 'objects.inv', stream=True)
r.raise_for_status()
inv = read_inventory(r.raw, uri, posixpath.join)
Stubborn fool that I am, I used 2to3 and the Sphinx deprecated APIs chart to revive #david-röthlisberger's ocv.py-based answer so it'll work with Sphinx 2.3 on Python 3.5.
The fixed-up version is here:
https://gist.github.com/ssokolow/a230b27b7ea4a31f7fb40621e6461f9a
...and the quick version of what I did was:
Run 2to3 -w ocv.py && rm ocv.py.bak
Cycle back and forth between running Sphinx and renaming functions to their replacements in the chart. I believe these were the only changes I had to make on this step:
Directive now has to be imported from docutils.parsers.rst
Replace calls to l_(...) with calls to _(...) and remove the l_ import.
Replace calls to env.warn with calls to log.warn where log = sphinx.util.logging.getLogger(__name__).
Then, you just pair it with this intersphinx definition and you get something still new enough to be relevant for most use cases:
'cv2': ('https://docs.opencv.org/3.0-last-rst/', None)
For convenience, I made a small extension for aliasing intersphinx cross references. This is useful as sometimes the object inventory gets confused when an object from a submodule is imported from a package's __init__.py.
See also https://github.com/sphinx-doc/sphinx/issues/5603
###
# Workaround of
# Intersphinx references to objects imported at package level can"t be mapped.
#
# See https://github.com/sphinx-doc/sphinx/issues/5603
intersphinx_aliases = {
("py:class", "click.core.Group"):
("py:class", "click.Group"),
("py:class", "click.core.Command"):
("py:class", "click.Command"),
}
def add_intersphinx_aliases_to_inv(app):
from sphinx.ext.intersphinx import InventoryAdapter
inventories = InventoryAdapter(app.builder.env)
for alias, target in app.config.intersphinx_aliases.items():
alias_domain, alias_name = alias
target_domain, target_name = target
try:
found = inventories.main_inventory[target_domain][target_name]
try:
inventories.main_inventory[alias_domain][alias_name] = found
except KeyError:
print("could not add to inv")
continue
except KeyError:
print("missed :(")
continue
def setup(app):
app.add_config_value("intersphinx_aliases", {}, "env")
app.connect("builder-inited", add_intersphinx_aliases_to_inv)
To use this, I paste the above code in my conf.py and add aliases to the intersphinx_aliases dictionary.
Related
The following flags were defined in a misc_fun.py file to include machine and directories info:
import tensorflow as tf
flags = tf.app.flags
FLAGS = flags.FLAGS
# definitions
flags.DEFINE_string(
'DEFAULT_IN',
'~/PycharmProjects/myNN/Data/',
"""Default input folder.""")
...
It worked fine in TensorFlow 1.0 - 1.4 versions (with Pycharm). After updating to TensorFlow 1.5.-rc0, the following error occurred:
Usage:
from misc_fun import FLAGS
FLAGS.DEFAULT_IN = FLAGS.DEFAULT_DOWNLOAD # change default input folder
Error:
UnparsedFlagAccessError: Trying to access flag --DEFAULT_DOWNLOAD before flags were parsed.
However print(FLAGS) worked fine, which gives:
misc_fun:
--DEFAULT_DOWNLOAD: default download folder for large datasets.
(default: '/home/username/Downloads/Data/')
--DEFAULT_IN: default input folder.
(default: '~/PycharmProjects/myNN/Data/')
...
I tried FLAGS = flags.FLAGS(sys.argv), resulting in the following error:
UnrecognizedFlagError: Unknown command line flag 'f'
Although there is a workaround using the class object, I wonder what could be the problem here.
I have tried adding the following line below.
tf.app.flags.DEFINE_string('f', '', 'kernel')
This solution is different from others in that it is simple and easy to try. You just need to add this into your code, and it doesn't change your system. Please let me know if this solution helps solve other people's problems.
The reference for this solution is from a Chinese website: https://blog.csdn.net/qq_39956625/article/details/80500291
With 1.5.0-rc0 the Tensorflow maintainers have switched tf.app.flags to the flags module from abseil. Unfortunately, it is not 100% API compatible to the previous implementation. I worked around your problem with something like
remaining_args = FLAGS([sys.argv[0]] + [flag for flag in sys.argv if flag.startswith("--")])
assert(remaining_args == [sys.argv[0]])
before accessing the FLAGS object the first time.
Alternatively you can use FLAGS(sys.argv, known_only=True) to parse all related flags (the ones defined using tf.app.flags.DEFINE_xxx). This will release any other args that are not known. Useful if you have some command line arguments that are not related to TF.
I am trying to put a reference to the following link into one of my reST documents: https://docs.python.org/3/library/stdtypes.html#typecontextmanager. I am trying to use a :ref: inline directive instead of a link.
I ran python -m sphinx.ext.intersphinx https://docs.python.org/3/objects.inv. The result shows, among other things:
...
std:label
23acks Acknowledgements : whatsnew/2.3.html#acks
23section-other Other Changes and Fixes : whatsnew/2.3.html#section-other
...
typebytearray Bytearray Objects : library/stdtypes.html#typebytearray
typebytes Bytes Objects : library/stdtypes.html#typebytes
typecontextmanager Context Manager Types : library/stdtypes.html#typecontextmanager
typeiter Iterator Types : library/stdtypes.html#typeiter
typememoryview Memory Views : library/stdtypes.html#typememoryview
...
The bolded URL is exactly what I am looking for since my intersphinx_mapping looks like this:
intersphinx_mapping = {
'python': ('https://docs.python.org/3', None),
}
I use the following directive:
:ref:`context manager <python:typecontextmanager>`
This seems to be pointing to the right label, but I get the following warning:
WARNING: undefined label: python:typecontextmanager (if the link has no caption the label must precede a section header)
The :ref: gets replaced by the string context manager, but there is no link.
What am I missing?
I am using sphinx 1.6.3 on an Anaconda install of Python 3.6.2
Note 1
I have the same problem using :ref:`with <python:with>`, which is supposed to point to https://docs.python.org/3/reference/compound_stmts.html#with, according to the inventory line (also under std:label):
with The with statement : reference/compound_stmts.html#with
I am guessing that a solution to the main question will most likely solve this as well.
Note 2
Probably not 100% relevant, but I can link to :py:meth:`~contextmanager.__enter__` in the same section without any problems.
Either of these work for me.
:ref:`python:typecontextmanager`
:ref:`typecontextmanager <python:typecontextmanager>`
Note that if you use angle brackets around the target, then you must include a title.
Here's how they render for me:
Context Manager Types
typecontextmanager
As an aside, I recently added context managers to Pyramid's documentation in its Glossary, which I think does a good job of explaining what they are. Here's the reST source:
:ref:`With Statement Context Managers <python:context-managers>`
:ref:`with <python:with>`
I'm trying to configure the HTMLTestRunner to output to a single file when multiple test classes are being called, but after much reading I've been unable to achieve this.
An example of what I'm doing is:
class TestOne(unittest.TestCase):
def test_one_is_one(self):
one = 1
self.assertEqual(1, one)
class TestTwo(unittest.TestCase):
def test_two_is_two(self):
two = 2
self.assertEqual(2, two)
I'm then adding these into a test suite and running the HTMLTestRunner as below:
output = 'C:\\Reports\TestReport.html'
test_suite = unittest.TestSuite(unittest.TestLoader().loadTestsFromModule(Tests))
runner = HTMLTestRunner(output=output)
runner.run(test_suite)
However when running like this I'm getting two HTML files generated, one for TestOne and another for TestTwo.
I've looked around and other examples of this I've come across use:
with open(output, 'wb') as o:
runner = HTMLTestRunner(output=o)
runner.run(test_suite)
However this doesn't appear to be supported anymore by HTMLTestRunner.
Is what I'm after possible?
I really like the reports generated, however I don't really want to have to deal with lots of small HTML files that need to be either merged together or viewed separately.
Additional info:
I'm using Python 3.5 with HTMLTestRunner 1.0.3
I know this is an old ticket, but thought it was worth sharing the following information.
I wanted to do the same as the original question, a single HTML report for the entire test suite. In the latest version of HtmlTestRunner (installed using pip install html-testRunner), the following option is available:
combine_test_reports=True.
Which can be used as follows:
html_runner = HtmlTestRunner.HTMLTestRunner(
stream=output_file,
combine_reports=True,
report_title='HTML test runner report')
Lw246,
I see two htmltestrunners. the 1.0.3 version you used seems to be different, and is still in beta version. The author calls it html-testrunner with a '-'
The original htmlrunner of tungwaiyip is called 'htmltestrunner' without the '-' , and it has been forked with new version. you can see it here: https://github.com/dash0002/HTMLTestRunner.
You can also see the 2 different htmltestrunners here:
https://pypi.python.org/pypi?%3Aaction=search&term=htmltestrunner&submit=search
In addition, there is also an htmltestrunner2 :)
Background
I've grown tired of the issue with pylint not being able to import files when you use namespace packages and divide your code-base into separate folders. As such I started digging into the astNG source-code which has been identified as the source of the trouble (see bugreport 8796 on astng). At the heart of the issue seems to be the use of pythons own imp.find_module in the process of finding imports.
What happens is that the import's first (sub)package - a in import a.b.c - is fed to find_module with a None path. Whatever path comes back is then fed into find_module the next pass in the look up loop where you try to find b in the previous example.
Pseudo-code from logilab.common.modutils:
path = None
while import_as_list:
try:
_, found_path, etc = find_module(import_as_list[0], path)
#exception handling and checking for a better version in the .egg files
path = [found_path]
import_as_list.pop(0)
The Problem
This is what's broken: you only get the first best hit from find_module, which may or may not have your subpackages in it. If you DON'T find the subpackages, you have no way to back out and try the next one.
I tried explicitly using sys.path instead of None, so that the result could be removed from the path list and a second attempt be made, but python's module finder is clever enough that there doesn't have to be an exact match in the paths, making this approach unusable - to the best of my knowledge anyway.
Teary-eyed Plea
Is there an alternative to find_modules which will return ALL possible matches or take an exclude list? I'm also open to completely different solutions. Preferably not patching python by hand, but it wouldn't be impossible - at least for a local solution.
(Caveat emptor: I'm running python 2.6 and for reasons of current company policy can't upgrade, suggestions for p3k etc won't get marked as accepted unless it's the only answer.)
Since Python 2.5, the right way to do this is with pkgutil.iter_modules() (for a flat list) or pkgutil.walk_packages() (for a subpackage tree). Both are fully compatible with namespace packages.
For example, if I wanted to find just the subpackages/submodules of 'jmb', I would do:
import jmb, pkgutil
for (module_loader, name, ispkg) in pkgutil.iter_modules(jmb.__path__, 'jmb.'):
# 'name' will be 'jmb.foo', 'jmb.bar', etc.
# 'ispkg' will be true if 'jmb.foo' is a package, false if it's a module
You can also use iter_modules or walk_packages to walk all the modules on sys.path; see the docs linked above for details.
I've grown tired of this limitation in PyLint too.
I don't know a replacement for imp.find_modules(), but I think I found another way to deal with namespace packages in PyLint. See my comment on the bug report you linked to (http://www.logilab.org/ticket/8796).
The idea is to use pkg_resources to find namespace packages. Here's my addition to logilab.common.modutils._module_file(), just after while modpath:
while modpath:
if modpath[0] in pkg_resources._namespace_packages and len(modpath) > 1:
module = sys.modules[modpath.pop(0)]
path = module.__path__
This not very refined and only handles top-level namespace packages though.
warning + disclaimer: not tested yet!
before:
for part in parts:
modpath.append(part)
curname = '.'.join(modpath)
# ...
if module is None:
mp_file, mp_filename, mp_desc = imp.find_module(part, path)
module = imp.load_module(curname, mp_file, mp_filename, mp_desc)
after: - thanks pjeby for mentioning pkgutil!
for part in parts:
modpath.append(part)
curname = '.'.join(modpath)
# ...
if module is None:
# + https://stackoverflow.com/a/14820895/611007
# # mp_file, mp_filename, mp_desc = imp.find_module(part, path)
# # module = imp.load_module(curname, mp_file, mp_filename, mp_desc)
import pkgutil
mp_file = None
for loadr,name,ispkg in pkgutil.iter_modules(path=path,prefix='.'.join(modpath[:-1])+'.'):
if name.split('.')[-1] == part:
if not hasattr(loadr,'path') and hasattr(loadr,'archive'):
# with zips `name` was like '.somemodule'
# it gives `RuntimeWarning: Parent module '' not found while handling absolute import`
# I expect the name I need to be 'somemodule'
# TODO: I don't know why python does this or what the correct usage is.
# https://stackoverflow.com/questions/2267984/
if name and name[0] == '.':
name = name[1:]
ldr= loadr.find_module(name,loadr.archive)
module = ldr.load_module(name)
break
imploader= loadr.find_module(name,loadr.path)
mp_file,mp_filename,mp_desc= imploader.file,imploader.filename,imploader.etc
module = imploader.load_module(imploader.fullname)
break
if module is None:
raise ImportError
I'm trying to create a python program (using pyUNO ) to make some changes on a OpenOffice calc sheet.
I've launched previously OpenOffice on "accept" mode to be able to connect from an external program. Apparently, should be as easy as:
import uno
# get the uno component context from the PyUNO runtime
localContext = uno.getComponentContext()
# create the UnoUrlResolver
resolver = localContext.ServiceManager.createInstanceWithContext(
"com.sun.star.bridge.UnoUrlResolver", localContext)
# connect to the running office
ctx = resolver.resolve("uno:socket,host=localhost,port=2002;"
"urp;StarOffice.ComponentContext")
smgr = ctx.ServiceManager
# get the central desktop object
DESKTOP =smgr.createInstanceWithContext("com.sun.star.frame.Desktop", ctx)
#The calling it's not exactly this way, just to simplify the code
DESKTOP.loadComponentFromURL('file.ods')
But I get an AttributeError when I try to access loadComponentFromURL. If I make a dir(DESKTOP), I've see only the following attributes/methods:
['ActiveFrame', 'DispatchRecorderSupplier', 'ImplementationId', 'ImplementationName',
'IsPlugged', 'PropertySetInfo', 'SupportedServiceNames', 'SuspendQuickstartVeto',
'Title', 'Types', 'addEventListener', 'addPropertyChangeListener',
'addVetoableChangeListener', 'dispose', 'disposing', 'getImplementationId',
'getImplementationName', 'getPropertySetInfo', 'getPropertyValue',
'getSupportedServiceNames', 'getTypes', 'handle', 'queryInterface',
'removeEventListener', 'removePropertyChangeListener', 'removeVetoableChangeListener',
'setPropertyValue', 'supportsService']
I've read that there are where a bug doing the same, but on OpenOffice 3.0 (I'm using OpenOffice 3.1 over Red Hat5.3). I've tried to use the workaround stated here, but they don't seems to be working.
Any ideas?
It has been a long time since I did anything with PyUNO, but looking at the code that worked last time I ran it back in '06, I did my load document like this:
def urlify(path):
return uno.systemPathToFileUrl(os.path.realpath(path))
desktop.loadComponentFromURL(
urlify(tempfilename), "_blank", 0, ())
Your example is a simplified version, and I'm not sure if you've removed the extra arguments intentionally or not intentionally.
If loadComponentFromURL isn't there, then the API has changed or there's something else wrong, I've read through your code and it looks like you're doing all the same things I have.
I don't believe that the dir() of the methods on the desktop object will be useful, as I think there's a __getattr__ method being used to proxy through the requests, and all the methods you've printed out are utility methods used for the stand-in object for the com.sun.star.frame.Desktop.
I think perhaps the failure could be that there's no method named loadComponentFromURL that has exactly 1 argument. Perhaps giving the 4 argument version will result in the method being found and used. This could simply be an impedance mismatch between Python and Java, where Java has call-signature method overloading.
This looks like issue 90701: http://www.openoffice.org/issues/show_bug.cgi?id=90701
See also http://piiis.blogspot.com/2008/10/pyuno-broken-in-ooo-30-with-system.html and http://udk.openoffice.org/python/python-bridge.html