I have a Flyte task function like this:
#task
def do_stuff(framework_obj):
framework_obj.get_outputs() # This calls Types.Blob.fetch(some_uri)
Trying to load a blob URI using flytekit.sdk.types.Types.Blob.fetch, but getting this error:
ERROR:flytekit: Exception when executing No temporary file system is present. Either call this method from within the context of a task or surround with a 'with LocalTestFileSystem():' block. Or specify a path when calling this function. Note: Cleanup is not automatic when a path is specified.
I can confirm I can load blobs using with LocalTestFileSystem(), in tests, but when actually trying to run a workflow, I'm not sure why I'm getting this error, as the function that calls blob-processing is decorated with #task so it's definitely a Flyte Task. I also confirmed that the task node exists on the Flyte web console.
What path is the error referencing and how do I call this function appropriately?
Using Flyte Version 0.16.2
Could you please give a bit more information about the code? This is flytekit version 0.15.x? I'm a bit confused since that version shouldn't have the #task decorator. It should only have #python_task which is an older API. If you want to use the new python native typing API you should install flytekit==0.17.0 instead.
Also, could you point to the documentation you're looking at? We've updated the docs a fair amount recently, maybe there's some confusion around that. These are the examples worth looking at. There's also two new Python classes, FlyteFile and FlyteDirectory that have replaced the Blob class in flytekit (though that remains what the IDL type is called).
(would've left this as a comment but I don't have the reputation to yet.)
Some code to help with fetching outputs and reading from a file output
#task
def task_file_reader():
client = SynchronousFlyteClient("flyteadmin.flyte.svc.cluster.local:81", insecure=True)
exec_id = WorkflowExecutionIdentifier(
domain="development",
project="flytesnacks",
name="iaok0qy6k1",
)
data = client.get_execution_data(exec_id)
lit = data.full_outputs.literals["o0"]
ctx = FlyteContext.current_context()
ff = TypeEngine.to_python_value(ctx, lv=lit,
expected_python_type=FlyteFile)
with open(ff, 'rb') as fh:
print(fh.readlines())
I am not sure if this is the rightest place to ask this but I try anyway. I have to integrate the Python aiortc library in an embedded system which uses Yocto for building the entire environment.
Because there is no recipe for such library, I've generated one using pipoe following this tutorial.
Using the command: pipoe --package aiortc --version 0.9.28 --python python3 I have generated few bb files inside a custom layer such as: python3-aioice_xx.xx.bb, python3-aiortc_xx.xx.bb, python3-cffi_xx.xx.bb and so on (I think those are dependencies).
Now I wanted to compile this recipe to check if everything is sorted with the command: bitbake python3-aiortc, it seems to proceed well and to find all the required files until this error occurs and I don't know how to address it. Can someone help me?
I think that some relevant lines are:
ERROR: python3-aiortc-0.9.28-r0 do_configure: Function failed: do_configure
ERROR: Do not try to fetch `cffi>=1.0.0' for building. Please add its native recipe to DEPENDS.
ERROR: python3-pyee-7.0.1-r0 do_configure: Function failed: do_configure
Have a look to the complete log that I have linked for further info.
===EDIT===
Added the python3-aiortc recipe content.
SUMMARY = "An implementation of WebRTC and ORTC"
HOMEPAGE = "https://github.com/aiortc/aiortc"
AUTHOR = "Jeremy Lainé <jeremy.laine#m4x.org>"
LICENSE = "BSD"
LIC_FILES_CHKSUM = "file://LICENSE;md5=907b5e856b2e6bcd8a3cc8d338a6166f"
SRC_URI = "https://files.pythonhosted.org/packages/1a/34/d9c8e19b4d5157721a5b77750116c6bb6355f1d85b92e7de491269b9ee51/aiortc-0.9.28.tar.gz"
SRC_URI[md5sum] = "50dc651d643b16c95b0e1ad259baeb51"
SRC_URI[sha256sum] = "4a41122e043a75c93a80dbb6d884b6f7cf27b774ebdef226d819a2c3a997c550"
S = "${WORKDIR}/aiortc-0.9.28"
RDEPENDS_${PN} = "python3-aioice python3-av python3-cffi python3-crc32c python3-cryptography python3-pyee python3-pylibsrtp"
inherit setuptools3
Your recipe depends on other recipes. I am guessing you don't have them.
If you have them you probably need to add them to IMAGE_INSTALL_append in local.conf
If you don't have them, you can try to search for them in OpenEmbedded Index, for example:
https://layers.openembedded.org/layerindex/branch/master/recipes/?q=cffi
I'm running iPython (Jupyter) through Anaconda, on a Mac Sierra, through iTerm, with $SHELL=bash - if I've missed any helpful set up details, just let me know.
I love the $HISTCONTROL aspect of bash, mentioned here. To sum that answer up: when traversing history (aka hitting the up arrow), it's helpful to remove duplicate entries so you don't scroll past the same command multiple times, and this is accomplished with $HISTCONTROL=ignoreboth:erasedups.
Is there any equivalent for this inside the Python interpreter (or iPython, specifically)? I have readline installed and feel like that's a good place to start, but nothing jumped out as obviously solving the problem, and I would've thought this was built in somewhere.
Through some deep-diving into IPython, sifting through poorly-explained and/or deprecated documentation, I've pieced together a solution that seems to work fine, though I'm sure it's not optimal for a number of reasons, namely:
it runs a GROUP BY query on the history database every time I run a line in IPython
it doesn't take care to clean up/coordinate the database tables - I only modify history, but ignore output_history and sessions tables
I put the following in a file (I named it dedupe_history.py, but name is irrelevant) inside $HOME/.ipython/profile_default/startup:
import IPython
import IPython.core.history as H
## spews a UserWarning about locate_profile() ... seems safe to ignore
HISTORY = H.HistoryAccessor()
def dedupe_history():
query = ("DELETE FROM history WHERE rowid NOT IN "
"(SELECT MAX(rowid) FROM history GROUP BY source)")
db = HISTORY.db
db.execute(query)
db.commit()
def set_pre_run_cell_event():
IPython.get_ipython().events.register("pre_run_cell", dedupe_history)
## dedupe history at start of new session - maybe that's sufficient, YMMV
dedupe_history()
## run dedupe history every time you run a command
set_pre_run_cell_event()
I'm trying to add cross-references to external API into my documentation but I'm facing three different behaviors.
I am using sphinx(1.3.1) with Python(2.7.3) and my intersphinx mapping is configured as:
{
'python': ('https://docs.python.org/2.7', None),
'numpy': ('http://docs.scipy.org/doc/numpy/', None),
'cv2' : ('http://docs.opencv.org/2.4/', None),
'h5py' : ('http://docs.h5py.org/en/latest/', None)
}
I have no trouble writing a cross-reference to numpy API with :class:`numpy.ndarray` or :func:`numpy.array` which gives me, as expected, something like numpy.ndarray.
However, with h5py, the only way I can have a link generated is if I omit the module name. For example, :class:`Group` (or :class:`h5py:Group`) gives me Group but :class:`h5py.Group` fails to generate a link.
Finally, I cannot find a way to write a working cross-reference to OpenCV API, none of these seems to work:
:func:`cv2.convertScaleAbs`
:func:`cv2:cv2.convertScaleAbs`
:func:`cv2:convertScaleAbs`
:func:`convertScaleAbs`
How to properly write cross-references to external API, or configure intersphinx, to have a generated link as in the numpy case?
In addition to the detailed answer from #gall, I've discovered that intersphinx can also be run as a module:
python -m sphinx.ext.intersphinx 'http://python-eve.org/objects.inv'
This outputs nicely formatted info. For reference: https://github.com/sphinx-doc/sphinx/blob/master/sphinx/ext/intersphinx.py#L390
I gave another try on trying to understand the content of an objects.inv file and hopefully this time I inspected numpy and h5py instead of only OpenCV's one.
How to read an intersphinx inventory file
Despite the fact that I couldn't find anything useful about reading the content of an object.inv file, it is actually very simple with the intersphinx module.
from sphinx.ext import intersphinx
import warnings
def fetch_inventory(uri):
"""Read a Sphinx inventory file into a dictionary."""
class MockConfig(object):
intersphinx_timeout = None # type: int
tls_verify = False
class MockApp(object):
srcdir = ''
config = MockConfig()
def warn(self, msg):
warnings.warn(msg)
return intersphinx.fetch_inventory(MockApp(), '', uri)
uri = 'http://docs.python.org/2.7/objects.inv'
# Read inventory into a dictionary
inv = fetch_inventory(uri)
# Or just print it
intersphinx.debug(['', uri])
File structure (numpy)
After inspecting numpy's one, you can see that keys are domains:
[u'np-c:function',
u'std:label',
u'c:member',
u'np:classmethod',
u'np:data',
u'py:class',
u'np-c:member',
u'c:var',
u'np:class',
u'np:function',
u'py:module',
u'np-c:macro',
u'np:exception',
u'py:method',
u'np:method',
u'np-c:var',
u'py:exception',
u'np:staticmethod',
u'py:staticmethod',
u'c:type',
u'np-c:type',
u'c:macro',
u'c:function',
u'np:module',
u'py:data',
u'np:attribute',
u'std:term',
u'py:function',
u'py:classmethod',
u'py:attribute']
You can see how you can write your cross-reference when you look at the content of a specific domain. For example, py:class:
{u'numpy.DataSource': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.DataSource.html#numpy.DataSource',
u'-'),
u'numpy.MachAr': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.MachAr.html#numpy.MachAr',
u'-'),
u'numpy.broadcast': (u'NumPy',
u'1.9',
u'http://docs.scipy.org/doc/numpy/reference/generated/numpy.broadcast.html#numpy.broadcast',
u'-'),
...}
So here, :class:`numpy.DataSource` will work as expected.
h5py
In the case of h5py, the domains are:
[u'py:attribute', u'std:label', u'py:method', u'py:function', u'py:class']
and if you look at the py:class domain:
{u'AttributeManager': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/attr.html#AttributeManager',
u'-'),
u'Dataset': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/dataset.html#Dataset',
u'-'),
u'ExternalLink': (u'h5py',
u'2.5',
u'http://docs.h5py.org/en/latest/high/group.html#ExternalLink',
u'-'),
...}
That's why I couldn't make it work as numpy references. So a good way to format them would be :class:`h5py:Dataset`.
OpenCV
OpenCV's inventory object seems malformed. Where I would expect to find domains there is actually 902 function signatures:
[u':',
u'AdjusterAdapter::create(const',
u'AdjusterAdapter::good()',
u'AdjusterAdapter::tooFew(int',
u'AdjusterAdapter::tooMany(int',
u'Algorithm::create(const',
u'Algorithm::getList(vector<string>&',
u'Algorithm::name()',
u'Algorithm::read(const',
u'Algorithm::set(const'
...]
and if we take the first one's value:
{u'Ptr<AdjusterAdapter>': (u'OpenCV',
u'2.4',
u'http://docs.opencv.org/2.4/detectorType)',
u'ocv:function 1 modules/features2d/doc/common_interfaces_of_feature_detectors.html#$ -')}
I'm pretty sure it is then impossible to write OpenCV cross-references with this file...
Conclusion
I thought intersphinx generated the objects.inv based on the content of the documentation project in an standard way, which seems not to be the case.
As a result, it seems that the proper way to write cross-references is API dependent and one should inspect a specific inventory object to actually see what's available.
An additional way to inspect the objects.inv file is with the sphobjinv module.
You can search local or even remote inventory files (with fuzzy matching). For instance with scipy:
$ sphobjinv suggest -t 90 -u https://docs.scipy.org/doc/scipy/reference/objects.inv "signal.convolve2d"
Remote inventory found.
:py:function:`scipy.signal.convolve2d`
:std:doc:`generated/scipy.signal.convolve2d`
Note that you may need to use :py:func: and not :py:function: (I'd be happy to know why).
How to use OpenCV 2.4 (cv2) intersphinx
Inspired by #Gall's answer, I wanted to compare the contents of the OpenCV & numpy inventory files. I couldn't get sphinx.ext.intersphinx.fetch_inventory to work from ipython, but the following does work:
curl http://docs.opencv.org/2.4/objects.inv | tail -n +5 | zlib-flate -uncompress > cv2.inv
curl https://docs.scipy.org/doc/numpy/objects.inv | tail -n +5 | zlib-flate -uncompress > numpy.inv
numpy.inv has lines like this:
numpy.ndarray py:class 1 reference/generated/numpy.ndarray.html#$ -
whereas cv2.inv has lines like this:
cv2.imread ocv:pyfunction 1 modules/highgui/doc/reading_and_writing_images_and_video.html#$ -
So presumably you'd link to the OpenCV docs with :ocv:pyfunction:`cv2.imread` instead of :py:function:`cv2.imread`. Sphinx doesn't like it though:
WARNING: Unknown interpreted text role "ocv:pyfunction".
A bit of Googling revealed that the OpenCV project has its own "ocv" sphinx domain: https://github.com/opencv/opencv/blob/2.4/doc/ocv.py -- presumably because they need to document C, C++ and Python APIs all at the same time.
To use it, save ocv.py next to your Sphinx conf.py, and modify your conf.py:
sys.path.insert(0, os.path.abspath('.'))
import ocv
extensions = [
'ocv',
]
intersphinx_mapping = {
'cv2': ('http://docs.opencv.org/2.4/', None),
}
In your rst files you need to say :ocv:pyfunc:`cv2.imread` (not :ocv:pyfunction:).
Sphinx prints some warnings (unparseable C++ definition: u'cv2.imread') but the generated html documentation actually looks ok with a link to http://docs.opencv.org/2.4/modules/highgui/doc/reading_and_writing_images_and_video.html#cv2.imread. You can edit ocv.py and remove the line that prints that warning.
The accepted answer no longer works in the new version (1.5.x) ...
import requests
import posixpath
from sphinx.ext.intersphinx import read_inventory
uri = 'http://docs.python.org/2.7/'
r = requests.get(uri + 'objects.inv', stream=True)
r.raise_for_status()
inv = read_inventory(r.raw, uri, posixpath.join)
Stubborn fool that I am, I used 2to3 and the Sphinx deprecated APIs chart to revive #david-röthlisberger's ocv.py-based answer so it'll work with Sphinx 2.3 on Python 3.5.
The fixed-up version is here:
https://gist.github.com/ssokolow/a230b27b7ea4a31f7fb40621e6461f9a
...and the quick version of what I did was:
Run 2to3 -w ocv.py && rm ocv.py.bak
Cycle back and forth between running Sphinx and renaming functions to their replacements in the chart. I believe these were the only changes I had to make on this step:
Directive now has to be imported from docutils.parsers.rst
Replace calls to l_(...) with calls to _(...) and remove the l_ import.
Replace calls to env.warn with calls to log.warn where log = sphinx.util.logging.getLogger(__name__).
Then, you just pair it with this intersphinx definition and you get something still new enough to be relevant for most use cases:
'cv2': ('https://docs.opencv.org/3.0-last-rst/', None)
For convenience, I made a small extension for aliasing intersphinx cross references. This is useful as sometimes the object inventory gets confused when an object from a submodule is imported from a package's __init__.py.
See also https://github.com/sphinx-doc/sphinx/issues/5603
###
# Workaround of
# Intersphinx references to objects imported at package level can"t be mapped.
#
# See https://github.com/sphinx-doc/sphinx/issues/5603
intersphinx_aliases = {
("py:class", "click.core.Group"):
("py:class", "click.Group"),
("py:class", "click.core.Command"):
("py:class", "click.Command"),
}
def add_intersphinx_aliases_to_inv(app):
from sphinx.ext.intersphinx import InventoryAdapter
inventories = InventoryAdapter(app.builder.env)
for alias, target in app.config.intersphinx_aliases.items():
alias_domain, alias_name = alias
target_domain, target_name = target
try:
found = inventories.main_inventory[target_domain][target_name]
try:
inventories.main_inventory[alias_domain][alias_name] = found
except KeyError:
print("could not add to inv")
continue
except KeyError:
print("missed :(")
continue
def setup(app):
app.add_config_value("intersphinx_aliases", {}, "env")
app.connect("builder-inited", add_intersphinx_aliases_to_inv)
To use this, I paste the above code in my conf.py and add aliases to the intersphinx_aliases dictionary.
Edit: So apparantly my install wasn't working. This pointed me to a mailing list Here where I figured out which commands I was missing. I have the answer for the update below. Now that I think about it, it does make sense. I just wish they'd put this somewhere simple on the dev pages.
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
yb.buildTransaction()
yb.processTransaction()
I'm trying to perform an update using yumbase when a server first boots with my kickstart script. At the moment I have a rather crude python subprocess to do "yum update" and would like to make this better.
I'm trying to hook into Yumbase, but the documentation is quite scarce. I have had a look at both the source code and documentation on this page: http://yum.baseurl.org/wiki/5MinuteExamples
I've figured out how to list all packages but not the ones that need updating using an SO answer from 2008: Given an rpm package name, query the yum database for updates
I've also figured out it's a very simple 3-line process to install a new package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.install(name='aws-cli')
However the following doesn't work to "update" the package:
yb = yum.YumBase()
yb.conf.assumeyes = True
yb.update(name='aws-cli')
So what I need is:
1: A way to list the packages that need updating, much like "yum check-update"
2: Install the packages above using "yum update"
From what I can see in the yum code, it doesn't seem to be written to be used as a library. The code you gave is not the right way to do it, there's much else happening behind the scenes.
Basically, as of yum-3.4.3, the process looks like this:
->yummain.__main__
<trap KeyboardInterrupt>
->yummain.user_main(sys.argv[1:], exit_code=True)
<check YUM_PROF,YUM_PDB envvars, wrap the following into debugger/profiler if set>
->yummain.main(args)
<set up locale, set up logging>
-><create a YumBaseCli (child of YumBase & YumOutput)>
<incl. fill a list field with YumCommand instances of known commands>
->cli.YumBaseCli.getOptionsConfig()
<parse args into the YumBaseCli instance, includes initializing plugins>
<obtain global yum lock>
<check write permissions for current dir>
->cli.YumBaseCli.doCommands()
<select a YumCommand from the list>
->YumCommand.needTs/needTsRemove if needed
->YumCommand.doCommand(self, self.basecmd, self.extcmds)
<handle errors & set error code if any>
'Resolving Dependencies'
->cli.YumBaseCli.buildTransaction()
<check for an unfinished transaction>
<resolve deps using the info written by the YumCommand into the object>
<honor clean_requirements_on_remove, protected_packages,
protected_multilib, perform some checks>
<handle errors & set error code if any>
'Dependencies Resolved'
->cli.YumBaseCli.doTransaction()
<download, transaction check, transaction test, transaction
using the info in the object>
<handle errors & set error code if any>
'Complete!'
<release global yum lock>
sys.exit(error_code)
As you can see, the main working sequence is embedded directly into main so you can only replicate this logic in-process by running it directly:
yummain.main(<sequence of cmdline arguments>)
Which is just the same as running a separate process minus process isolation.