I am getting a series of warnings that occurs during interpreter shutdown of a Python 2.7 script when using libvirt. This is reproducible on a variety of distros including python2-libvirt 3.7.0-1 on Fedora 27 to libvirt-python 3.2.0-1 on Centos 7.4.1708. The warnings I get when the script is exitings is:
Exception AttributeError: "'NoneType' object has no attribute 'virDomainFree'" in <bound method virDomain.__del__ of <libvirt.virDomain object at 0x7f34a194ee10>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'virDomainFree'" in <bound method virDomain.__del__ of <libvirt.virDomain object at 0x7f34a194ed90>> ignored
Exception AttributeError: "'NoneType' object has no attribute 'virConnectClose'" in <bound method virConnect.__del__ of <libvirt.virConnect object at 0x7f34a176a590>> ignored
Drilling down into the library, it seems to be an issue with assumptions in destructor order as in this code from libvirt.py:
class virDomain(object):
def __del__(self):
if self._o is not None:
libvirtmod.virDomainFree(self._o)
self._o = None
libvirtmod is a global created by an import at the top of the libvirt.py module. When the __del__() destructor for virDomain is finally run, libvirtmod has been replaced by the value None causing the code above to fail with a warning. We have been using this Python module for some time now, but only recently have these warning started showing up after we refactored the code quite heavily. What can I do to suppress these warnings from standard error or avoid the situation from occurring? Is there a way to ensure objects from libvirt.py are cleaned up before libvirtmod.so goes away?
We test on a variety of distros and would like to stick with using the stock (but updated) packages that come with the distro.
The standard trick is to equip __del__ with default arguments that hold whatever global values it needs (here, libvirtmod itself would suffice). There is a variation using weakref, which can also benefit from default arguments.
Related
I have a c++ class with several constructors:
MyClass(const std::string& configfilename);
MyClass(const MyClass& other);
I have python bindings for this class that were generated with cppyy - I don't do this myself, this is all part of the framework I'm using (CERN ROOT, in case you're wondering).
Now, I have a piece of python code that instantiates my class, with a nice try-except block:
try:
obj = MyClass(args.config)
except ConfigFileNotFoundError:
print("config file "+args.config+" was not found!")
exit(0)
Now, to test, I'm executing this with a wrong config file. But what I get is roughly this:
TypeError: none of the 2 overloaded methods succeeded. Full details:
MyClass(const std::string&) => ConfigFileNotFoundError
MyClass::MyClass(const MyClass&) => TypeError
So I'm wondering:
Since cppyy seems to handle function overloading with a try/except block, is there any reasonable way to do error handling for such applications?
I'd love to actually get the ConfigFileNotFoundError to handle it properly, rather than getting this TypeError. Also, what determines the actual error class I get in the end - does it depend on the order in which the overloads appear in the header file?
Any help, suggestions or pointers on where to find more information on this would be highly appreciated.
cppyy doesn't use try/except for overload resolution, hence there are also no __context__ and __cause__ set. To be more precise: the C++ exception is not an error that occurs during a handler. Rather, as-yet unresolved overloads are prioritized, then tried in order, with no distinction made between a Python failure (e.g. from an argument conversion) or a C++ failure (any exception that was automatically converted into a Python exception). This is a historic artifact predating run-time template instantiation and SFINAE: it allowed for more detailed run-time type matching in pre-instantiated templates.
If all overloads fail (Python or C++), the collected errors are summarized. Python still requires an exception type, however, and if the exception types across the collected types differ, a generic TypeError is raised, with a message string made up of all the collected exceptions. This is what happens here: there is ConfigFileNotFoundError raised by C++ in one overload and TypeError from argument conversion failure in the other.
There's an improvement now in the cppyy repo; to be released with 2.3.0, where in clear cases such as this one (a single overload succeeding in argument match but failing in the callee), you'll get the actual ConfigFileNotFoundError instance as long as its class is publicly derived from std::exception (but I think it already does, otherwise the error report you posted would have looked quite different).
(Note that CERN's ROOT contains an old fork of cppyy that has quite a bit diverged; you'll have to request them for a separate update there if that fork matters to you.)
I create a LazyLoader class which downloads files from S3 to the local file system, but only if they were not downloaded before. This class deletes the stuff it downloaded once it gets destroyed:
def __del__(self):
"""Remove downloaded files when this object is deleted."""
for downloaded_file in self.loaded_data.values():
os.remove(downloaded_file)
The tests pass, but after pytest tells me that the tests passed I get:
Exception ignored in: <bound method LazyLoader.__del__ of LazyLoader({})>
Traceback (most recent call last):
File "my_lazy_loader.py", line 47, in __del__
TypeError: 'NoneType' object is not callable
Line 47 is os.remove(downloaded_file). So os.remove is None evaluates to True. Why? How can I fix that?
If I move the code in __del__ to a method clean(), I don't have that problem.
https://docs.python.org/3/reference/datamodel.html#object.del
'del()' can be executed during interpreter shutdown. As a consequence, the global variables it needs to access (including other modules) may already have been deleted or set to None. Python guarantees that globals whose name begins with a single underscore are deleted from their module before other globals are deleted; if no other references to such globals exist, this may help in assuring that imported modules are still available at the time when the del() method is called.
If it's just unit tests, use tearDown method.
If the problem occurs when running your application and you want to do a cleanup at the end of the program, consider using atexit.register.
You might also use tempfile module for temporary files automatically removed when closed.
If your use-case allows that, turn LazyLoader into a context manager.
If none of the above applies, simply call clean() explicitly. It will follow explicit is better than implicit rule.
I try to dynamically load a jupyter notebook into a module and found a working code example in https://github.com/axil/import-ipynb/blob/master/import_ipynb.py. However, since this implementation uses some deprecated functions and I want to consolidate some common functionality into a single package I want to implement my own version. However, when I try to transform jupyter magic code into python code before execution, I get this strange error.
(Pdb) self
<IPython.core.inputsplitter.IPythonInputSplitter object at 0x102198c50>
(Pdb) IPythonInputSplitter
<class 'IPython.core.inputsplitter.IPythonInputSplitter'>
(Pdb) type(self)
<class 'IPython.core.inputsplitter.IPythonInputSplitter'>
(Pdb) IPythonInputSplitter is type(self)
False
(Pdb) super(IPythonInputSplitter, self)
*** TypeError: super(type, obj): obj must be an instance or subtype of type
Here is some context for the offending code:
for cell in notebook.cells:
if cell.cell_type == "code":
code = self.shell.input_transformer_manager.transform_cell(cell.source)
From the error type I don't think this is a problem special to ipython, but I don't quite get it
This Problem was unrelated to the provided Code
The Problem occurred because the module that contained IPython.core.inputsplitter.IPythonInputSplitter was reloaded using importlib.reload between instantiation and the typecheck.
pylint complain about pandas class instances::
I have db instance that has data (a panda Dataframe) as instance.
If I call e.g. iloc or shape on it::
cols = db.data.shape
xxx = db.data.iloc[1:4, 0:9]
pylint complain about::
E: 36,18: Instance of 'TextFileReader' has no 'iloc' member (no-member)
E: 92,30: Instance of 'TextFileReader' has no 'shape' member (no-member)
E: 92,30: Instance of 'tuple' has no 'shape' member (no-member)
I've try How do I get PyLint to recognize numpy members? and Disabling Pylint no member- E1101 error for specific libraries with no success.
This seems to be an open issue in pylint that has been marked high priority as of March 14th 2022.
The issue stems from the underlying AST engine that pylint uses that has a limit to the number of inferences it can make at runtime as a bid to improve performance.
For posteriority in case the link above ever dies, the workaround is as follows:
pylint some_file_to_lint.py --init-hook "import astroid; astroid.context.InferenceContext.max_inferred = 500"
Or in .pylintrc:
[MASTER]
init-hook = "import astroid; astroid.context.InferenceContext.max_inferred = 500"
I have a complex Python program. I'm trying to use the multiprocess Pool to parallelize it. I get the error message
PicklingError: Can't pickle <type 'function'>: attribute lookup __builtin__.function failed.
The traceback shows the statemen return send(obj)
My hypothesis is that its the "obj" that is causing the problem and that I need to make it pickle-able.
How can I determine which object is the cause of the problem? The program is complex and simply guessing might take a long time.
Why not just add some logging right before return send(obj)? At a very minimum, this could help:
print repr(obj)
return send(obj)
Or, to only print when it results in an error:
try:
return send(obj)
except:
print '***', repr(obj)
raise
The error you're seeing could be caused by passing the wrong kind of function to the multiprocessing.Pool methods. The passed function must be directly importable from its parent module. It cannot be a method of a class, for instance.