I create a file called foo_module.py containing the following code:
import shelve, whichdb, os
from foo_package.g import g
g.shelf = shelve.open("foo_path")
g.shelf.close()
print whichdb.whichdb("foo_path") # => dbhash
os.remove("foo_path")
Next to that file I create a directory called foo_package than contains an empty __init__.py file and a file called g.py that just contains:
class g:
pass
Now when I run foo_module.py I get a weird error message:
Exception TypeError: "'NoneType' object is not callable" in ignored
But then, if I rename the directory from foo_package to foo, and change the import line in foo_module.py, I don't get any error. Wtf is going on here?
Running Python 2.6.4 on WinXP.
I think you've hit a minor bug in 2.6.4's code related to the cleanup at end of program. If you run python -v you can see exactly at what point of the cleanup the error comes:
# cleanup[1] foo_package.g
Exception TypeError: "'NoneType' object is not callable" in ignored
Python sets references to None during the cleanup at the end of program, and it looks like it's getting confused about the status of g.shelf. As a workaround you could set g.shelf = None after the close. I would also recommend opening a bug in Python's bug tracker!
After days of hair loss, I finally had success using an atexit function:
import atexit
...
cache = shelve.open(path)
atexit.register(cache.close)
It's most appropriate to register right after opening. This works with multiple concurrent shelves.
(python 2.6.5 on lucid)
This is indeed a Python bug, and I've posted a patch to the tracker issue you opened (thanks for doing that).
The problem is that shelve's del method calls its close method, but if the shelve module has already been through cleanup, the close method fails with the message you see.
You can avoid the message in your code by adding 'del g.shelf' after g.shelf.close. As long as g.shelf is the only reference to the shelf, this will result in CPython calling the shelve's del method right away, before the interpreter cleanup phase, and thus avoid the error message.
It seems to be an exception in a shutdown function registered by the shelve module. The "ignored" part is from the shutdown system, and might get its wording improved sometime, per Issue 6294. I'm still hoping for an answer on how to eliminate the exception itself, though...
for me a simple shelve.close() on an unclosed one did the job.
shelve.open('somefile') returns a "persistent dictionary for reading and writing" object which i used throughout the app's runtime.
when I terminated the app I received the "TypeError" Exception as mentioned.
I plased a 'close()' call in my termination sequence and that seemed to fix the problem.
e.g.
shelveObj = shelve.open('fileName')
...
shelveObj.close()
OverByThere commented on Jul 17, 2018
This seems to be fixable.
In short open /usr/lib/python3.5/weakref.py and change line 109 to:
def remove(wr, selfref=ref(self), _atomic_removal=_remove_dead_weakref):
And line 117 to:
_atomic_removal(d, wr.key)
Note you need to do this with spaces, not tabs as this will cause other errors.
Related
I tried to call python code from c, the example runs ok for sample code on my environment（python3.6）, but when I integrate it into my program, I got following error when I call Py_Initialize();:
...
sem_init: Success
Fatal Python error: Can't initialize threads for interpreter
Could you provide some clues to solve this problem?
It seems the error comes from here, but I am still not sure how to avoid this.
The failing code is
if (head_mutex == NULL)
Py_FatalError("Can't initialize threads for interpreter");
Searching the code back for head_mutex references finds
#define HEAD_INIT() (void)(head_mutex || (head_mutex = PyThread_allocate_lock()))
which is called right before the failing code.
So, the reason is that PyThread_allocate_lock returns NULL. There are a few different implementations for it in Python codebase depending on the OS and build flags, so you need to debug it or otherwise figure out which one is used in your case to track the error further to an OS call.
There is a function named sem_init in my program, which may conflict with the system library, the program runs ok after I modify the name of this function(but still not sure the reason).
The documentation in section 4.1 clearly states:
https://pylint.readthedocs.io/en/latest/faq.html#message-control
4.1 Is it possible to locally disable a particular message?
Yes, this feature has been added in Pylint 0.11. This may be done by adding “#pylint: disable=some-message,another-one” at the desired block level or at the end of the desired line of code
Great! but it doesn't work. Boo.
I get the the following pylint error for the following line of code
W: 26, 2: Redefining built-in 'zip' (redefined-builtin)
try:
from itertools import izip as zip # pylint: disable=bad-builtin
except ImportError:
pass
But pylint just complains even louder about my attempt to shut it up:
E: 26, 0: Bad option value 'bad-builtin' (bad-option-value)
I've also tried the error code # pylint: disable=W0141, that also produces a similar error.
Any idea what I'm doing wrong?
I have been in a similar situation.
Unsolvable pylint issue
class A:
pass
There are many warnings in pylint for the code above, but I want to talk about old-style-class.
In Python 2.7, you will get an old-style-class error.
Of course, you can change your code like this:
class A(object):
pass
However, you will receive a useless-object-inheritance warning in Python 3.
If you are writing a package compatible with python 2.7 and 3 and using pylint, then you are down.
Unavoidable bad-option-value
Yes, if it is accepted to disable either of old-style-class or useless-object-inheritance in a comment, you can go further.
In Python 2.7:
# pylint: disable=old-style-class
class A:
pass
In Python 3:
# pylint: disable=useless-object-inheritance
class A(object):
pass
Eventually, you will get a bad-option-value, just the same as this question.
Disable bad-option-value
I have tried, but bad-option-value can not be disabled locally in this case.
I have to disable bad-option-value in a pylint configuration file, like .pylintrc.
[TYPECHECK]
disable=bad-option-value
Note: My pylint version is 1.9.4 in python 2.7, 2.2.2 in python 3.
Ah, simple answer, it should be # pylint: disable=bad-option-value which is presented in the error message in parenthesis:
E: 26, 0: Bad option value 'bad-builtin' (bad-option-value)
When you get this message:
W: 26, 2: Redefining built-in 'zip' (redefined-builtin)
You have to disable the exact error message you are getting (the one in parenthesis):
try:
from itertools import izip as zip # pylint: disable=redefined-builtin
except ImportError:
pass
That seems to work fine in pylint 2.5.
It can be annoying if you are testing with multiple versions of python or different venvs and the same code base and you get different errors. Be sure you fix the version to one version across all your builds/tests. It sounds like that may have happened here (not sure where you got bad-builtin from).
As evidenced here, I have noticed a strange new behavior in Ansible.
I have an Ansible role named "degoss" which installs Goss along with some test files to a system, executes the tests, and then removes all traces of them. I'm using a callback plugin to format the output and a custom module to invoke Goss.
When tests fail, rather than getting sensible output, in newer versions of Ansible (around 2.3 as far as I can tell), I get the following unintuitive error message and no output whatsoever from my plugin:
[WARNING]: Failure using method (v2_runner_on_failed) in callback plugin
(<ansible.plugins.callback.default.CallbackModule object at 0x7fb5e92efd50>):
'module' object has no attribute 'dumps'
Now, when I search through ansible.plugins.callback.default.CallbackModule, I don't see any invocations of dumps. When I search through ansible.plugins.callback.CallbackBase, I do see three invocations of dumps. In my own callback plugin and custom module, I have try/except blocks wrapping all of my calls to json.dumps.
What's even worse is that I can't seem to replicate the issue consistently. I can't replicate it locally, and it only appears when a task has failed. My success/failure logic is below:
succeed(module, **result) if rc == 0 else fail(
module, "Goss Tests Failed.", **result
)
My succeed/fail methods are defined as such:
def succeed(module, **kwargs):
module.exit_json(changed=False, failed=False, goss_failed=False, **kwargs)
def fail(module, message, **kwargs):
module.fail_json(msg=message, failed=True, goss_failed=True, **kwargs)
Any ideas?
Here's a guess:
ansible's callback plugin loader creates a module in ansible.plugins.callback which happens to also contain a json module. When your callback plugin imports json you get ansible.plugins.callback.json, which doesn't contain dumps. You can fix this by adding from __future__ import absolute_import to your plugin.
The reason you can't reproduce it might be that you're using Python 3 locally, which defaults to absolute imports.
In addition to the accepted answer, just want to point out that there are two files into which the import statement needs to be added.
playbooks/callback_plugins/sqs.py
and
playbooks/callback_plugins/task_timing.py
I had missed updating the task_timing.py file and so the error persisted. Pointing this out here, in case someone else also faces the same issue.
Source :edx configuration repo - diff
I'm running lint as follows:
$ python -m pylint.lint m2test.py
with this code:
import M2Crypto
def f():
M2Crypto.RSA.new_pub_key("").as_pem(cipher=None).split("\n")
The lint output ends with:
Exception AttributeError: '_shutdown' in <module 'threading' from '/usr/lib/python2.7/site-packages/M2Crypto-0.21.1-py2.7-linux-x86_64.egg/M2Crypto/threading.pyc'> ignored
This code works fine when run (the above is actually a minimal test case; but the full version does work). The exception is ignored, but Bitten considers this a failure, so stops on this step.
I've tried adding 'M2Crypto.threading.init()'/'M2Crypto.threading.cleanup()' around the definition of the function, but that didn't fix the problem.
How can I prevent this problem from occurring?
I'm using M2Crypto 0.21.1, pylint 0.24 and Python 2.7 (also tried 2.7.2) on Debian Lenny x86_64.
The exception that you are seeing is caused by a bug in the astng package (presumably “Abstract Syntax Tree, Next Generation”?) which is a toolkit on which pylint depends, written by the same people. I should note in passing that I always encourage people to use pyflakes instead of pylint when possible, because it is quick, simple, fast, and predictable, whereas pylint tries to do several kinds of deep magic that are not only slow but that can get it into exactly this kind of trouble. :)
Here are the two packages on PyPI:
http://pypi.python.org/pypi/pylint
http://pypi.python.org/pypi/astng
And note that this problem had to be, necessarily, a bug in pylint and not in your code, because pylint does not run your code in order to produce its report — imagine the havoc that could be wreaked if it did (since code being linted might delete files, etcetera)! Since your code does not get run, no amount of caution, like protecting your call with threading init() or cleanup() functions, could possibly have prevented this error — unless the code snippets happened, for other reasons, to alter the behavior we are about to investigate.
So, on to your actual exception.
I had never actually heard of _shutdown before! A quick search of the Python standard library showed its definition in threading.py but not a call of the function from anywhere; only by searching the Python C source code did I discover where in pythonrun.c, during interpreter shutdown, the function is actually called:
static void
wait_for_thread_shutdown(void)
{
...
PyObject *threading = PyMapping_GetItemString(tstate->interp->modules,
"threading");
if (threading == NULL) {
/* threading not imported */
PyErr_Clear();
return;
}
result = PyObject_CallMethod(threading, "_shutdown", "");
if (result == NULL) {
PyErr_WriteUnraisable(threading);
}
...
}
Apparently it is some sort of cleanup function that the threading Standard Library module requires, and they have special-cased the Python interpreter itself to make sure that it gets called.
As you can see from the code above, Python quietly and without complaint handles the case where the threading module never gets imported during a program's run. But if threading does get imported, and still exists at shutdown time, then the interpreter looks inside for a _shutdown function and goes so far as to print an error message — and then return a non-zero exit status, the cause of your problems — if it cannot call it.
So we have to discover why the threading module exists but has no _shutdown method at the moment when pylint is done examining your program and Python is exiting. Some instrumention is called for. Can we print out what the module looks like as pylint exits? We can! The pylint/lint.py module, in its last few lines, runs its “main program” by instantiating a Run class it has defined:
if __name__ == '__main__':
Run(sys.argv[1:])
So I opened lint.py in my editor — one of the magnificent things about having each little project installed in a Python Virual Environment is that I can jump in and edit third-party code for quick experiments — and added the following print statement down at the bottom of the Run class's __init__() method:
sys.path.pop(0)
print "*****", sys.modules['threading'].__file__ # added by me!
if exit:
sys.exit(self.linter.msg_status)
I re-ran the command:
python -m pylint.lint m2test.py
And out came the __file__ string of the threading module:
***** /home/brandon/venv/lib/python2.7/site-packages/M2Crypto/threading.pyc
Well, look at that.
This is the problem!
According to this path, there actually exists an M2Crypto/threading.py module that, under all normal circumstances, should just be called M2Crypto.threading, and therefore sit in the sys.modules dictionary under the name:
sys.modules['M2Crypto.threading']
But somehow that file is also getting loaded as the main Python threading module, shadowing the official threading module that sits in the Standard Library. Because of this, the Python exit logic is quite correctly complaining that the Standard Library _shutdown() function is missing.
How could this happen? Top-level modules can only appear in paths that are listed explicitly in sys.path, not in sub-directories beneath them. This leads to a new question: is there any point during the pylint run that the …/M2Crypto/ directory itself is getting put on sys.path as though it contained top-level modules? Let's see!
We need more instrumentation: we need to have Python tell us the moment that a directory with M2Crypto in the name appears in sys.path. It will really slow things down, but let's add a trace function to pylint's __init__.py — because that is the first module that gets imported when you run -m pylint.lint — that will write an output file telling us, for every line of code executed, whether sys.path has any bad values in it:
def install_tracer():
import sys
output = open('mytracer.out', 'w')
def mytracer(frame, event, arg):
broken = any(p.endswith('M2Crypto') for p in sys.path)
output.write('{} {}:{} {}\n'.format(
broken, frame.f_code.co_filename, frame.f_lineno, event))
return mytracer
sys.settrace(mytracer)
install_tracer()
del install_tracer
Note how careful I am here: I define only one name in the module's namespace, and then carefully delete it to clean up after myself before I let pylint continue loading! And all of the resources that the trace function itself needs — namely, the sys module and the output open file — are available in the install_tracer() closure so that, from the outside, pylint looks exactly the same as always. Just in case anyone tries to introspect it, like pylint might!
This generates a file mytracer.out of about 800k lines, that each look something like this:
False /home/brandon/venv/lib/python2.7/posixpath.py:118 call
The False says that sys.path looks clean, the filename and line number are the line of code being executed, and call indicates what stage of execution the interpreter is in.
So does sys.path ever get poisoned? Let's look at just the first True or False on each line, and see how many successive lines start with each value:
$ awk '{print$1}' mytracer.out | uniq -c
607997 False
3173 True
4558 False
33217 True
4304 False
41699 True
2953 False
110503 True
52575 False
Wow! That's a problem! For runs of several thousand lines at a time, our test case is True, which means that the interpreter is running with …/M2Crypto/ — or some variant of a pathname with M2Crypto in it — on the path, where it should not be; only the directory that contains …/M2Crypto should ever be on the path. Looking for the first False to True transition in the file, I see this:
False /home/brandon/venv/lib/python2.7/site-packages/logilab/astng/builder.py:132 line
False /home/brandon/venv/lib/python2.7/posixpath.py:118 call
...
False /home/brandon/venv/lib/python2.7/posixpath.py:124 line
False /home/brandon/venv/lib/python2.7/posixpath.py:124 return
True /home/brandon/venv/lib/python2.7/site-packages/logilab/astng/builder.py:133 line
And looking at lines 132 and 133 in the builder.py file reveals our culprit:
130 # build astng representation
131 try:
132 sys.path.insert(0, dirname(path)) # XXX (syt) iirk
133 node = self.string_build(data, modname, path)
134 finally:
135 sys.path.pop(0)
Note the comment, which is part of the original code, not an addition of my own! Obviously, XXX (syt) iirk is an exclamation in this programmer's strange native language for the phrase, “put this module's parent directory on sys.path so that pylint will break mysteriously every time someone forces pylint to introspect a package with a threading sub-module.” It is, obviously, a very compact native language. :)
If you adjust the tracing module to watch sys.modules for the actual import of threading — an exercise I will leave to the reader — you will see that it happens when SocketServer, which is imported by some other Standard Library module during the analysis, in turn tries to innocently import threading.
So let us review what is happening:
pylint is dangerous magic.
As part of its magic, if it sees you import foo, then it runs off trying to find foo.py on disk, to parse it, and to predict whether you are loading valid or invalid names from its namespace.
[See my comment, below.] Because you call .split() on the return value of RSA.as_pem(), pylint tries to introspect the as_pem() method, which in turn uses the M2Crypto.BIO module, which in turn makes calls that induce pylint to import threading.
As part of loading any module foo.py, pylint throws the directory containing foo.py on sys.path, even if that directory is inside a package, and therefore gives modules in that directory the privilege of shadowing Standard Library modules of the same name during its analysis.
When Python exits, it is upset that the M2Crypto.threading library is sitting where threading belongs, because it wants to run the _shutdown() method of threading.
You should report this as a bug to the pylint / astng folks at logilab.org. Tell them I sent you.
If you decide to keep using pylint after it has done this to you, then there seem to be two solutions in this case: either don't inspect code that calls M2Crypto, or import threading during the pylint import process — by sticking import threading into the pylint/__init__.py, for example — so that the module gets the chance to grab the sys.modules['threading'] slot before pylint gets all excited and tries to let M2Crypto/threading.py grab the slot instead.
In conclusion, I think the author of astng says it best: XXX (syt) iirk. Indeed.
Many thanks to Brandon Craig Rhodes for having tracing this down and for such a detailed post.
I've removed the offending line from astng, code available from the hg repository until logilab-astng 0.23.0 is out. And I can confirm this fixes the OP's pb.
This looks more like a hack but I think it works. Copying the result of "as_pem()" and splitting it.
import M2Crypto
def f():
M2Crypto.RSA.new_pub_key("").as_pem(cipher=None)[:].split("\n")
I'm using Python 2.6.7, M2Crypto 0.21.1, pylint 0.23
I was unable to reproduce (pylint 0.24 and M2Crypto 0.21.1 on Ubuntu 11.04 64bit) but two suggestions:
Explicitly initialize threading:
import M2Crypto
def f():
M2Crypto.threading.init()
M2Crypto.RSA.new_pub_key("").as_pem(cipher=None).split("\n")
M2Crypto.threading.cleanup()
Or recompile without threading:
m2crypto = Extension(name = 'M2Crypto.__m2crypto',
sources = ['SWIG/_m2crypto.i'],
extra_compile_args = ['-DTHREADING'],
#extra_link_args = ['-Wl,-search_paths_first'], # Uncomment to build Universal Mac binaries
)
In my python script I would like to trap a "Data truncated for column 'xxx'" warning durnig my query using MySql.
I saw some posts suggesting the code below, but it doesn' work.
Do you know if some specific module must be imported or if some option/flag should be called before using this code?
Thanks all
Afeg
import MySQLdb
try:
cursor.execute(some_statement)
# code steps always here: No Warning is trapped
# by the code below
except MySQLdb.Warning, e:
# handle warnings, if the cursor you're using raises them
except Warning, e:
# handle warnings, if the cursor you're using raises them
Warnings are just that: warnings. They get reported to (usually) stderr, but nothing else is done. You can't catch them like exceptions because they aren't being raised.
You can, however, configure what to do with warnings, and turn them off or turn them into exceptions, using the warnings module. For instance, warnings.filterwarnings('error', category=MySQLdb.Warning) to turn MySQLdb.Warning warnings into exceptions (in which case they would be caught using your try/except) or 'ignore' to not show them at all. You can (and probably should) have more fine-grained filters than just the category.
Raise MySQL Warnings as errors:
import warnings, MySQLdb
warnings.filterwarnings('error', category=MySQLdb.Warning)
To ignore instead of raising an error, replace "error" with "ignore".
Handle them in a try-except block like:
try:
# a MySQL DB operation that raises a warning
# for example: a data truncated warning
except Warning as a_warning:
# do something here
I would try first to set the sql_mode. You can do this at the session level. If you execute a:
SET ##sql_mode:=TRADITIONAL;
(you could also set it at the server level, but you need access to the server to do that. and, that setting an still be overridden at the session level, so the bottom line is, always set it at the session level, immediately after establishing the connection)
then many things that are normally warnings become errors. I don't know how those manifest themselves at the python level, but the clear advantage is that the changes are not stored in the database. See: http://dev.mysql.com/doc/refman/5.1/en/server-sql-mode.html#sqlmode_traditional
If you want to trap it to ignore it see "Temporarily suppressing warnings" in Python's documentation:
https://docs.python.org/2/library/warnings.html#temporarily-suppressing-warnings
import warnings
with warnings.catch_warnings():
warnings.simplefilter("ignore")
# Put here your query raising a warning
Else, just read the doc about "warnings" module of Python, you can transform them into exception if you want to catch them later, etc... it's all here: https://docs.python.org/2/library/warnings.html
Just to add to Thomas Wouters reply, there is no need to import the warnings module to turn them into errors. Just run your script with "-W error" (or ignore) as flag for Python.
Have you tried using MySQL's SHOW WARNINGS command?
Stop using MySQLdb. It has such stupid behavior as truncating data and issuing only a warning. Use oursql instead.