I have a script that uses threads, that simply freezes after running for an hour or so, which makes it pretty difficult to debug.
Eventually, I found pyrasite, and basically, I could "hook" into the script as soon as I waited it out and it froze. It looks somewhat like this in terminal:
$ pyrasite-shell 3437
Pyrasite Shell 2.0
Connected to 'python3 code/MyTestScript.py'
Python 3.5.3 (default, Sep 27 2018, 17:25:39)
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(DistantInteractiveConsole)
>>> import sys, traceback
>>> for thread_id, frame in sys._current_frames().items():
... print('Stack for thread {}'.format(thread_id))
... traceback.print_stack(frame)
... print('')
...
Stack for thread 1888482416
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "<string>", line 167, in run
File "/usr/lib/python3.5/code.py", line 228, in interact
more = self.push(line)
File "/usr/lib/python3.5/code.py", line 250, in push
more = self.runsource(source, self.filename)
File "/usr/lib/python3.5/code.py", line 75, in runsource
self.runcode(code)
File "/usr/lib/python3.5/code.py", line 91, in runcode
exec(code, self.locals)
File "<console>", line 3, in <module>
Stack for thread 1898968176
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/pi/code/my_test_lib.py", line 187, in sendData
self.evexit.wait(sleep_time_s)
File "/usr/lib/python3.5/threading.py", line 549, in wait
signaled = self._cond.wait(timeout)
File "/usr/lib/python3.5/threading.py", line 297, in wait
gotit = waiter.acquire(True, timeout)
Stack for thread 1996019264
File "code/MyTestScript.py", line 1355, in <module>
main(sys.argv[1:])
File "code/MyTestScript.py", line 1245, in main
myObject.waitForJoin() # seems to work fine...
File "/home/pi/code/my_test_lib.py", line 251, in waitForJoin
self.myThread.join()
File "/usr/lib/python3.5/threading.py", line 1054, in join
self._wait_for_tstate_lock()
File "/usr/lib/python3.5/threading.py", line 1070, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
Stack for thread 1908950128
File "/usr/lib/python3.5/threading.py", line 882, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "/home/pi/code/my_testB_lib.py", line 511, in UpdateMyThread
time.sleep(_update_interval_sec)
This looks great, but what I cannot tell is, whether the script including all its threads is now paused (as if a breakpoint in gdb hass been set), or does the script keep running in the background?
Anyways, I know that in gdb, I could have just issued say thread 1, and then I'd end up in the corresponding stack frame, and then I could print local variables etc.
Here, however, I cannot tell how to change stack frames, nor how to switch threads, so that I could inspect variables.
Is this possible with pyrasite? And if not, is there any other library for Python3, that would allow me the same (that is, ability to hook into an uninstrumented script with threads that freezes), while allowing me to inspect any/all threads and stackframes for local variable values?
Starting from Pyrasite's included dump_stacks.py payload, I created the snippet below which dumps local variable names and values after each stack trace. However it doesn't seem to work correctly as a Pyrasite payload -- it looks like the globals get displayed instead for some reason? I'm new to Pyrasite myself so I'm not sure if this is a bug or if I'm not using it correctly. If I run pyrasite-shell instead and paste the code in directly, it works as expected (ignoring the extra trace from the shell thread itself).
If you need to see different info, you can either modify the code and re-paste it or just explore interactively within the pyrasite shell.
import sys, traceback
for thread, frame in sys._current_frames().items():
print('Thread 0x%x' % thread)
print('=' * 25)
traceback.print_stack(frame)
print('-' * 25)
for name, value in frame.f_locals.items():
print(f"{name}: {value}")
print()
This script might be what you are looking for:
import sys, traceback
for thread_id, frame in sys._current_frames().items():
print('Stack for thread {}'.format(thread_id))
stack = traceback.StackSummary.extract(traceback.walk_stack(frame), capture_locals=True)
stack.reverse()
traceback.print_list(stack)
print('')
It basically does the same thing as traceback.print_stack(frame), but it also supplies the capture_locals=True, which is false in the print_stack convenience method.
Related
I'm just trying to make a simple keyboard listener that prints whatever key I press (pynput.keyboard.Listener). Here is my code:
from pynput.keyboard import Listener
def on_press(k):
print(k)
with Listener(on_press=on_press) as lis:
lis.join()
But, when I run it just does nothing. So I press CTRL+C and it shows up these errors:
Traceback (most recent call last):
File "/Users/me/Desktop/py/KeyLogger/keylogger.py", line 8, in <module>
lis.join()
File "/usr/local/lib/python3.7/site-packages/pynput/_util/__init__.py", line 252, in join
super(AbstractListener, self).join(*args)
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1044, in join
self._wait_for_tstate_lock()
File "/usr/local/Cellar/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python3.7/threading.py", line 1060, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
Here with anaconda/Python3.8 instead of Python3.7:
Traceback (most recent call last):
File "/Users/me/Desktop/py/KeyLogger/keylogger.py", line 8, in <module>
lis.join()
File "/Users/me/opt/anaconda3/lib/python3.8/site-packages/pynput/_util/__init__.py", line 252, in join
super(AbstractListener, self).join(*args)
File "/Users/me/opt/anaconda3/lib/python3.8/threading.py", line 1011, in join
self._wait_for_tstate_lock()
File "/Users/me/opt/anaconda3/lib/python3.8/threading.py", line 1027, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
KeyboardInterrupt
It seems the error is when I tell the listener to start. I've been searching for similar issues (threading and join() issues, mostly) but I haven't found the answer to my problem. ¿Could it be a pynput problem?
Thanks.
I want to use IPython notebook's IPython.display.display from within a thread, as such:
def foo():
display(HTML('<div id="foobar">foobarbaz</div>'))
threading.Thread(target=foo).start()
IPython notebook spits out the following error:
Exception in thread Thread-17:
Traceback (most recent call last):
File "/usr/lib/python3.5/threading.py", line 914, in _bootstrap_inner
self.run()
File "/usr/lib/python3.5/threading.py", line 862, in run
self._target(*self._args, **self._kwargs)
File "<ipython-input-47-921e56bac735>", line 2, in foo
display(HTML('<div id="foobar">foobarbaz</div>'))
File "/usr/local/lib/python3.5/dist-packages/IPython/core/display.py", line 171, in display
publish_display_data(data=format_dict, metadata=md_dict)
File "/usr/local/lib/python3.5/dist-packages/IPython/core/display.py", line 121, in publish_display_data
metadata=metadata,
File "/usr/local/lib/python3.5/dist-packages/ipykernel/zmqshell.py", line 111, in publish
for hook in self.thread_local.hooks:
AttributeError: '_thread._local' object has no attribute 'hooks'
I figure IPython's display is not thread safe? Is there a way to get it to point to the main thread instead of the local thread? Or perhaps a way to get the IPython.display.DisplayHandle? Maybe the thing to do is to have a loop around a queue.Queue on the main thread and display objects as they get added into that queue?
Thank you!
You can use an Output widget to display your HTML. This worked for me:
import ipywidgets as widgets
from IPython.display import display, HTML
import threading
out = widgets.Output()
def foo():
with out:
display(HTML('<div id="foobar">foobarbaz</div>'))
threading.Thread(target=foo).start()
out
My program appears to run in a deadlock sometimes when I hit Ctrl+C. I'm trying to catch the keyboard interrupt and gracefully stop all running threads, but I'm not quite there yet.
I'm using a concurrent.futures.ThreadPoolExecutor. To find the location of the deadlock, I'm using the this receipe from ActiveState.
Now, here's the full stacktrace:
# ThreadID: 4856
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 884, in _bootstrap
self._bootstrap_inner()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 66, in run
self.stacktraces()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 80, in stacktraces
fout.write(stacktraces())
File: "C:\Users\niklas\repos\nodepy\craftr\lib\utils\tracer.py", line 28, in stacktraces
for filename, lineno, name, line in traceback.extract_stack(stack):
# ThreadID: 6068
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\runpy.py", line 193, in _run_module_as_main
"__main__", mod_spec)
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File: "C:\Users\niklas\repos\nodepy\craftr\.nodepy_modules\.bin\craftr.exe\__main__.py", line 9, in <module>
sys.exit(nodepy.main.main())
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\main.py", line 103, in main
ctx.load_module(ctx.main_module, do_init=False)
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\context.py", line 253, in load_module
module.load()
File: "c:\users\niklas\repos\nodepy\nodepy\nodepy\loader.py", line 43, in load
exec(code, vars(self.namespace))
File: "C:\Users\niklas\repos\nodepy\craftr\lib\main.py", line 110, in <module>
sys.exit(main())
File: "C:\Users\niklas\repos\nodepy\craftr\lib\main.py", line 106, in main
return backend.build_main(backend_args, session, module)
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 194, in build_main
executor.run(actions)
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 171, in run
self.wait()
File: "C:\Users\niklas\repos\nodepy\craftr\lib\build_backends\default.py", line 137, in wait
self.pool.shutdown(wait=True)
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\concurrent\futures\thread.py", line 144, in shutdown
t.join()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 1056, in join
self._wait_for_tstate_lock()
File: "c:\users\niklas\appdata\local\programs\python\python36\lib\threading.py", line 1072, in _wait_for_tstate_lock
elif lock.acquire(block, timeout):
I can't make sense from this traceback. It appears that Thread._wait_for_tstate_lock() never returns (I checked multiple times, it always hangs at that line). There is no thread running other than the main thread (6068) and the tracer thread (4856).
I don't quite understand the implementation details of threading.Thread. What could cause Thread._tstate_lock.acquire() to block indefinitely?
Update 2017/11/07 -- 01:45am CEWT
This seems to happen when pool.shutdown() is called multiple times...
I'm not 100% certain it's the reason you're seeing this as you are using Windows, but I encountered this on Linux with Python 3.6 in a similar scenario.
I was using .shutdown() on a concurrent.futures.ThreadPoolExecutor and the program would seem to hang up.
Sometimes it would finally exit after 30-60 seconds.
Ctrl-C always resulted in a traceback showing it was sitting in _wait_for_tstate_lock()
Note: in Python 3 a second Ctrl-C actually exits
My problem occurred when the function submitted was using time.sleep() in a loop.
Looking at the HtmlFileTracer implementation in current nodepy code on github I see a similar scenario to what I was doing, (continuously loop and sleep for an interval unless some sort of flag was set)
I am unable to run django nose tests, all processes fail with this traceback:
Traceback (most recent call last):
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/Users/jeffquinn/Code/Nuna/data-intrawebVE/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 652, in runner
keyboardCaught, shouldStop, loaderClass, resultClass, config)
File "/Users/jeffquinn/Code/Nuna/data-intrawebVE/lib/python2.7/site-packages/nose/plugins/multiprocess.py", line 669, in __runner
config.plugins.begin()
File "/Users/jeffquinn/Code/Nuna/data-intrawebVE/lib/python2.7/site-packages/nose/plugins/manager.py", line 99, in __call__
return self.call(*arg, **kw)
File "/Users/jeffquinn/Code/Nuna/data-intrawebVE/lib/python2.7/site-packages/nose/plugins/manager.py", line 167, in simple
result = meth(*arg, **kw)
File "/Users/jeffquinn/Code/Nuna/data-intrawebVE/lib/python2.7/site-packages/nose/plugins/cover.py", line 156, in begin
self.coverInstance.exclude('#pragma[: ]+[nN][oO] [cC][oO][vV][eE][rR]')
AttributeError: 'NoneType' object has no attribute 'exclude'
Here are the relevant bits from pip freeze:
Django==1.5.1
MySQL-python==1.2.5
South==0.7.6
Werkzeug==0.8.3
boto==2.8.0
dj-database-url==0.2.1
django-appconf==0.6
django-compressor==1.2
django-extensions==1.0.3
django-json-field==0.5.5
django-nose==1.1
django-storages==1.1.6
django-waffle==0.9.0
djangorestframework==2.1.17
nose==1.3.1
Anyone else seen this? I've seen it mentioned in this pull request (https://github.com/nose-devs/nose/pull/710) but not clear what the solution was.
I just went into the nose source code and commented out lines 156-158.
#self.coverInstance.exclude('#pragma[: ]+[nN][oO] [cC][oO][vV][eE][rR]')
#self.coverInstance.load()
#self.coverInstance.start()
All my tests run fine now. O_o
I am relatively new to Python, and at the same time, am attempting to install mod_dav_svn into my Apache web server. I am looking to get some idea of the scope of the error I'm receiving.
At the command line, I type in 'sudo yum install mod_dav_svn' and receive this output:
Loaded plugins: fastestmirror
Determining fastest mirrors
Traceback (most recent call last):
File "/usr/bin/yum", line 29, in ?
yummain.user_main(sys.argv[1:], exit_code=True)
File "/usr/share/yum-cli/yummain.py", line 229, in user_main
errcode = main(args)
File "/usr/share/yum-cli/yummain.py", line 104, in main
result, resultmsgs = base.doCommands()
File "/usr/share/yum-cli/cli.py", line 339, in doCommands
self._getTs(needTsRemove)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 101, in _getTs
self._getTsInfo(remove_only)
File "/usr/lib/python2.4/site-packages/yum/depsolve.py", line 112, in _getTsInfo
pkgSack = self.pkgSack
File "/usr/lib/python2.4/site-packages/yum/init.py", line 591, in
pkgSack = property(fget=lambda self: self._getSacks(),
File "/usr/lib/python2.4/site-packages/yum/init.py", line 434, in _getSacks
self.repos.populateSack(which=repos)
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 223, in populateSack
self.doSetup()
File "/usr/lib/python2.4/site-packages/yum/repos.py", line 71, in doSetup
self.ayum.plugins.run('postreposetup')
File "/usr/lib/python2.4/site-packages/yum/plugins.py", line 176, in run
func(conduitcls(self, self.base, conf, **kwargs))
File "/usr/lib/yum-plugins/fastestmirror.py", line 181, in postreposetup_hook
all_urls = FastestMirror(all_urls).get_mirrorlist()
File "/usr/lib/yum-plugins/fastestmirror.py", line 333, in get_mirrorlist
self._poll_mirrors()
File "/usr/lib/yum-plugins/fastestmirror.py", line 376, in _poll_mirrors
pollThread.start()
File "/usr/lib/python2.4/threading.py", line 416, in start
_start_new_thread(self.__bootstrap, ())
thread.error: can't start new thread
The only other question I could find with a similar error was this one: https://stackoverflow.com/search?q=can%27t+start+new+thread+python, but I'm not sure it has to do with running too many threads since I am the only one using the server, and this is one of the first python commands I have used. Could someone point me in the right direction, or towards some material that may help me troubleshoot the issue? Thanks!
Disable yum-fastermirror and you should be able to complete the install.
yum --disableplugin=fastestmirror update
Are you on a virtual machine?