I am debugging a Python (3.5) program with PyCharm (PyCharm Community Edition 2016.2.2 ; Build #PC-162.1812.1, built on August 16, 2016 ; JRE: 1.8.0_76-release-b216 x86 ; JVM: OpenJDK Server VM by JetBrains s.r.o) on Windows 10.
The problem: when stopped at some breakpoints, the Debugger window is stuck at "Collecting data", which eventually timeout. (with Unable to display frame variables)
The data to be displayed is neither special, nor particularly large. It is somehow available to PyCharm since a conditional break point on some values of the said data works fine (the program breaks) -- it looks like the process to gather it for display only (as opposed to operational purposes) fails.
When I step into a function around the place I have my breakpoint, its data is displayed correctly. When I go up the stack (to the calling function, the one I stepped down from and where I wanted initially to have the breakpoint) - I am stuck with the "Collecting data" timeout again.
There have been numerous issues raised with the same point since at least 2005. Some were fixed, some not. The fixes were usually updates to the latest version (which I have).
Is there a general direction I can go to in order to fix or work around this family of problems?
EDIT: a year later the problem is still there and there is still no reaction from the devs/support after the bug was raised.
EDIT April 2018: It looks like the problem is solved in the 2018.1 version, the following code which was hanging when setting a breakpoint on the print line now works (I can see the variables):
import threading
def worker():
a = 3
print('hello')
threading.Thread(target=worker).start()
I had the same issue with Pycharm 2018.2 when working on a complex Flask project with SocketIO.
When I put a debug breakpoint inside the code and pressed the debug button, it stopped at the breakpoint, but the variables didn't load. It was just infinitely collecting data. I enabled Gevent compatibility and it resolved the issue. Here is where you can find the setting:
In case you landed here because you are using PyTorch (or any other deep learning library) and try to debug in PyCharm (torch 1.31, PyCharm 2019.2 in my case) but it's super slow:
Enable Gevent compatible in the Python Debugger settings as linkliu mayuyu pointed out. The problem might be caused due to debugging large deep learning models (BERT transformer in my case), but I'm not entirely sure about this.
I'm adding this answer as it's end of 2019 and this doesn't seem to be fixed yet. Further I think this is affecting many engineers using deep learning, so I hope my answer-formatting triggers their stackoverflow algorithm :-)
Note (June 2020):
While adding the Gevent compatible allows you to debug PyTorch models, it will prevent you from debug your Flask application in PyCharm! My breakpoints were not working anymore and it took me a while to figure out that this flag is the reason for it. So make sure to enable it only on a per-project base.
I also had this issue when I was working on code using sympy and the Python module 'Lea' aiming to calculate probability distributions.
The action I took that resolved the timeout issue was to change the 'Variables Loading Policy' in the debug setting from the default 'Asynchronously' to 'Synchronously'.
I think that this is caused by some classes having a default method __str__() that is too verbose. Pycharm calls this method to display the local variables when it hits a breakpoint, and it gets stuck while loading the string.
A trick I use to overcome this is manually editing the class that is causing the error and substitute the __str__() method for something less verbose.
As an example, it happens for pytorch _TensorBase class (and all tensor classes extending it), and can be solved by editing the pytorch source torch/tensor.py, changing the __str__() method as:
def __str__(self):
# All strings are unicode in Python 3, while we have to encode unicode
# strings in Python2. If we can't, let python decide the best
# characters to replace unicode characters with.
return str() + ' Use .numpy() to print'
#if sys.version_info > (3,):
# return _tensor_str._str(self)
#else:
# if hasattr(sys.stdout, 'encoding'):
# return _tensor_str._str(self).encode(
# sys.stdout.encoding or 'UTF-8', 'replace')
# else:
# return _tensor_str._str(self).encode('UTF-8', 'replace')
Far from optimum, but comes in hand.
UPDATE: The error seems solved in the last PyCharm version (2018.1), at least for the case that was affecting me.
I met the same problem when I try to run some Deep Learning scripts written by PyTorch (PyCharm 2019.3).
I finally figured out that the problem is I set num_workers in DataLoader to a large value (in my case 20).
So, in the debug mode, I would suggest to set num_workers to 1.
For me, the solution was removing manual watches every-time before starting to debug. If there were any existing manual watches in the "variables" window then it would remain stuck in "Collecting data...".
Using Odoo or Other Large Python Server
None of the above solution worked for me despite I tried all.
It normally works but saldomly gives this annoying Collecting data... or sometimes Timed Out....
The solution is to restart Pycharm and set less breakpoints as possible. after that it starts to work again.
I don't know way is doing that (maybe too many breakpoint) but it worked.
Related
Is there a way to exclude parameters being autocompleted in IntelliSense?
I have used IntelliJ before, and I recently switched over to VS Code. Upon switching and installing IntelliSense, I have had issues with some of the autocompletes. For example, on tab completion of the print function, I would just expect it to complete as:
print()
but it instead completes as:
print(sep=..., end=..., file=..., flush=...)
Only for some functions does the autocomplete include the parameter suggestions, and not for other, and I cant seem to figure out why. I tried looking through the settings but without much luck. Any help?
According to the documentation:rules, doing the following should add a simple rule to the iptables list of rules:
rule = iptc.Rule()
rule.src = "127.0.0.1"
rule.protocol = "udp"
rule.target = rule.create_target("ACCEPT")
match = rule.create_match("comment")
match.comment = "this is a test comment"
chain = iptc.Chain(iptc.Table(iptc.Table.FILTER), "INPUT")
chain.insert_rule(rule)
However, running this example, results in absolutely zero new rules.
I'm verifying this by doing:
iptables -L --line-number
Before I submit a bug issue, I'd like to know if anyone else has encountered this and if so, how you worked around it.
I'm running everything as root just to be on the safe side, I also tried verifying the rules by running another example code from the same section of the documentation:
table = iptc.Table(iptc.Table.FILTER)
for chain in table.chains:
print ("=======================")
print ("Chain ", chain.name)
for rule in chain.rules:
print ("Rule", "proto:", rule.protocol, "src:", rule.src, "dst:", \
rule.dst, "in:", rule.in_interface, "out:", rule.out_interface,)
print ("Matches:")
for match in rule.matches:
print (match.name)
print ("Target:"),
print (rule.target.name)
print ("=======================")
(modified slightly to work with Python3).
This was to make sure there wasn't an issue with the auto-commit, however, still the same results.
I will also point out that it did work for a short bit, for roughly 3 additions to iptables. And it might work to do a systemctl restart iptables, but I'd like to if possible - figure out why this is going wrong before I do the classic old "windows trick" of rebooting stuff. (nothing in journald/systemd either mentioning anything about iptables)
Seeing as #larsks couldn't reproduce the issue I dug a little further.
It appears that a system update had been performed (classic mistake, I apologize).
This causes the loaded kernel version to differ from the kernel module of iptables, there's some fixes in place that solves this issue using the iptables command so that you can still add rules.
However, using the lib python-iptables does not work.
What the actual difference is is beyond me, I dug a little bit but couldn't locate where this would cause an issue.
Rebooting the machine in this instance is the only (to me known) way to solve this issue unfortunately. This is so that the loaded kernel module and installed tools match the version they're working against.
(another solution would be to keep the old iptables command and libraries, meaning backing them up and pointing the libraries to the backed up version until a reboot can be made).
I'm working on a plug-in for Vim, and I'd like to test that it behaves correctly, under start-up, when users edit files e.t.c.
To do this, I'd like to start a terminal, and feed keys in to it.
I'm thinking of doing it all from a python script. Is there a way to do this?
In pseudo-python it might look something like this:
#start a terminal. Here konsole
konsole = os.system('konsole --width=200 --height=150')
#start vim in that terminal
konsole.feed_keys("vim\n")
#run the vim function to be tested
konsole.feed_keys(":let my_list = MyVimFunction()\n")
#save the return value to the file system
konsole.feed_keys(":writefile(my_list, '/tmp/result')\n")
#load result into python
with open('/tmp/result', 'r') as myfile:
data = myfile.read()
#validate the result
assertEqual('expect result', data)
I think you should verify the core functionality of your plugin inside Vim, using unit tests. There's a wide variety of Vim plugins, but most provide some additional mappings or commands, to be invoked by the user, and they usually leave behind some side effects in the buffer, or output, or opened windows. That can be verified from inside Vim. There are a various approaches for that, mine is the runVimTests test framework; the plugin page has links to several alternatives.
With the core functionality thus covered, there's little left to test "interactively". (I mean stuff like forgotten debug output, too long execution times, display mess-ups.) Since you're usually a heavy user of Vim and your plugin yourself, that mostly covers it.
Of course, if your plugin embeds itself tightly into Vim (like an "IDE for XXX"; though this is usually frowned upon), you may consider some external test driver. Maybe others will contribute pointers to some general-purpose, terminal-driven test frameworks. I'm almost sure such exist.
While I'm maintaining a plugin that permits to run unit tests on VimL functions and feed the quickfix window with the results, I use another couple of tools to check the state of the buffer after some actions, and even run the thing from travis -> vimrunner+rspec, and VimFlavour for installing the dependencies. (I vaguely remember a Python alternative inspired by vimrunner)
It mostly works well. Alas it uses the client-server feature and :redir (instead of the more recent execute() function). Even with the use of :silent, :redir catches noise which it returns to the client. Thus sometimes I fight tests that fail for very odd reasons. I also find myself inserting some pseudo-pauses to be sure that Vim has finished to interpret what I've feed it.
You'll find example of use in some of my plugins. See for instance lh-brackets or lh-cpp tests (.travis.yml file + .rspec/ directory + Rakefile + Gemfile + some helpers from vim-UT)
I'd like to be able to raise another application's Window using Python.
I did see this, which I suppose I could try:
X11: raise an existing window via command line?
However, I'd prefer to do it in Python if at all possible.
To activate another window, the right thing to do on the Xlib protocol layer is to send a _NET_ACTIVE_WINDOW message as described in the EWMH spec
http://standards.freedesktop.org/wm-spec/wm-spec-1.3.html
This could be done with python-xlib (presumably) or with gdk_window_focus() on a foreign GdkWindow using GDK through pygtk
_NET_ACTIVE_WINDOW is superior to XRaiseWindow() and has been in all the important WMs for many many years.
You should avoid XSetInputFocus() which will cause problems (especially if you get the timestamp wrong). The issue is that the WM can't intercept the SetInputFocus() so it causes weird race conditions and UI inconsistencies.
Really only _NET_ACTIVE_WINDOW works properly, which is why it was invented, because the previous hacks were bad.
There is a library called libwnck that will let you activate windows (among other things) but unfortunately it adds quite a lot of overhead because it always tracks all open windows from any app, even if you don't need to do that. However if you want to track windows from other apps anyway, then libwnck has a function to activate those windows that does the right thing and would be a good choice.
The strictly correct approach is to check for EWMH _NET_ACTIVE_WINDOW support (EWMH documents how to do this) and fall back to XRaiseWindow if the WM doesn't have _NET_ACTIVE_WINDOW. However, since any WM that's been actively worked on in the last many years has EWMH, lots of people are lazy about the fallback for legacy WMs.
You need to use python-xlib and call .circulate(Xlib.X.RaiseLowest) on the window object (which can be identified in many, many different ways -- can't guess which one is appropriate for you from the zero amount of info about it in your Q;-). For a great example of using python-xlib, check out the tinywm window manager -- after the C version, the author gives a Python version that takes about 30 non-blank, non-comment lines (for a usable, if tiny, window manager...!-).
You can have a look at the python ewmh package. Documentation contains examples, but here is how you can achieve what you want:
from ewmh import EWMH
import random
ewmh = EWMH()
# get every displayed windows
wins = ewmh.getClientList()
# let's active one window randomly
ewmh.setActiveWindow(random.choice(wins))
# flush requests - that's actually do the real job
ewmh.display.flush()
I'm noticing that even for system modules, code completion doesn't work too well.
For example, if I have a simple file that does:
import re
p = re.compile(pattern)
m = p.search(line)
If I type p., I don't get completion for methods I'd expect to see (I don't see search() for example, but I do see others, such as func_closure(), func_code()).
If I type m., I don't get any completion what so ever (I'd expect .groups(), in this case).
This doesn't seem to affect all modules.. Has any one seen this behaviour and knows how to correct it?
I'm running Vim 7.2 on WinXP, with the latest pythoncomplete.vim from vim.org (0.9), running python 2.6.2.
Completion for this kind of things is tricky, because it would need to execute the actual code to work.
For example p.search() could return None or a MatchObject, depending on the data that is passed to it.
This is why omni-completion does not work here, and probably never will. It works for things that can be statically determined, for example a module's contents.
I never got the builtin omnicomplete to work for any languages. I had the most success with pysmell (which seems to have been updated slightly more recently on github than in the official repo). I still didn't find it to be reliable enough to use consistently but I can't remember exactly why.
I've resorted to building an extensive set of snipMate snippets for my primary libraries and using the default tab completion to supplement.