I am looking for the following (IMHO, very important) feature:
Suppose I have two functions fa() and fb(), both of them has a breakpoint.
I am now stopped in the breakpoint in fa function.
In the interactive debugger console I am calling fb().
I want to stop in fb breakpoint, but, unfortunately pb() runs but ignores the breakpoint.
someone in another SO thread called it "nested breakpoints".
I am a developer that come from Matlab, in Matlab no matter how a function is called, from console, from debugger. if it has a breakpoint it stops.
I read past threads about this subject and did not find any solution.
I also tried latest pycharm community and latest pydev and no luck.
I also read that visual studio can not make it.
Is this inherent in Python and technically can not be done?
Is there a technique / another IDE that supports it?
I followed this question and saw nobody is answering, I thought about this feature too so I started digging through google and found your old question :)
TL;DR - You can't do that
(I tried with PyCharm, Visual-Studio and Eric Python IDE).
My guess that not working because it adds more complexity to debugging - what happened when you step to the next line? and if you have many threads/processes? what happens to mutable types?
My way to do it
If you have 2 functions and you want to debug both of them:
def parent():
dummy_debuggable_var = 1
print('Running child() function')
out = child(dummy_debuggable_var) #BP1 is here
print(out)
def child(x):
print('Calculating ...')
return x+2
And your goal is to debug the dummy_debuggable_var and also the child function:
Put the break point where is comment is
Run the script
When the break point stops, inspect your dummy_debuggable_var
step-into (on PyCharm - F7)
Inspect your child function
While you're inside the child function, you can look on your PyCharm's frames stack tab on your debugger window, this will let you jump back to the parent's frame and inspect it.
This is more sequential process, and not parallel, but it's simple and it works.
Take a look at Eric Python IDE and VSC(Visual Studio Code)
Related
I know how to use PyCharm's debugger but that has only deepened my curiosity of how it accomplishes the task of being so tightly coupled to the Python interpreter.
Does cPython have some sort of intrepreter hooks buried in itself or does PyCharm somehow copy the source code, instrument the code, and then exec it?
Thanks to #unholySheep I was able to go from the github src on PyDev.Debugger back to sys.settrace which lead to a post on Python Module of the week on settrace.
Once the tracing script has the stack frame, it is likely a non-trivial task of inspecting the frame's stack content and or using code/exec/eval to run "watch" statements in context. As for break points, that would be trivial as it is just a task of matching the frame's line number and filepath.
I am debugging a Python (3.5) program with PyCharm (PyCharm Community Edition 2016.2.2 ; Build #PC-162.1812.1, built on August 16, 2016 ; JRE: 1.8.0_76-release-b216 x86 ; JVM: OpenJDK Server VM by JetBrains s.r.o) on Windows 10.
The problem: when stopped at some breakpoints, the Debugger window is stuck at "Collecting data", which eventually timeout. (with Unable to display frame variables)
The data to be displayed is neither special, nor particularly large. It is somehow available to PyCharm since a conditional break point on some values of the said data works fine (the program breaks) -- it looks like the process to gather it for display only (as opposed to operational purposes) fails.
When I step into a function around the place I have my breakpoint, its data is displayed correctly. When I go up the stack (to the calling function, the one I stepped down from and where I wanted initially to have the breakpoint) - I am stuck with the "Collecting data" timeout again.
There have been numerous issues raised with the same point since at least 2005. Some were fixed, some not. The fixes were usually updates to the latest version (which I have).
Is there a general direction I can go to in order to fix or work around this family of problems?
EDIT: a year later the problem is still there and there is still no reaction from the devs/support after the bug was raised.
EDIT April 2018: It looks like the problem is solved in the 2018.1 version, the following code which was hanging when setting a breakpoint on the print line now works (I can see the variables):
import threading
def worker():
a = 3
print('hello')
threading.Thread(target=worker).start()
I had the same issue with Pycharm 2018.2 when working on a complex Flask project with SocketIO.
When I put a debug breakpoint inside the code and pressed the debug button, it stopped at the breakpoint, but the variables didn't load. It was just infinitely collecting data. I enabled Gevent compatibility and it resolved the issue. Here is where you can find the setting:
In case you landed here because you are using PyTorch (or any other deep learning library) and try to debug in PyCharm (torch 1.31, PyCharm 2019.2 in my case) but it's super slow:
Enable Gevent compatible in the Python Debugger settings as linkliu mayuyu pointed out. The problem might be caused due to debugging large deep learning models (BERT transformer in my case), but I'm not entirely sure about this.
I'm adding this answer as it's end of 2019 and this doesn't seem to be fixed yet. Further I think this is affecting many engineers using deep learning, so I hope my answer-formatting triggers their stackoverflow algorithm :-)
Note (June 2020):
While adding the Gevent compatible allows you to debug PyTorch models, it will prevent you from debug your Flask application in PyCharm! My breakpoints were not working anymore and it took me a while to figure out that this flag is the reason for it. So make sure to enable it only on a per-project base.
I also had this issue when I was working on code using sympy and the Python module 'Lea' aiming to calculate probability distributions.
The action I took that resolved the timeout issue was to change the 'Variables Loading Policy' in the debug setting from the default 'Asynchronously' to 'Synchronously'.
I think that this is caused by some classes having a default method __str__() that is too verbose. Pycharm calls this method to display the local variables when it hits a breakpoint, and it gets stuck while loading the string.
A trick I use to overcome this is manually editing the class that is causing the error and substitute the __str__() method for something less verbose.
As an example, it happens for pytorch _TensorBase class (and all tensor classes extending it), and can be solved by editing the pytorch source torch/tensor.py, changing the __str__() method as:
def __str__(self):
# All strings are unicode in Python 3, while we have to encode unicode
# strings in Python2. If we can't, let python decide the best
# characters to replace unicode characters with.
return str() + ' Use .numpy() to print'
#if sys.version_info > (3,):
# return _tensor_str._str(self)
#else:
# if hasattr(sys.stdout, 'encoding'):
# return _tensor_str._str(self).encode(
# sys.stdout.encoding or 'UTF-8', 'replace')
# else:
# return _tensor_str._str(self).encode('UTF-8', 'replace')
Far from optimum, but comes in hand.
UPDATE: The error seems solved in the last PyCharm version (2018.1), at least for the case that was affecting me.
I met the same problem when I try to run some Deep Learning scripts written by PyTorch (PyCharm 2019.3).
I finally figured out that the problem is I set num_workers in DataLoader to a large value (in my case 20).
So, in the debug mode, I would suggest to set num_workers to 1.
For me, the solution was removing manual watches every-time before starting to debug. If there were any existing manual watches in the "variables" window then it would remain stuck in "Collecting data...".
Using Odoo or Other Large Python Server
None of the above solution worked for me despite I tried all.
It normally works but saldomly gives this annoying Collecting data... or sometimes Timed Out....
The solution is to restart Pycharm and set less breakpoints as possible. after that it starts to work again.
I don't know way is doing that (maybe too many breakpoint) but it worked.
I am new to Python, have some experience in MatLab and r. My question is: Is it possible to run part of the code in .py block by block (line by line)?
In r or Matlab, I can first have some data and variables loaded in the memory first. Then experimentally I can run a line or two to try out the syntax... this is particularly useful for new learners I believe. I know there is something called the iPython which can execute Python code line by line however this is not what I am after. Thanks.
Since ipython has already been discounted, I'm not sure this answer will be better. But I will tell you the two things that I do.
I drop into the debugger at the point where I want to "try out" something, so the code will run up to that point, and then drop me into the debugger. You do this simply by inserting this code at that point:
import pdb; pdb.set_trace()
Once you've done what needs to be done, you can either press q to quit, or c to continue running the process.
I use the -i option to python. This enters interactive mode at the end of your python code. This is useful if you want to set up a bunch of data structures, and try out some code on it, instead of typing all of it into a python shell first. (that might be why you rejected ipython?)
I think what you need is a debugger.
You can use the pydev plugin for Eclipse which has a debugger.
Another option is pdb as already suggested but it's not very easy to use.
Think about this scenario:
I debug my Django project and I step through the code (in and out). The debugger sometimes enters Django libraries or other external libraries.
Does anyone know how to prevent the debugger from entering external code? Or at least a 'big' step out to get the debugger back to the project code?
Does anyone know how to prevent the debugger from entering external code?
Yes, Dmitry Trofimov knows;
(...) add modules you don't want to trace to the dict DONT_TRACE in <pycharm-distr>/helpers/pydev/pydevd.py
That is a hacky solution (...)
If you want this feature to be less hacky you can vote on it by visiting issue
PY-9101 Implement "Do not step into the classes" option for Python debugger
Those using pdb might be interested to know there is such a feature in pdb;
Starting with Python 3.1, Pdb class has a new argument called skip -
class pdb.Pdb(completekey='tab', stdin=None, stdout=None, skip=None, nosigint=False)
The skip argument, if given, must be an iterable of glob-style module
name patterns. The debugger will not step into frames that originate
in a module that matches one of these patterns. 1
1 Whether a frame is considered to originate in a certain module is
determined by the __name__ in the frame globals.
The example given in the docs shows how to skip Django's packages -
import pdb; pdb.Pdb(skip=['django.*']).set_trace()
Everything looks the same to the debugger, it can't distinguish between your code or Django's code – it's all Python. So it will run everything, however if you want to stop it from drilling down so low you'll have to start “stepping over” lines of code instead of “stepping into” them.
According to the PyCharm docs you'll want to use F8 when ever you see a line of code that looks like it could be a gateway into Django's internals. If you accidently find yourself inside Django's source code you can hit Shift+F8 until you're out of it.
I try Eclipse+PyDev pair for some of my work. (Eclipse v3.5.0 + PyDev v1.5.6) I couldn't find a way to expose all of my variables to the PyDev console (Through PyDev console -> Console for current active editor option) I use a simple code to describe the issue. When I step-by-step go through the code I can't access my "x" variable from the console. It is viewed on Variables tab, but that's not really what I want.
Any help is appreciate.
See my screenshot for better description:
EDIT:
Assume adding a simple func like:
def myfunc(x):
return x**x
When I debug with the function added in the code I can access myfunc from the console easily. (Type myfunc and it will be available after this automatic execution:
>>> from part2.test import myfunc
>>> myfunc
Then when I do myfunc(5) it acts just like in the Python interpreter. It would be so useful to access variables in the similar fashion for debugging my code. I have big arrays and I do various tests and operations during debug process. Like:
Get my x and do x.sum(), later do x[::10], or transpose operate with other arrays observe results, experiment etc...
Hope there will be a better solution.
Update:
In the latest PyDev versions, it's possible to right-click a frame in the stack and select PyDev > Debug console to have the interactive console with more functions associated to a context during a debug session.
Unfortunately, the actual interactive console, which would be the preferred way of playing with code (with code-completion, etc -- http://pydev.org/manual_adv_interactive_console.html) has no connection to a debug session right now (this is planned but still not implemented).
Still, with the 'simpler' console available, you are still able to interactively inspect and play with the variables available in a breakpoint scope: http://pydev.org/manual_adv_debug_console.html (which is the same as you'd have with pdb -- it's just a matter of typing code in the available console after a breakpoint is hit).
Cheers,
Fabio
For this sort of exploratory debugging I like to use pdb, the batteries-included debugger. I haven't used it inside PyDev so I don't know how it would all fit together. My guess is it will do what you expect it to. An example of its usage:
import pdb
def myfunc(x):
pdb.set_trace()
return x**x
This will break right before executing the return statement, and it allows you to use full Pythonic statements to figure out what's going on. I use it like an interactive print statement: setting the place where I want to dive in, examining values and figuring results, and stepping through to watch it happen. Perhaps this is a lazy way of debugging, but sometimes you need more information before you can make less-lazy decisions :-)
The page I usually reference is at Python Conquers The Universe which also links a few other sources of information.