I am new to Python programming and having a problem with a multithreaded program (using the "threading" module) that runs fine at the beginning, but after a while starts repeatedly printing "IOStream.flush timed out" errors.
I am not even sure how to debug such an error, because I don't know what line is causing it. I read a bit about this error and saw that it might be related to memory consumption, so I tried profiling my program using a memory profiler on the Spyder IDE. Nothing jumped out at me, however (although I admit that I am not sure what to look for when it comes to Python memory leaks).
A few more observations:
I have an outer loop that runs my function over a large number of files. The files are just number data with the same formatting (they are quite large though and there is download latency, which is why I have made my application multithreaded so that each thread works on different files). If I have a long list of files, the problem occurs. If I shorten the list, the program concludes without problem. I am not sure why that is, although if it is some kind of memory leak then I would assume when I run the program longer, the problem grows until it reaches some kind of memory limit.
Normally, I use 128 threads in the program. If I reduce the number of threads to 48 or less, the program works fine and completes correctly. So clearly the problem is caused by multithreading (I'm using the "threading" module). This makes it a bit trickier to debug and figure out what is causing the problem. It seems something around 64 threads starts causing problems.
The program never explicitly crashes out. Once it gets to the point where it has this error, it just keeps repeatedly printing "IOStream.flush timed out". I have to close the Spyder IDE to stop it (Restart kernel doesn't work).
Right before this error happens, the program appears to stall. At least no more "prints" happen to the console (the various threads are all printing debug information to the screen). The last lines printed are standard debugging/status print statements that usually work when the number of threads is reduced or the number of files to process is decreased.
I have no idea how to debug this and get to the bottom of the problem. Any suggestions on how to get to the bottom of this would be much appreciated. Thanks in advance!
Specs:
Python 3.8.8
Spyder 4.2.5
Windows 10
Related
I have some Python code that's a bit buggy. It runs in an infinite loop and I expect it to print something about once every 50 milliseconds, but sometimes it hangs and stops printing or even outright segfaults. Unfortunately, the problem seems to be fairly rare, so I've had trouble nailing down the exact cause of the issue. I want to have the code up and running while I debug the problem, so while I try to figure it out, I'd like to create a dead man's switch that runs my code, but stops if the code doesn't print anything in a certain time frame (say, 5 seconds) or exits and finally executes a command to notify me that something went wrong (e.g. 'spd-say terminated').
I put this into the terminal for my first attempt at this:
python alert.py; spd-say terminated;
Unfortunately, it didn't seem to work - at this point I realized that the code was not only crashing but also hanging (and I'm also not sure whether this would even work if the code crashes). Unfortunately, I'm not very familiar with bash yet (I assume that's what I'm using when I run stuff in the terminal), and I'm not sure how I could set up something like what I want. I'm also open to using other things besides bash to do this if it would be particularly difficult to do for some reason.
What would be the best way to implement what I want to do?
You could run two python programs with a pipeline between them. On one side you have you buggy script writing something on the pipeline every less than 5 seconds. On the receiving end of the pipeline you have a very simple script that checks how long it has been since it last received anything. If this time is more than 5 seconds.
This way you decouple your watchdog from your buggy script.
Something I've noticed is that when there is an error in our script (regardless of the programming language), it often takes longer to "execute" and then output the error, compared to its execution time when there are no errors in our script.
Why does this happen? Shouldn't outputting an error take less time because the script is not being fully run? Or does the computer still attempt to fully run the script regardless of whether there is an error or not?
For example, I have a Python script that takes approximately 10 seconds to run if there are no errors. When there is an error, however, it takes an average of 15 seconds. I've noticed something similar in NodeJS, so I'm just assuming that this is the case for many programming languages? Apologies if this is a bad question - I'm relatively new to programming and still lack some fundamental understandings.
The program doesn't attempt to run the script fully in case of an error, the execution is interrupted at the point where an error happens. This is by default but you can always set up your own exception handlers in your scripts which will execute some code.
Anyway, raising and handling (logging) exceptions also requires some code execution (internal code of the programming language) thus this also takes some time.
It's hard to tell why your script execution takes longer in case of an error without looking at your script though, I personally never noticed such differences in general but maybe I just didn't pay attention...
I am encountering a strange error that occurs once every few days. I have several virtual machines running on Google Cloud running a Python script. The Python file is very large, but the part that gets stuck is the following:
try:
f = urlopen('https://resources.lendingclub.com/SecondaryMarketAllNotes.csv')
df = pd.read_csv(f)
except:
print('error')
The first line of code always works, but the second line will occasionally stop the program. What I mean by that is that the program will not continue execution, but it does not throw any kind of error. I have logger running in my code in debug mode and it does not record anything.
Again, this happens very rarely, but when it does happen my virtual machines will stop. When I look at the processes in top, I see Python running with 0% CPU, and there is still plenty of system memory available. It will continue to sit there for hours without moving on to the next line of code or returning an error.
My application is very time sensitive and using urlopen is faster than using pd.read_csv to directly open the file.
I notice that when this rare error occurs, it happens at the same time in all of my virtual machines which means that there is likely something about the file being downloaded that is triggering this issue. Why it doesn't cause an error is beyond me.
I would greatly appreciate any ideas on what might be causing this and what workarounds might be available.
I am using Python 3.5.3 and pandas 0.19.2
In PyCharm (JetBrains), I have been having trouble with typing full statements without getting an interuption. I first thought it was due to me not having updated the software, so I updated it, but the problem remains.
So if I type any statement or word, PyCharm seems to delay before I can proceed. An example:
import csv
Even before I finish typing "import" - if I delay a keystroke - PyCharm begins to "think" and the window is not accessible for about one to two seconds (quite literally). I assume it is going to give me suggestions or show a tip/error about the code.
Any thoughts to prevent this from happening?
Edit:
Windows 8.1; PyCharm 2016.2
Code Complete turned off via Settings->Editor->General->Code Completion, but did not solve problem.
Key PC Spec:
Intel Core i5-337U
4GB Ram
64-bit
Edit2:
I receive this error when I run anything now, including simply print("test"):
Process finished with exit code -1073741511 (0xC0000139)
Will separate the question somewhere else, since this may be a separate problem altogether.
Try disabling code completion. I believe that your computer can't search through all of Python's librarys fast enough so it freezes for a bit.
A fairly large Python program I write, runs, but sometimes, after running for minutes or hours, in a non easily reproducible moment, hangs and outputs nothing to the screen.
I have no idea what it is doing at that moment, and in what part of code it is.
How can I run this in a debugger or something to see what lines of code is the program executing in the moment it hangs?
Its too large to put "print" statements all over the place.
I did:
python -m trace --trace /usr/local/bin/my_program.py
but that gives me so much output that I can't really see anything, just millions of lines scrolling on the screen.
Best would be if I could send some signal to the program with "kill -SIGUSR1" or something, and at that moment the program would drop into a debugger and show me the line it stopped at and possibly allow me to step through the program then.
I've tried:
pdb usr/local/bin/my_program.py
and then:
(Pdb) cont
but what do I do to see where I am when it hangs?
It doesn't throw and exception, just seems like it waits for something, possibly in an infinite loop.
One more detail: when the program hangs, and I press ^C and then (not sure if that is necessary) the program continues normally (without throwing any exception and without giving me any hint on the screen why did it stop).
This could be useful to you. I usually do
>>> import pdb
>>> import program2debug
>>> pdb.run('program2debug.test()')
I usually add a -v option to my programs, which enables tons of print statements explaining what I'm doing in detail. When you write a program in the future, consider doing the same before it gets thousands of lines big.
You could try running it in debug mode in an IDE like pydev (eclipse) or pycharm. You can break the program at any moment and get to its current execution point.
No program is ever too big to put print statements all over the place. You need to read up on the logging module and insert lots of logging.debug() statements. This is just a better form of print statement that outputs to a file, and can be turned off easily in production software. But years from now, when you need to modify the code, you can easily turn it all back on and get the benefit of the insight of the original programmer.