I'm loading a large csv file into pandas and when I load too many lines at once I get a dialog box telling me "Python has stopped working" without any error messages in the terminal (screenshot). I suspect it's a memory limitation but it'd be nice to confirm with the python stacktrace directly. Anyone have a similar experience and know how to get at what's happening?
update: Turns out not to have been a memory limitation after all. I think the root cause was a pandas issue, upgrading from 20.3 -> 22.0 seems to have fixed it. I suspect it was related to this: https://github.com/pandas-dev/pandas/issues/16798
Use the trace module to try and force it.
python -u -m trace -t program.py
or
python -m pdb program.py
Python debugger might also provide insight. If neither of these work, it's most likely a memory issue based on the context you provided.
Hope this helps.
Related
So I am trying to write some python code that will do two things, that seem to be mutually exclusive on my machine. My PC's host operating system is windows and I run Kali-Linux in WSL2 when I need to test my code on Linux. My code's main function creates two separate multiprocessing.Process objects, assigning a different thread, starting them both one after the other and then calling for them both to be joined. The plan is to allow each to run a simple server application simultaneously on different ports. This does not work when running python3 in PowerShell, as it seems to require access to os.fork() which doesn't work in said environment. When I found this out I pivoted to running in WSL2 which worked fantastically, for a time. After a while of experimenting with some ideas I decided to take some of my code and spin it off into its own file, which I placed in its own 'Libs' folder. WSL2 however, was unable to import this new file, instead giving me the exception ModuleNotFoundError: No module named 'NetStuff'. I originally had added:
sys.path.append('./Libs')
as has worked for me in the past, however when I found that WSL2 was unable to find my module, I printed out sys.path and it revealed that rather than appending my $current_working_directory/Libs like I intended, I was just appending the literal string, which wasn't useful. I then decided to try:
sys.path.append(str(pathlib.Path().resolve()) + '/Libs')
which at the bare minimum shows up as I would expect in sys.path. This, however still didn't work, python was unable to find my module and would unceremoniously crash every time. This led me to try something else, I ran my code in python3 under PowerShell again, which had no issue importing my module, it did still crash due to lacking os.fork() but the import gave no issues. Confused and annoyed I opened my code in IDLE 3.9 which, for some inexplicable reason, was able to import the file, and seemingly use os.fork(). The only major issue with running in IDLE is that it is seemingly incapable of understanding ascii colour escape characters. Given that the goal is to run my code in bash, and ideally also PowerShell, I am not satisfied with this as a solution. I returned to trying to fix the issue in WSL2 by adding my module to /home/Noah/bin, and appending this directory to sys.path, but this has still not so much as given me a new symptom.
I am utterly at a loss at this point. none of the fixes I know off hand are working, and neither are the new ones I've found online. I can't tell if I'm just missing something fundamental about python or if I'm running into a bug, if it's the latter, i can't seem to find other people with the same issue. As a result of my confusion and frustration I am appealing to you, kind users of stackoverflow.
The following is the snippet that is causing me problems in WSL2:
path0 = ('/home/Noah/bin')
path1 = (str(pathlib.Path().resolve()) + '/Libs')
sys.path.append(path0)
sys.path.append(path1)
print(sys.path)
import NetStuff
The following is output of print(sys.path) in WSL2:
['/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack', '/usr/lib/python39.zip', '/usr/lib/python3.9', '/usr/lib/python3.9/lib-dynload', '/home/noah/.local/lib/python3.9/site-packages', '/usr/local/lib/python3.9/dist-packages', '/usr/lib/python3/dist-packages', '/home/Noah/bin', '/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack/Libs']
The following is the error being thrown by WSL2:
Traceback (most recent call last):
File "/mnt/c/Users/Noah/VSCodeRepos/Python/BlackPack/BlackPackServer.py", line 21, in <module>
import NetStuff
ModuleNotFoundError: No module named 'NetStuff'
I am specifically hoping to fix the issue with WSL2 at the moment as I am fairly certain that getting the code to run on PowerShell is merely going to require rewriting my code so that it doesn't rely on os.fork(). Thank you for reading my problem, and if I left out any information that you would like to see just tell me and I'll add it in an edit!
Edit: I instantly realized that I should specify that my host machine is running windows 10.
i have no idea what i have done to cause this but when running C:>python .... (image 1) i get this error (image two)
i've tried repairing the install and running a full scan for any viruses and it's all going normal but then i still get this error
image 1:
image 2:
i'd just like to thank anyone who may be able to help in advance too!
When you type py command Python 3.9.2 is launched. However, when you type python the error message shows that python38.dll wasn't found which is a different version. It seems like you had different versions of Python and maybe some uninstallation process went wrong.
Are you able to check what do you have on PATH? The issue might be there. You can also try to uninstall all Python-related apps and install a fresh Python 3.9
Attempting a setup of the Honeypot Snare here and getting an error of sorts.
Installed onto Ubuntu.14 variant, following setups from the github: https://github.com/mushorg/snare
Onto step 3 as it asks: Clone a page: sudo python3 clone.py --target http://example.com
When I try this command on any domain i get the same error. Even with no domain. Seems to be an error with the python code, had a look and can see the def line it points too but that kind of code is through the whole clone.py file.
error
Not sure whats causing it or what ive done wrong if anything. Starting to get into the use of Linux but would still consider myself a beginner.
Any help would be awesome! Thanks!
Updating to a newer OS version .16 worked with all the same commands previously used. Posting to let others know if they experience this! :D
Glances v2.11.1 with psutil v5.4.3
/usr/lib/python3.6/site-packages/psutil/_pslinux.py:1152: RuntimeWarning: ignoring OSError(6, 'No such device or address')
warnings.warn("ignoring %r" % err, RuntimeWarning)
OS is antegos running 4.15.3-2ARCH Kernel.
Why is it a problem? It is scrolling the console.
Temporary fix: I commented the message of the exception.
The issue seems to be related to the fan speed. I am running thinkfan and tlp.
Is there any real fix? I did not notice any problems using glances.
I have the same issue with glances.
I got ride of the scrolling issue by starting glances like this:
glances 2>/dev/null
Hope that helps
The David Tourrel is a nice workaround, and it's working, but the same issue is something that happened to me since yesterday, when I had to uninstall/reinstall glances.
I still don't know why it's happening. it seems related to the sensors (heat? fan) idk). When looking for in the internet I found the
glances --disable-sensors
makes appear the warning message just once during its startup, and not during the running.
The best way is to try run it this way by typing in terminal:
sudo glances --enable-plugin sensors
or just
sudo glances
Running it as user, without sudo/root privileges, automatically shows the error mentioned above. It looks like gathering data from sensors via psutil can be done only as an elevated user.
I ran cProfile on a python 3 script, worked nicely, then tried to visualize it using runsnake. Howvever, I got an empty screen and the error 'bad marshal data'.
I removed .pyc file but that did not work either.
The code I used to install runsnake was:
sudo apt-get install python-profiler python-wxgtk2.8 python-setuptoolD
sudo easy-install installSquareMap RunSnakeRun
I am using UBUNTU.
Many thanks.
note: I should add I installed everything while py3k was activated
TL;DR: This error occurs when profiling in Python 2.x and viewing the profile in Python 3.x or vice versa.
I had the same problem. As far as I can tell, the RunSnakeRun package has not been ported to Python3. At least, I could pip it to python2 but not to python3 (SyntaxError). Further, I think the output format of cProfile is not compatible between python 2/3. I did not take the time to find an definitive confirmation of this, but in the doc of cProfile class pstats.Stats(*filenames, stream=sys.stdout), they do say "The file selected by the above constructor must have been created by the corresponding version of profile or cProfile. To be specific, there is no file compatibility guaranteed with future versions of this profiler, and there is no compatibility with files produced by other profilers.". This seems to be the origin of your problem. For e.g., I made a profile output from python3
import cProfile
cProfile.run('some code to profile', 'restats')
and tried to open it in RunSnakeRun and got the same marhsal error you got. Further, if I do
import pstats
p = pstats.Stats('restats')
p.strip_dirs().sort_stats(-1).print_stats()
in python3, it works like a charm. If I do it in python2, it gives the marshal error. Now, RunSnakeRun is executed in python2 (unless you found some way to make it run in python3). So, my guess is that you have performed your profiling in python3 and are using tools relying on python2 to analyze them, which tools are expecting the output to be compatible with python2.
The RunSnakeRun project seems to be inactive for a while now (copyright on the home page is 2005-2011) and there is no indication that it will be ported to python3.... Maybe considering alternative visualization tool might be the best way to go for you if you want to develop in Python3. pyprof2calltree in combination with KCachegrind worked fine for me in Linux. It can provide a similar visual view of the profiling output as you would get from RunSnakeRun.
Also ran into the same problem and I think there's no (good) way to use runsnake for Python3 (as it already was mentioned in the previous answer). However, SnakeViz might help. It's a relatively intuitive graphical overview of profiling data that, like runsnake, builds on top of profile outputs. Nice bonus: works also for Jupyter notebooks!