We Keep Coding

app package built by pyinstaller fails after it uses tqdm

I have a program which can optionally use tqdm to display a progress bar for long calculations.
I've packaged the program using pyinstaller.
The packaged program runs to completion just fine if I don't enable tqdm.
If I enable tqdm, the package works fine and displays progress bars until it gets to the very end, then the message below is displayed and the packaged program is relaunched (with the wrong args).
multiprocessing/resource_tracker.py:104: UserWarning: resource_tracker:
process died unexpectedly, relaunching. Some resources might leak.
There is nothing fancy about the way the program invokes tqdm.
There is a loop that iterates over the members of a list.
The program does not import the multiprocessing module.
def best_cost(self, guesses, steps=1, verbose=0) -> None:
""" find the guess with the minimum cost """
self.cost = float("inf")
for guess in guesses if verbose < 3 else tqdm(guesses, mininterval=1.0):
groups = dict()
for (word, frequency) in self.words.items():
result = calc_result(guess, word)
groups.setdefault(result, {})[word] = frequency
self.check_cost(guess, groups, steps=steps)
return
The problem occurs whether pyinstaller creates the package with --onedir or --onefile.
I'm running on MACOS Monterey V12.4 on an Intel processor.
I'm using python 3.10.4, but the problem also occurs with 3.9. Pip installed packages include: pyinstaller 5.1, pyinstaller-hooks-contrib 2022.7, and tqdm 4.64.0.
I package the program in a virtualenv.
I found this in the documentation for the multiprocessing module:
On Unix using the spawn or forkserver start methods will also start a resource tracker process which tracks the unlinked named system resources (such as named semaphores or SharedMemory objects) created by processes of the program. When all processes have exited the resource tracker unlinks any remaining tracked object. Usually there should be none, but if a process was killed by a signal there may be some “leaked” resources. (Neither leaked semaphores nor shared memory segments will be automatically unlinked until the next reboot. This is problematic for both objects because the system allows only a limited number of named semaphores, and shared memory segments occupy some space in the main memory.)
I can't figure out how to debug the the problem since it only occurs with the pyinstaller packaged program and my IDE (vscode) doesn't help.
My program doesn't use the multiprocessing module.
The tqdm documentation doesn't mention issues like this, but it does include the multiprocessing module to create locks (semaphores) in the tqdm class which
get used by the instances.
There aren't any tqdm methods I can find to delete the semaphores resources;
I guess that is just supposed to happen automatically when the program ends.
I think the pyinstaller packaged program may also use the multiprocessing
module.
Perhaps pyinstaller is interfering with deletion of the tqdm semaphores somehow?
The packaged program works fine till it starts to exit and then gets restarted.
Does anyone know how I can get the packaged program to exit cleanly?

Workaround to successfully profile python script using scalene profiler on macOS? Just forget it and use machine with Windows or Linux?

Computational Science SE question How amenable is this 2D Frenkel–Kontorova-like energy minimization problem in Python to the use of a modest PC + GPU? (Heavy reliance on indexing) contains a short example script and note 3 links to my first attempt at profiling using scalene
The results were uninformative, so I followed a recommendation that I try the --profile-all option. After running for 30 minutes without finishing on a script that took seconds to run I added a CPU percent limit; what I assume means only things that used at least 2% of the CPU time would be profiled in depth.
scalene --html --outfile prof.html --profile-all --cpu-percent-threshold 2 myscript.py
I received a two line error and was exited from python in a normal way.
Error getting real path: 2
Scalene error: received signal SIGABRT
This issue was closed in scalene issue #110 and there are links there to
https://github.com/plasma-umass/scalene/commit/6636d95d7ea9a16adedaedbd4ff0926374798f2b
Q: How do I use Scalene with PyTorch?
A: Scalene works with PyTorch version 1.5.1. There's a bug in newer versions of PyTorch (https://github.com/pytorch/pytorch/issues/57185) that interferes with Scalene (discussion here: https://github.com/plasma-umass/scalene/issues/110).
there's more information there as well
and
Pytorch open issue #57185 Segmentation fault with ITIMER_REAL PyTorch throws SIGSEGV when running alongside timer on MacOS x86
Question: Is there a workaround to successfully profile python script using scalene profiler on macOS? Or should I just move to a machine with Windows or Linux and forget about trying for now?

The message, Error getting real path: 2, seems to have to do with scalene finding your script or a path internal to your script possibly.
Ensure you're referencing a full, valid path for myscript.py, taking into account which directory you're at in the terminal. You may need to change directory.
The SIGABRT message is different from the SIGSEGV issue listed, though, if you wanted to test it, you could uninstall PyTorch (pip uninstall torch) and reinstall a specific version (pip install 'torch==1.5.1').

import readline segfaults after import TVM on Ubuntu

I'm working with TVM, a deep learning compiler. In my workflow, it is useful to be able to use import pdb; pdb.set_trace(), and drop into the debugger. However, pdb.set_trace() causes a segfault anytime after TVM has been imported.
My current setup is
- Ubuntu 16.04 (running on Windows Subsystem for Linux, but this happens on my native 16.04 machine too!)
- Python 3.6
This problem does not occur on Windows or Mac.
This problem only occurs when running a script from the commandline (i.e. python3 minimum-reproducible-example.py), not when running from the python3 repl.
I've done some debugging with gdb, and narrowed it down: the error occurs when the readline package is imported.
Minimum reproducible example:
import tvm
import readline
After debugging with gdb, I traced it to a specific line in cpython:
Program received signal SIGSEGV, Segmentation fault.
PyModule_GetState (m=0x0) at Objects/moduleobject.c:558
558 if (!PyModule_Check(m)) {
In this case, m is 0x0, which the function doesn't seem to expect.
If anyone could even just point me towards more useful ways to debug this, that would be helpful!

Slow down when subprocesses launched in parallel when using mpi4py

Using mpi4py, I'm running a python program which launches multiple fortran processes in parallel, starting from a SLURM script using (for example):
mpirun -n 4 python myprog.py
but have noticed that myprog.py takes longer to run the higher the number of tasks requested eg. running myprog.py (following code shows only the mpi part of program):
comm = MPI.COMM_WORLD
size = comm.Get_size()
rank = comm.Get_rank()
data = None
if rank == 0:
data = params
recvbuf = np.empty(4, dtype=np.float64)
comm.Scatter(data, recvbuf, root=0)
py_task(int(recvbuf[0]), recvbuf[1], recvbuf[2], int(recvbuf[3]))
with mpirun -n 1 ... on a single recvbuf array takes 3min- whilst running on four recvbuf arrays (expectedly in parallel), on four processors using mpirun -n 4 ... takes about 5 min. However, I would expect run times to be approximately equal for both the single and four processor cases.
py_task is effectively a python wrapper to launch a fortran program using:
subprocess.check_call(cmd)
There seems to be some interaction between subprocess.check_call(cmd) and the mpi4py package that is stopping the code from properly operating in parallel.
I've looked up this issue but can't seem to find anything that's helped it. Are there any fixes to this issue/ detailed descriptions explaining what's going on here/ recommendations on how to isolate the cause of the bottleneck in this code?
Additional note:
This pipeline has been adapted to mpi4py from "joblib import Parallel", where there was no previous issues with the subprocess.check_call() running in parallel, and is why I suspect this issue is linked with the interaction between subprocess and mpi4py.

The slowdown was initially fixed by adding in:
export SLURM_CPU_BIND=none
to the slurm script that was launching the jobs.
Whilst the above did provide a temporary fix, the issue was actually much deeper and I will provide a very basic description of it here.
1) I uninstalled the mpi4py I had installed with conda, then reinstalled it with Intel MPI loaded (the recommended MPI version for our computing cluster). In the SLURM script, I then changed the launching of the python program to:
srun python my_prog.py .
and removed the export... line above, and the slowdown was removed.
2) Another slowdown was found for launching > 40 tasks at once. This was due to:
Each time the fortran-based subprocess gets launched, there's a cost to the filesystem requesting the initial resources (eg. supplying file as an argument to the program). In my case there were a large number of tasks being launched simultaneously and each file could be ~500mb, which probably exceeded the IO capabilities of the cluster filesystem. This led to a large overhead introduced to the program from the slowdown of launching each subprocess.
The previous joblib implementation of the parallelisation was only using a maximum of 24 cores at a time, and then there was no significant bottleneck in the requests to the filesystem- hence why no performance issue was previously found.
For 2), I found the best solution was to significantly refactor my code to minimise the number of subprocesses launched. A very simple fix, but one I hadn't been aware of before finding out about the bottlenecks in resource requests on filesystems.
(Finally, I'll also add that using the subprocess module within mpi4py is generally not recommended online, with the multiprocessing module preferred for single node usage. )

How to limit memory usage within a python process

I run Python 2.7 on a Linux machine with 16GB Ram and 64 bit OS. A python script I wrote can load too much data into memory, which slows the machine down to the point where I cannot even kill the process any more.
While I can limit memory by calling:
ulimit -v 12000000
in my shell before running the script, I'd like to include a limiting option in the script itself. Everywhere I looked, the resource module is cited as having the same power as ulimit. But calling:
import resource
_, hard = resource.getrlimit(resource.RLIMIT_DATA)
resource.setrlimit(resource.RLIMIT_DATA, (12000, hard))
at the beginning of my script does absolutely nothing. Even setting the value as low as 12000 never crashed the process. I tried the same with RLIMIT_STACK, as well with the same result. Curiously, calling:
import subprocess
subprocess.call('ulimit -v 12000', shell=True)
does nothing as well.
What am I doing wrong? I couldn't find any actual usage examples online.
edit: For anyone who is curious, using subprocess.call doesn't work because it creates a (surprise, surprise!) new process, which is independent of the one the current python program runs in.

resource.RLIMIT_VMEM is the resource corresponding to ulimit -v.
RLIMIT_DATA only affects brk/sbrk system calls while newer memory managers tend to use mmap instead.
The second thing to note is that ulimit/setrlimit only affects the current process and its future children.
Regarding the AttributeError: 'module' object has no attribute 'RLIMIT_VMEM' message: the resource module docs mention this possibility:
This module does not attempt to mask platform differences — symbols
not defined for a platform will not be available from this module on
that platform.
According to the bash ulimit source linked to above, it uses RLIMIT_AS if RLIMIT_VMEM is not defined.