We Keep Coding

Debugging fails when reading input from file [duplicate]

I am developing FUSE filesystem with python. The problem is that after mounting a filesystem I have no access to stdin/stdout/stderr from my fuse script. I don't see anything, even tracebacks. I am trying to launch pdb like this:
import pdb
pdb.Pdb(None, open('pdb.in', 'r'), open('pdb.out', 'w')).set_trace()
All works fine but very inconvenient. I want to make pdb.in and pdb.out as fifo files but don't know how to connect it correctly. Ideally I want to type commands and see output in one terminal, but will be happy even with two terminals (in one put commands and see output in another). Questions:
1) Is it better/other way to run pdb without stdin/stdout?
2) How can I redirect stdin to pdb.in fifo (All what I type must go to pdb.in)? How can I redirect pdb.out to stdout (I had strange errors with "cat pdb.out" but maybe I don't understand something)

Ok. Exactly what I want, has been done in http://pypi.python.org/pypi/rpdb/0.1.1 .

Before starting the python app
mkfifo pdb.in
mkfifo pdb.out
Then when pdb is called you can interact with it using these two cat commands, one running in the background
cat pdb.out & cat > pdb.in
Note the readline support does not work (i.e. up arrow)

I just ran into a similar issue in a much simpler use-case:
debug a simple Python program running from the command line that had a file piped into sys.stdin, meaning, no way to use the console for pdb.
I ended up solving it by using wdb.
Quick rundown for my use-case. In the shell, install both the wdb server and the wdb client:
pip install wdb.server wdb
Now launch the wdb server with:
wdb.server.py
Now you can navigate to localhost:1984 with your browser and see an interface listing all Python programs running. The wdb project page above has instructions on what you can do if you want to debug any of these running programs.
As for a program under your control, you can you can debug it from the start with:
wdb myscript.py --script=args < and/stdin/redirection
Or, in your code, you can do:
import wdb; wdb.set_trace()
This will pop up an interface in your browser (if local) showing the traced program.
Or you can navigate to the wdb.server.py port to see all ongoing debugging sessions on top of the list of running Python programs, which you can then use to access the specific debugging session you want.
Notice that the commands for navigating the code during the trace are different from the standard pdb ones, for example, to step into a function you use .s instead of s and to step over use .n instead of n. See the wdb README in the link above for details.

Python script produces zombie processes only in Docker

I have quite complicated setup with
Luigi https://github.com/spotify/luigi
https://github.com/kennethreitz/requests-html
and https://github.com/miyakogi/pyppeteer
But long story short - everything works fine at my local Ubuntu (17.10) desktop, but when run in Docker(18.03.0-ce) via docker-compose(1.18.0) (config version 3.3, some related details here https://github.com/miyakogi/pyppeteer/issues/14) it bloats up with zombie chrome processes spawned from python.
Any ideas why it might happen and what to do with it?

Try installing dumb-init: https://github.com/Yelp/dumb-init (available both in alpine and debian) inside your container and use it as entry point (more reading here: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/)

Seems like it is because python process is not meant to be a root-level process - the topmost one in the processes tree. It just does not handle zombies properly. So after few hours of struggling I end up with the next ugly docker-compose config entry:
entrypoint: /bin/bash -c
command: "(luigid) &"
Where luigid is a python process. This makes bash a root process which handles zombies perfectly.
Would be great to know a more straightforward way of doing this.

Disassociating process pipes from the calling shell

I am trying to use Fabric to send commands which will run many physics simulations (executables) on many different computers which all share the same storage. I would like my script to
ssh into a machine
begin the simulation, for example by running run('nohup nice -n 5 ./interp 1 2 7') (the executable is called interp and run is a function from the Fabric.api library)
detach from the shell and run another simulation on another (or the same) computer.
However I cannot get Fabric to accomplish part 3. It hangs up on the first simulation and doesn't detach until the simulation stops, which defeats the whole point.
My problem, according to the documentation
is that
Because Fabric executes a shell on the remote end for each invocation of run or sudo (see also), backgrounding a process via the shell will not work as expected. Backgrounded processes may still prevent the calling shell from exiting until they stop running, and this in turn prevents Fabric from continuing on with its own execution.
The key to fixing this is to ensure that your process’ standard pipes are all disassociated from the calling shell
The documentation provides 3 suggestions, but it is not possible for me to "use a pre-existing daemonization technique," the computers I have access to do not have screen, tmux, or dtach installed (nor can I install them), and the second proposal of including >& /dev/null < /dev/null in my command has not worked either (as far as I can tell it changed nothing).
Is there another way I can disassociate the process pipes from the calling shell?

The documentation you linked to gives an example of nohup use which you haven't followed all that closely. Merging that example with what you've tried so far gives me something that I, since I don't have Fabric installed, cannot test, but might be interesting to try:
run('nohup nice -n 5 ./interp 1 2 7 < /dev/null &> /tmp/interp127.out &')
Redirect output to /dev/null rather than my contrived output file (/tmp/interp127.out) if you don't care what the interp command emits to its stdout/stderr.
Assuming the above works, I'm unsure how you would detect that a simulation has completed, but your question doesn't seem to concern itself with that detail.

Python spawn detatched non-python process on linux?

I was to make a launcher application, I haven't found a way to detach a sub-process entirely from the spawning python process.
When I launch a program with my desktop's (cinnamon's) launcher the process tree goes:
/sbin/init -> mdm -> mdm -> cinnamon-session -> cinnamon -> the-app-i-launched
Of the threads I read, this one was the most insightful/helpful: Launch a completely independent process. But gets muddied answers as the OP is looking to run python code, which can often be achieved in many usually-preferred ways than by spawning an independent process.
From other posts from stack overflow that do not answer how to launch a detatched python process:
Running daemonalized python code: Applicable to running python code/module as a daemon, (not another process/application) detached from the python instance.
subprocess.call: Process spawns as a child of the python process.
os.system: Process spawns as a child of the python process.
close_fds: (Apparently) Windows(R)-only solution, need portable solution (primary target is Debian linux). Attempting to use close_fds=True on linux, process spawns as a child of the python process.
creationflags: Windows(R)-only solution. On linux raises: ValueError: creationflags is only supported on Windows platforms.
prefix launched process with nohup: Process spawns as a child of the python process. As far as I know, nohup or equivalent is not available on all platforms, making it a linux-only solution.
os.fork: Same as "Running daemonalized python code".
multiprocessing: Same problem as "Running daemonalized python code": useful only for running python code/module.
os.spawnl* + os.P_NOWAIT: Deprecated functions are undesirable to use for new code. In my testing I was not able to see my process actually spawned at all.
os.spawnl* + os.P_DETACH: Windows(R)-only, seemingly removed in current python 2.X versions: AttributeError: 'module' object has no attribute 'P_DETACH'.
os.system + shell fork: I was able to actually see my process run detatched from the python process with this, however I am woried that it has the faults:
Relies on running commands in a shell, which is more vulnerable to maliciousness, intentional or otherwise?.
Relies on non-portable? POSIX/shell? syntaxies that may not be interpenetrate on non-Linux platforms. Which I haven't dug up any good reference for portability on Partial Ref.
subprocess.Popen Alt: I still only observed the sub-process running as a child of the python process.

A working solution can be found in JonMc's answer here. I use it to open documents using 'xdg-open'.
You can change the stderr argument to stderr=open('/dev/null', 'w'), if you do not want a logfile.

The only workable solution I have, and that may be non-portable Linux-only is to use shell evaluation with shell-detach-ampersand syntax.
#!/usr/bin/env python2
import os
os.system('{cmd} &'.format('firefox'))
This might go too-far up the process tree though, outside of the window manager session, potentially not-exiting with your desktop session.
/sbin/init -> firefox

Pydev Django project compilation

I installed pydev and eclipse to compile a django project. Everything looks good but one thing makes me crazy. When i change something in the code, i click to save it and hope to see the effect of changes. However, I cannot see the effect of what i change unless I terminate the program and re-run as as shown below. It is such a pain...
I used to use Pycharm, but it expired. In Pycharm, when the program runs once, i do not need to run it again and again. I can easily see the effect of my changes on the code by clicking save button. Is it possible to see the same thing in pydev and eclipse? Do you guys also see this issue?

To debug Django with the autoreload feature, the Remote Debugger must be used and a patch must be applied to your code (just before the if _name_ == "_main_": in your manage.py module):
import pydevd
pydevd.patch_django_autoreload(
patch_remote_debugger=True, #Connect to the remote debugger.
patch_show_console=True
)
So, doing that, starting the remote debugger and making a regular run should enable all the regular breakpoints that are put inside Eclipse to work in the Django process with the Remote Debugger (and the --noreload that PyDev added automatically to the launch configuration must be removed).
I have plans on improving that so that the debugger will automatically add tracing to spawned processes (probably for the next release), but on PyDev 3.3.3 this still requires doing this manual patch and using the remote debugger.
The above is related to a debug run. Now, on to the regular run...
When you do a run as > pydev: django, it should create a run configuration (which you can access at run > run configuration). Then, open that run configuration > arguments and remove the '--noreload' (leaving only the 'runserver' argument).
Then, you can simply rerun it with Ctrl+F11 (if you've set it to launch the previously launched application as indicated in http://pydev.org/manual_101_run.html)
-- (Or alternatively you can run it with: Alt + R, T, 1).
The only problem is that if you kill the process inside Eclipse it's possible that it leaves zombie processes around, so, you can use the pydevd.patch_django_autoreload(patch_show_console=True) as indicated above to open a console each time it spawns a new process (you can do a pydevd|<- ctrl space here to add an import to pydevd).
Note that this should work. If it's not working, make sure you don't have zombie processes around from a previous session (in windows you can do: taskkill /F /IM python.exe to make sure all Python processes are killed).
Another note (for when you don't actually have automatic reload): Ctrl+Shift+F9 will kill the currently running process and re-run it.