I have a Python process that uses os.popen to run tcpdump in the background. It then reads and processes the output from tcpdump. The process runs in the background as a daemon. When I execute this process from the command line, it runs just fine--it fires up tcpdump and reads the output properly. However, I want this process to run automatically at boot and I've directed it to do so in cron. When I do this, my process is running (per the ps command) but tcpdump is not.
Is there some reason the behavior is different starting a process in cron vs starting it from the command line? My code looks something like this:
p = os.popen('/usr/sbin/tcpdump -l -i eth0')
while True:
data = p.readline()
# do something with data
cron will send you an email when there is a problem. So the first thing is to look into your mailbox (run mailx to access it).
If there is no mail, make sure the processes write messages to stdout/stderr when there is a problem.
Also: Check that you're using the correct user. On some systems, tcpdump needs to be run as root, so you need to install the job into root's crontab (instead of the one of your normal user).
Related
Not able to send commands to shell I logged into
Originally, I wrote a Python script. It was able to send commands like
subprocess.run(['kubectl', 'config', 'get-context'], shell=True)
but when it came time to get to the child shell, in this case bash, the command wouldn't run until I exited that shell and it would say things like it couldn't find the command.
I then tried to do it with the module "sh," but was also unsuccessful
I thought maybe using Python was problem and also realized my ultimate goal was to use a different shell (cypher-shell) and so skipped immediately to that with bash as the parent shell. In there I have a line that is sometimes successful, sometimes not
kubectl run -it --rm cypher-shell --image=gcr.io/cloud-marketplace/neo4j-public/causal-cluster-k8s:3.4 --restart=Never --namespace=default --command -- ./bin/cypher-shell -u neo4j -p "password" -a "domain.name"
But even when it successfully logs in it, it just hangs until I manually exit and then it runs the next commands
Note: I saw this and so, perhaps, it's not a child shell? Run shell command from child shell
I can't say I know exactly what you are doing, but if I understand your objective correctly you want the Python program to continue to log while the script continues to run? The problem is that the logger continues to run and holds up your program. The way I would deal with that would be to run the logger as a background process.
With bash, that would be ./script.sh & which would make it run without holding the rest of the program back from running.
Hopefully that may give you an idea! Good luck.
I was trying to run slurm jobs with srun on the background. Unfortunately, right now due to the fact I have to run things through docker its a bit annoying to use sbatch so I am trying to find out if I can avoid it all together.
From my observations, whenever I run srun, say:
srun docker image my_job_script.py
and close the window where I was running the command (to avoid receiving all the print statements) and open another terminal window to see if the command is still running, it seems that my running script is for some reason cancelled or something. Since it isn't through sbatch it doesn't send me a file with the error log (as far as I know) so I have no idea why it closed.
I also tried:
srun docker image my_job_script.py &
to give control back to me in the terminal. Unfortunately, if I do that it still keeps printing things to my terminal screen, which I am trying to avoid.
Essentially, I log into a remote computer through ssh and then do a srun command, but it seems that if I terminate the communication of my ssh connection, the srun command is automatically killed. Is there a way to stop this?
Ideally I would like to essentially send the script to run and not have it be cancelled for any reason unless I cancel it through scancel and it should not print to my screen. So my ideal solution is:
keep running srun script even if I log out of the ssh session
keep running my srun script even if close the window from where I sent the command
keep running my srun script and let me leave the srun session and not print to my scree (i.e. essentially run to the background)
this would be my idea solution.
For the curious crowd that want to know the issue with sbatch, I want to be able to do (which is the ideal solution):
sbatch docker image my_job_script.py
however, as people will know it does not work because sbatch receives the command docker which isn't a "batch" script. Essentially a simple solution (that doesn't really work for my case) would be to wrap the docker command in a batch script:
#!/usr/bin/sh
docker image my_job_script.py
unfortunately I am actually using my batch script to encode a lot of information (sort of like a config file) of the task I am running. So doing that might affect jobs I do because their underlying file is changing. That is avoided by sending the job directly to sbatch since it essentially creates a copy of the batch script (as noted in this question: Changing the bash script sent to sbatch in slurm during run a bad idea?). So the real solution to my problem would be to actually have my batch script contain all the information that my script requires and then somehow in python call docker and at the same time pass it all the information. Unfortunately, some of the information are function pointers and objects, so its not even clear to me how I would pass such a thing to a docker command ran in python.
or maybe being able to run docker directly to sbatch instead of using a batch script with also solve the problem.
The outputs can be redirected with the options -o stdout and -e for stderr.
So, the job can be launched in background and with the outputs redirected:
$ srun -o file.out -e file.errr docker image my_job_script.py &
Another approach is to use a terminal multiplexer like tmux or screen.
For example, create a new tmux window type tmux. In that window, use srun with your script. From there, you can then detach the tmux window, which returns you to your main shell so you can go about your other business, or you can logoff entirely. When you want to check in on your script, just reattach to the tmux window. See the documentation tmux -h for how to detach and reattach on your OS.
Any output redirects using the -o or -e will still work with this technique and you can run multiple srun commands concurrently in different tmux windows. I’ve found this approach useful to run concurrent pipelines (genomics in this case).
I was wondering this too because the differences between sbatch and srun are not very clearly explainer or motivated. I looked at the code and found:
sbatch
sbatch pretty much just sends a shell script to the controller, tells it to run it and then exits. It does not need to keep running while the job is happening. It does have a --wait option to stay running until the job is finished but all it does is poll the controller every 2 seconds to ask it.
sbatch can't run a job across multiple nodes - the code simply isn't in sbatch.c. sbatch is not implemented in terms of srun, it's a totally different thing.
Also its argument must be a shell script. Bit of a weird limitation but it does have a --wrap option so that it can automatically wrap a real program in a shell script for you. Good luck getting all the escaping right with that!
srun
srun is more like an MPI runner. It directly starts tasks on lots of nodes (one task per node by default though you can override that with --ntasks). It's intended for MPI so all of the jobs will run simultaneously. It won't start any until all the nodes have a slot free.
It must keep running while the job is in progress. You can send it to the background with & but this is still different to sbatch. If you need to start a million sruns you're going to have a problem. A million sbatchs should (in theory) work fine.
There is no way to have srun exit and leave the job still running like there is with sbatch. srun itself acts as a coordinator for all of the nodes in the job, and it updates the job status etc. so it needs to be running for the whole thing.
I am running a script that launches
run_app.py >& log.out
In run_app.py, it will start a few subprocesses and will read stdout/stderr of the subprocesses through pipe. I can run the script fine but if I try to put it into background by:
run_app.py >& log.out &
The run_app.py will hang on reading data from subprocess. It seems that it is similar to this thread:
ffmpeg hangs when run in background
My subprocess also write a lot which might overflow the PIPE_BUF.
However, I am redirecting&writing my stdout/stderr to a file. Are there any suggestions might prevent hanging when I put the script to background while able to save output in a file instead of redirecting them to /dev/null?
When a background process is running, its standard I/O streams are still connected to the screen and keyboard. Processes will be suspended (stopped) if they try to read from the keyboard.
You should have to a message saying something like: Stopped (tty input). That would have been sent to the shell's stderr.
Normally redirecting stdin covers that problem, but some programs access the keyboard directly rather than using stdin, typically those prompting for a password.
Assume there exists a python script resolve_ip.py which magically returns the string IP of a machine we care about.
I'm interested in learning how to achieve the python equivalent of the following bash command:
user#host:~$ ssh $(./resolve_ip.py)
In this bash example, after the python script runs, it is replaced or rather substituted with its return value which is, in turn, provided as a input to the program ssh. The result is, the python program terminates and the user interacts with ssh initialization.
The problem is, this solution requires the use of either 2 scripts (a bash script to run ssh, combined with the python script to return the arguments) or alternatively human intervention to type the bash command directly as formatted above.
Question:
Is there a way, only using python, to start/fork an interactive service (like ssh), by using subprocess or some comparable module, and have this forked child process remain alive in the foreground, even after its parent (python) has sys.exit()'d?
Considerations:
In this case, the point of this question is not to script the submission of ssh credentials to ssh from within Python. It is how to spawn a subprocess which continues to run foregrounded after its parent/spawner terminates.
EDIT:
The following resources are related:
Run a program from python, and have it continue to run after the script is killed
how to spawn new independent process in python
Indefinite daemonized process spawning in Python
Python spawn off a child subprocess, detach, and exit
I think you want to "exec". Like this:
import resolve_ip
import os
host = resolve_ip.get_host_to_use() # you figure this part out
os.execlp('ssh', 'ssh', host)
That will replace the Python interpreter process image with the ssh process image, and keep running. So you will end up as if you had run ssh instead of Python, and the interpreter will no longer execute at all.
I am running a python process (through Django's manage.py) that will take about 6 days to complete. I would start it through an SSH tunnel, but that would require me to keep my Terminal window running and not let my computer fall asleep so the connection stays intact.
While the process is running, it reports the percentage on a single line (by replacing the line over and over again).
How can I start the process and be able to disconnect from SSH, but still be able to check on its progress when I reconnect?
use GNU Screen, your life will be so much more beautiful when you could load/reload your working session of terminals anytime you want --- they never die, and yet they ain't no zombies.
This is not really a Python question, rather an OS question.
So what you can do in linux is run your process with no hang up.
tmp.py
import time
for i in range(10):
print i
time.sleep(1)
No you can run it like so:
sudo nohup python -u tmp.py >> output &
The output will be placed in this file.
Then you can watch the output comming to the file by doing
tail -f output