I would like to control adb (Android Debug Bridge) from a python script.
In order to do this I want to use the shell command from adb:
import subprocess as sp
adb = 'path-to-adb.exe'
print("Running shell command!")
p = sp.Popen([adb, 'shell'], stdout=sp.PIPE, stderr=sp.PIPE, stdin=sp.PIPE)
p.stdin.write('ls\r\n'.encode('utf-8'))
print(p.stdout.readlines().decode('utf-8'))
print('Do more stuff and eventually shut down.')
The idea is that I would write a command to the android shell, wait for the response then write another and so on... But whenever I call read() or readlines() on the running process it just does not return.
If however I call communicate() it works fine and returns the expected result. The problem with communicate() is that it ends the process.
I have looked at several questions here on Stackoverflow but here the answere always seems to be to wait the process to terminate (by either using communicate() or subprocess.run()). Am I missing something here? Am I just not supposed to interact with a running process?
Related
I wrote a simple python script ./vader-shell which uses subprocess.Popen to launch a spark-shell and I have to deal with KeyboardInterrupt, since otherwise the child process would not die
command = ['/opt/spark/current23/bin/spark-shell']
command.extend(params)
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
This is what I see with ps f
When I actually interrupt with ctrl-C, I see the processes dying (most of the time). However the terminal starts acting weird: I don't see any cursor, and all the lines starts to appear randomly
I am really lost in what is the best way to run a subprocess with this library and how to handle killing of the child processes. What I want to achieve is basic: whenever my python process is killed with a ctrl-C, I want all the family of process being killed. I googled several solutions os.kill, p.wait() after termination, calling subprocess.Popen(['reset']) after termination but none of them worked.
Do you know what is the best way to kill when KeyboardInterrupt happens? Or do you know any other more reliable library to use to spin-up processes?
There is nothing blatantly wrong with your code, the problem is that the command you are launching tries to do stuff with the current terminal, and does not correctly restore the settings where shutting down. Replacing your command with a "sleep" like below will run just fine and stop on Ctrl+C without problems:
import subprocess
command = ['/bin/bash']
command.extend(['-c', 'sleep 600'])
p = subprocess.Popen(command)
try:
p.communicate()
except KeyboardInterrupt:
p.terminate()
I don't know what you're trying to do with spark-shell, but if you don't need it's output you could try to redirect it to /dev/null so that it's doesn't mess up the terminal display:
p = subprocess.Popen(command, stdout=subprocess.DEVNULL)
I have a python script in blender where it has
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
followed by many other code which depends on this shell script to finish. What happens is that it doesn't wait for it to finish, I don't know why? I even tried using Popen instead of call as shown:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
p1.wait()
and I tried using commuincate but it still didn't work:
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True).communicate()
this shell script works great on MacOS (after changing paths) and waits when using subprocess.call(['sh', '/userA/Test/run-my-script.sh'])
but on Windows this is what happens, I run the below python script in Blender then once it gets to the subprocess line Git bash is opened and runs the shell script while blender doesn't wait for it to finish it just prints Hello in its console without waiting for the Git Bash to finish. Any help?
import bpy
import subprocess
subprocess.call(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
print('Hello')
You can use subprocess.call to do exactly that.
subprocess.call(args, *, stdin=None, stdout=None, stderr=None, shell=False, timeout=None)
Run the command described by args. Wait for command to complete, then return the returncode attribute.
Edit: I think I have a hunch on what's going on. The command works on your Mac because Macs, I believe, support Bash out of the box (at least something functionally equivalent) while on Windows it sees your attempt to run a ".sh" file and instead fires up Git Bash which I presume performs a couple forks when starting.
Because of this Python thinks that your script is done, the PID is gone.
If I were you I would do this:
Generate a unique, non-existing, absolute path in your "launching" script using the tempfile module.
When launching the script, pass the path you just made as an argument.
When the script starts, have it create a file at the path. When done, delete the file.
The launching script should watch for the creation and deletion of that file to indicate the status of the script.
Hopefully that makes sense.
You can use Popen.communicate API.
p1 = subprocess.Popen(os.path.abspath('D:/Test/run-my-script.sh'),shell=True)
sStdout, sStdErr = p1.communicate()
The command
Popen.communicate(input=None, timeout=None)
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for the process to terminate.
subprocess.run will by default wait for the process to finish.
Use subprocess.Popen and Popen.wait:
process = subprocess.Popen(['D:/Test/run-my-script.sh'],shell=True, executable="/bin/bash")
process.wait()
You could also use check_call() instead of Popen.
You can use os.system, like this:
import bpy
import os
os.system("sh "+os.path.abspath('D:/Test/run-my-script.sh'))
print('Hello')
There are apparently cases when the run command fails.
This is my workaround:
def check_has_finished(pfi, interval=1, timeout=100):
if os.path.exists(pfi):
if pfi.endswith('.nii.gz'):
mustend = time.time() + timeout
while time.time() < mustend:
try:
# Command is an ad hoc one to check if the process has finished.
subprocess.check_output('command {}'.format(pfi), shell=True)
except subprocess.CalledProcessError:
print "Caught CalledProcessError"
else:
return True
time.sleep(interval)
msg = 'command {0} not working after {1} tests. \n'.format(pfi, timeout)
raise IOError(msg)
else:
return True
else:
msg = '{} does not exist!'.format(pfi)
raise IOError(msg)
A wild try, but are you running the shell as Admin while Blender as regular user or vice versa?
Long story short (very short), Windows UAC is a sort of isolated environment between admin and regular user, so random quirks like this can happen. Unfortunately I can't remember the source of this, the closest I found is this.
My problem was the exact opposite of yours, the wait() got stuck in a infinite loop because my python REPL was fired from an admin shell and wasn't able to read the state of the regular user subprocess. Reverting to normal user shell got it fixed. It's not the first time I'm bit from this UAC snafu.
Environment: Raspberry Pi Wheezy
I have a python program that uses Popen to call another python program
from subprocess import *
oJob = Popen('sudo python mypgm.py',shell=True)
Another menu option is supposed to end the job immediately
oJob.kill()
but the job is still running??
When you add the option shell=True, python launches a shell and the shell in turn launches the process python mymgm.py. You are killing the shell process here which doesn't kill its own child that runs mymgm.py.
To ensure, that child process gets killed on oJob.kill, you need to group them all under one process group and make shell process, the group leader.
The code is,
import os
import signal
import subprocess
# The os.setsid() is passed in the argument preexec_fn so
# it's run after the fork() and before exec() to run the shell.
pro = subprocess.Popen(cmd, stdout=subprocess.PIPE,
shell=True, preexec_fn=os.setsid)
os.killpg(pro.pid, signal.SIGTERM) # Send the signal to all the process groups
When you send SIGTERM signal to the shell process, it will kill all its child process as well.
You need to add a creation flag arg
oJob = Popen('sudo python mypgm.py',shell=True, creationflags = subprocess.CREATE_NEW_PROCESS_GROUP)
source
subprocess.CREATE_NEW_PROCESS_GROUP
A Popen creationflags parameter to specify that a new process group will be created. This flag is necessary for using os.kill() on the subprocess.
EDIT I agree with the comment on how to import stuff and why you are getting something is undefined. Also the other answer seems to be on the right track getting the pid
import subprocess as sub
oJob = sub.Popen('sudo python mypgm.py', creationflags = sub.CREATE_NEW_PROCESS_GROUP)
oJob.kill()
Warning Executing shell commands that incorporate unsanitized input from an untrusted source makes a program vulnerable to shell injection, a serious security flaw which can result in arbitrary command execution. For this reason, the use of shell=True is strongly discouraged in cases where the command string is constructed from external input:
I'm trying to kill a subprocess started with:
playing_long = Popen(["omxplayer", "/music.mp3"], stdout=subprocess.PIPE)
and after a while
pid = playing_long.pid
playing_long.terminate()
os.kill(pid,0)
playing_long.kill()
Which doesn't work.
Neither the solution pointed out here
How to terminate a python subprocess launched with shell=True
Noting that I am using threads, and it is not recommended to use preexec_fn when you use threads (or at least this is what I read, anyway it doesn't work either).
Why it is not working? There's no error message in the code, but I have to manually kill -9 the process to stop listening the mp3 file.
Thanks
EDIT:
From here, I have added a wait() after the kill().
Surprisingly, before re-starting the process I check if this is still await, so that I don't start a chorus with the mp3 file.
Without the wait(), the system sees that the process is alive.
With the wait(), the system understands that the process is dead and starts again it.
However, the process is still sounding. Definitively I can't seem to get it killed.
EDIT2: The problem is that omxplayer starts a second process that I don't kill, and it's the responsible for the actual music.
I've tried to use this code, found in several places in internet, it seems to work for everyone but not for me
playing_long.stdin.write('q')
playing_long.stdin.flush()
And it prints 'NoneType' object has no attribute 'write'. Even when using this code immediately after starting the popen process, it fails with the same message
playing_long = subprocess.Popen(["omxplayer", "/home/pi/Motion_sounds/music.mp3"], stdout=subprocess.PIPE)
time.sleep(5)
playing_long.stdin.write('q')
playing_long.stdin.flush()
EDIT3: The problem then was that I wasn't establishing the stdin line in the popen line. Now it is
playing_long = subprocess.Popen(["omxplayer", "/home/pi/Motion_sounds/music.mp3"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
time.sleep(5)
playing_long.stdin.write(b'q')
playing_long.stdin.flush()
*needing to specify that it is bytes what I write in stdin
Final solution then (see the process edited in the question):
playing_long = subprocess.Popen(["omxplayer", "/home/pi/Motion_sounds/music.mp3"], stdin=subprocess.PIPE, stdout=subprocess.PIPE)
time.sleep(5)
playing_long.stdin.write(b'q')
playing_long.stdin.flush()
I'm running a tool via Python in cmd. For each sample in a given directory I want that tool to do something. However, when I use process = subprocess.Popen(command) in the loop, the commands does not wait untill its finished, resulting in 10 prompts at once. And when I use subprocess.Popen(command, stdout=subprocess.PIPE) the command remains black and I can't see the progress, although it does wait untill the command is finished.
Does anyone know a way how to call an external tool via Python in cmd, that does wait untill the command is finished and thats able to show the progress of the tool in the cmd?
#main.py
for sample in os.listdir(os.getcwd()):
if ".fastq" in sample and '_R1_' in sample and "Temp" not in sample:
print time.strftime("%H:%M:%S")
DNA_Bowtie2.DNA_Bowtie2(os.getcwd()+'\\'+sample+'\\'+sample)
#DNA_Bowtie2.py
# Run Bowtie2 command and wait for process to be finished.
process = subprocess.Popen(command, stdout=subprocess.PIPE)
process.wait()
process.stdout.read()
Edit: command = a perl or java command. With above make-up I cannot see tool output since the prompt (perl window, or java window) remains black.
It seems like your subprocess forks otherwise there is no way the wait() would return before the process has finished.
The order is important here: first read the output, then wait.
If you do it this way:
process.wait()
process.stdout.read()
you can experience a deadlock if the pipe buffer is completely full: the subprocess blocks on waiting on stdout and never reaches the end, your program blocks on wait() and never reaches the read().
Do instead
process.stdout.read()
process.wait()
which will read until EOF.
This holds for if you want the stdout of the process at all.
If you don't want that, you should omit the stdout=PIPE stuff. Then the output is directed into that prompt window. Then you can omit process.stdout.read() as well.
Normally, the process.wait() should then prevent that 10 instances run at once. If that doesn't work, I don't know why not...