We Keep Coding

Python3/Linux - Open text file in default editor and wait until done

I need to wait until the user is done editing a text file in the default graphical application (Debian and derivates).
If I use xdg-open with subprocess.call (which usually waits) it will continue after opening the file in the editor. I assume because xdg-open itself starts the editor asynchronously.
I finally got a more or less working code by retrieving the launcher for the text/plain mime-type and use that with Gio.DesktopAppInfo.new to get the command for the editor. Provided that the editor is not already open in which case the process ends while the editor is still open.
I have added solutions checking the process.pid and polling for the process. Both end in an indefinite loop.
It seems such a overly complicated way to wait for the process to finish. So, is there a more robust way to do this?
#! /usr/bin/env python3
import subprocess
from gi.repository import Gio
import os
from time import sleep
import sys
def open_launcher(my_file):
print('launcher open')
app = subprocess.check_output(['xdg-mime', 'query', 'default', 'text/plain']).decode('utf-8').strip()
print(app)
launcher = Gio.DesktopAppInfo.new(app).get_commandline().split()[0]
print(launcher)
subprocess.call([launcher, my_file])
print('launcher close')
def open_xdg(my_file):
print('xdg open')
subprocess.call(['xdg-open', my_file])
print('xdg close')
def check_pid(pid):
""" Check For the existence of a unix pid. """
try:
os.kill(int(pid), 0)
except OSError:
return False
else:
return True
def open_pid(my_file):
pid = subprocess.Popen(['xdg-open', my_file]).pid
while check_pid(pid):
print(pid)
sleep(1)
def open_poll(my_file):
proc = subprocess.Popen(['xdg-open', my_file])
while not proc.poll():
print(proc.poll())
sleep(1)
def open_ps(my_file):
subprocess.call(['xdg-open', my_file])
pid = subprocess.check_output("ps -o pid,cmd -e | grep %s | head -n 1 | awk '{print $1}'" % my_file, shell=True).decode('utf-8')
while check_pid(pid):
print(pid)
sleep(1)
def open_popen(my_file):
print('popen open')
process = subprocess.Popen(['xdg-open', my_file])
process.wait()
print(process.returncode)
print('popen close')
# This will end the open_xdg function while the editor is open.
# However, if the editor is already open, open_launcher will finish while the editor is still open.
#open_launcher('test.txt')
# This solution opens the file but the process terminates before the editor is closed.
#open_xdg('test.txt')
# This will loop indefinately printing the pid even after closing the editor.
# If you check for the pid in another terminal you see the pid with: [xdg-open] <defunct>.
#open_pid('test.txt')
# This will print None once after which 0 is printed indefinately: the subprocess ends immediately.
#open_poll('test.txt')
# This seems to work, even when the editor is already open.
# However, I had to use head -n 1 to prevent returning multiple pids.
#open_ps('test.txt')
# Like open_xdg, this opens the file but the process terminates before the editor is closed.
open_popen('test.txt')

Instead of trying to poll a PID, you can simply wait for the child process to terminate, using subprocess.Popen.wait():
Wait for child process to terminate. Set and return returncode attribute.
Additionally, getting the first part of get_commandline() is not guaranteed to be the launcher. The string returned by get_commandline() will match the Exec key spec, meaning the %u, %U, %f, and %F field codes in the returned string should be replaced with the correct values.
Here is some example code, based on your xdg-mime approach:
#!/usr/bin/env python3
import subprocess
import shlex
from gi.repository import Gio
my_file = 'test.txt'
# Get default application
app = subprocess.check_output(['xdg-mime', 'query', 'default', 'text/plain']).decode('utf-8').strip()
# Get command to run
command = Gio.DesktopAppInfo.new(app).get_commandline()
# Handle file paths with spaces by quoting the file path
my_file_quoted = "'" + my_file + "'"
# Replace field codes with the file path
# Also handle special case of the atom editor
command = command.replace('%u', my_file_quoted)\
.replace('%U', my_file_quoted)\
.replace('%f', my_file_quoted)\
.replace('%F', my_file_quoted if app != 'atom.desktop' else '--wait ' + my_file_quoted)
# Run the default application, and wait for it to terminate
process = subprocess.Popen(
shlex.split(command), stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
process.wait()
# Now the exit code of the text editor process is available as process.returncode
I have a few remarks on my sample code.
Remark 1: Handling spaces in file paths
It is important the file path to be opened is wrapped in quotes, otherwise shlex.split(command) will split the filename on spaces.
Remark 2: Escaped % characters
The Exec key spec states
Literal percentage characters must be escaped as %%.
My use of replace() then could potentially replace % characters that were escaped. For simplicity, I chose to ignore this edge case.
Remark 3: atom
I assumed the desired behaviour is to always wait until the graphical editor has closed. In the case of the atom text editor, it will terminate immediately on launching the window unless the --wait option is provided. For this reason, I conditionally add the --wait option if the default editor is atom.
Remark 4: subprocess.DEVNULL
subprocess.DEVNULL is new in python 3.3. For older python versions, the following can be used instead:
with open(os.devnull, 'w') as DEVNULL:
process = subprocess.Popen(
shlex.split(command), stdout=DEVNULL, stderr=DEVNULL)
Testing
I tested my example code above on Ubuntu with the GNOME desktop environment. I tested with the following graphical text editors: gedit, mousepad, and atom.

python subprocess module stuck while executing a batch file

I am trying to learn to code using python on my own but I ran into a problem.
I am using python's subprocess module to execute a .bat file, but the process seems to get stuck at the bat file. The python code currently looks like this:
import getpass
username = getpass.getuser()
from subprocess import Popen
p = Popen("hidefolder.bat", cwd=r"C:\Users\%s\Desktop" % username)
stdout, stderr = p.communicate()
import sys
sys.exit()
And the .bat file looks like this:
if exist "C:\Users\%username%\Desktop\HiddenFolder\" (
attrib -s -h "HiddenFolder"
rename "HiddenFolder" "Projects"
exit
)
if exist "C:\Users\%username%\Desktop\Projects\" (
rename "Projects" "HiddenFolder"
attrib +s +h "HiddenFolder"
exit
)
if not exist "C:\Users\%username%\Desktop\HiddenFolder\" (
mkdir "C:\Users\%username%\Desktop\HiddenFolder\"
)
exit
Is there a way to kill the child process even if the python script is waiting for the child process to be terminated before continuing? Or is the problem in the child process to start with?
Thank you in advance.

You need to use subprocess.PIPE for stdout and stderr, or else they can't be fetched through Popen.communicate, and is the reason why your process is stuck.
from subprocess import Popen, PIPE
import getpass
username = getpass.getuser()
p = Popen("hidefolder.bat", cwd=r"C:\Users\%s\Desktop" % username, stdout=PIPE, stderr=PIPE)
stdout, stderr = p.communicate()
import sys
sys.exit()

I am a new programmer but i could solve my problem writting below code.
import subprocess
subprocess.call([r'ProcurementSoftwareRun.bat'])
print ('Software run successful')
My bat file was like:
#ECHO OFF
cmd /c start "" "C:\Program Files (x86)\UserName\ERPModule\PROCUREMENT.exe
exit

Stream stdout from subprocess to python function, and back to subprocess

I am trying to use python in a unix style pipe.
For example, in unix I can use a pipe such as:
$ samtools view -h somefile.bam | python modifyStdout.py | samtools view -bh - > processed.bam
I can do this by using a for line in sys.stdin: loop in the python script and that appears to work without problems.
However I would like to internalise this unix command into a python script. The files involved will be large so I would like to avoid blocking behaviour, and basically stream between processes.
At the moment I am trying to use Popen to manage each command, and pass the stdout of the first process to the stdin of the next process, and so on.
In a seperate python script I have (sep_process.py):
import sys
f = open("sentlines.txt", 'wr')
f.write("hi")
for line in sys.stdin:
print line
f.write(line)
f.close()
And in my main python script I have this:
import sys
from subprocess import Popen, PIPE
# Generate an example file to use
f = open('sees.txt', 'w')
f.write('somewhere over the\nrainbow')
f.close()
if __name__ == "__main__":
# Use grep as an example command
p1 = Popen("grep over sees.txt".split(), stdout=PIPE)
# Send to sep_process.py
p2 = Popen("python ~/Documents/Pythonstuff/Bam_count_tags/sep_process.py".split(), stdin=p1.stdout, stdout=PIPE)
# Send to final command
p3 = Popen("wc", stdin=p2.stdout, stdout=PIPE)
# Read output from wc
result = p3.stdout.read()
print result
The p2 process however fails [Errno 2] No such file or directory even though the file exists.
Do I need to implement a Queue of some kind and/or open the python function using the multiprocessing module?

The tilde ~ is a shell expansion. You are not using a shell, so it is looking for a directory called ~.
You could read the environment variable HOME and insert that. Use
os.environ['HOME']
Alternatively you could use shell=True if you can't be bothered to do your own expansion.

Thanks #cdarke, that solved the problem for using simple commands like grep, wc etc. However I was too stupid to get subprocess.Popen to work when using an executable such as samtools to provide the data stream.
To fix the issue, I created a string containing the pipe exactly as I would write it in the command line, for example:
sam = '/Users/me/Documents/Tools/samtools-1.2/samtools'
home = os.environ['HOME']
inpath = "{}/Documents/Pythonstuff/Bam_count_tags".format(home)
stream_in = "{s} view -h {ip}/test.bam".format(s=sam, ip=inpath)
pyscript = "python {ip}/bam_tags.py".format(ip=inpath)
stream_out = "{s} view -bh - > {ip}/small.bam".format(s=sam, ip=inpath)
# Absolute paths, witten as a pipe
fullPipe = "{inS} | {py} | {outS}".format(inS=stream_in,
py=pyscript,
outS=stream_out)
print fullPipe
# Translates to >>>
# samtools view -h test.bam | python ./bam_tags.py | samtools view -bh - > small.bam
I then used popen from the os module instead and this worked as expected:
os.popen(fullPipe)

Multiple pipes in subprocess

I am trying to use Sailfish, which takes multiple fastq files as arguments, in a ruffus pipeline. I execute Sailfish using the subprocess module in python, but <() in the subprocess call does not work even when I set shell=True.
This is the command I want to execute using python:
sailfish quant [options] -1 <(cat sample1a.fastq sample1b.fastq) -2 <(cat sample2a.fastq sample2b.fastq) -o [output_file]
or (preferably):
sailfish quant [options] -1 <(gunzip sample1a.fastq.gz sample1b.fastq.gz) -2 <(gunzip sample2a.fastq.gz sample2b.fastq.gz) -o [output_file]
A generalization:
someprogram <(someprocess) <(someprocess)
How would I go about doing this in python? Is subprocess the right approach?

To emulate the bash process substitution:
#!/usr/bin/env python
from subprocess import check_call
check_call('someprogram <(someprocess) <(anotherprocess)',
shell=True, executable='/bin/bash')
In Python, you could use named pipes:
#!/usr/bin/env python
from subprocess import Popen
with named_pipes(n=2) as paths:
someprogram = Popen(['someprogram'] + paths)
processes = []
for path, command in zip(paths, ['someprocess', 'anotherprocess']):
with open(path, 'wb', 0) as pipe:
processes.append(Popen(command, stdout=pipe, close_fds=True))
for p in [someprogram] + processes:
p.wait()
where named_pipes(n) is:
import os
import shutil
import tempfile
from contextlib import contextmanager
#contextmanager
def named_pipes(n=1):
dirname = tempfile.mkdtemp()
try:
paths = [os.path.join(dirname, 'named_pipe' + str(i)) for i in range(n)]
for path in paths:
os.mkfifo(path)
yield paths
finally:
shutil.rmtree(dirname)
Another and more preferable way (no need to create a named entry on disk) to implement the bash process substitution is to use /dev/fd/N filenames (if they are available) as suggested by #Dunes. On FreeBSD, fdescfs(5) (/dev/fd/#) creates entries for all file descriptors opened by the process. To test availability, run:
$ test -r /dev/fd/3 3</dev/null && echo /dev/fd is available
If it fails; try to symlink /dev/fd to proc(5) as it is done on some Linuxes:
$ ln -s /proc/self/fd /dev/fd
Here's /dev/fd-based implementation of someprogram <(someprocess) <(anotherprocess) bash command:
#!/usr/bin/env python3
from contextlib import ExitStack
from subprocess import CalledProcessError, Popen, PIPE
def kill(process):
if process.poll() is None: # still running
process.kill()
with ExitStack() as stack: # for proper cleanup
processes = []
for command in [['someprocess'], ['anotherprocess']]: # start child processes
processes.append(stack.enter_context(Popen(command, stdout=PIPE)))
stack.callback(kill, processes[-1]) # kill on someprogram exit
fds = [p.stdout.fileno() for p in processes]
someprogram = stack.enter_context(
Popen(['someprogram'] + ['/dev/fd/%d' % fd for fd in fds], pass_fds=fds))
for p in processes: # close pipes in the parent
p.stdout.close()
# exit stack: wait for processes
if someprogram.returncode != 0: # errors shouldn't go unnoticed
raise CalledProcessError(someprogram.returncode, someprogram.args)
Note: on my Ubuntu machine, the subprocess code works only in Python 3.4+, despite pass_fds being available since Python 3.2.

Whilst J.F. Sebastian has provided an answer using named pipes it is possible to do this with anonymous pipes.
import shlex
from subprocess import Popen, PIPE
inputcmd0 = "zcat hello.gz" # gzipped file containing "hello"
inputcmd1 = "zcat world.gz" # gzipped file containing "world"
def get_filename(file_):
return "/dev/fd/{}".format(file_.fileno())
def get_stdout_fds(*processes):
return tuple(p.stdout.fileno() for p in processes)
# setup producer processes
inputproc0 = Popen(shlex.split(inputcmd0), stdout=PIPE)
inputproc1 = Popen(shlex.split(inputcmd1), stdout=PIPE)
# setup consumer process
# pass input processes pipes by "filename" eg. /dev/fd/5
cmd = "cat {file0} {file1}".format(file0=get_filename(inputproc0.stdout),
file1=get_filename(inputproc1.stdout))
print("command is:", cmd)
# pass_fds argument tells Popen to let the child process inherit the pipe's fds
someprogram = Popen(shlex.split(cmd), stdout=PIPE,
pass_fds=get_stdout_fds(inputproc0, inputproc1))
output, error = someprogram.communicate()
for p in [inputproc0, inputproc1, someprogram]:
p.wait()
assert output == b"hello\nworld\n"

Python 2.5 and Tskill

I had the tskill operation working fine through a python script when using Python 2.7 but when I switched to 2.5 this code fails.
import os
import time
import win32com.client
os.startfile('cmd')
shell = win32com.client.Dispatch("WScript.Shell")
shell.AppActivate('Administrator:C:\Windows\System32\cmd.exe')
time.sleep(2)
shell.SendKeys('tskill java + {ENTER}')
time.sleep(2)
shell.Sendkeys('tskill javaw + {ENTER}')
#shell.SendKeys('exit + {ENTER}')
I use os.startfile to run a python script that opens and runs the application(It's a graphics tool).
Note: The application is already running by the time this script is called, hence I don't need to open it here.
Can some one please tell me how to close an application successfully using this or similar method on Python 2.5. Thanks for the help!

I don't have tskill on my system, but I think it's similar to taskkill /F /FI "IMAGENAME eq java*". Anyway, here's how to call a normal console application in the background (i.e. a hidden window) using the subprocess module (tested in Python 2.5.2, but using taskkill instead):
import subprocess
def tskill(*args):
si = subprocess.STARTUPINFO()
si.dwFlags |= subprocess.STARTF_USESHOWWINDOW
command = ('tskill',) + args
rc = subprocess.call(command, startupinfo=si)
return rc == 0
if tskill('java') and tskill('javaw'):
#success
I used call because it only matters whether the call succeeds or fails, but in general you can use subprocess.PIPE to capture stdout and stderr using subprocess.Popen.
Edit: How to kill a process started with the subprocess module
command = ['path/to/executable', 'arg0', 'arg1', '...']
p = subprocess.Popen(command)
#do something
#kill if it's still alive
if p.poll() is None:
p.kill()