We Keep Coding

python sending argument to a running process

I have a flutter project called zed, my goal is to monitor the output of flutter run, as long as pressing r, the output will increase.
To automatically implement this workflow, my implementation is
import subprocess
bash_commands = f'''
cd ../zed
flutter run --device-id web-server --web-hostname 192.168.191.6 --web-port 8352
'''
process = subprocess.Popen('/bin/bash', stdin=subprocess.PIPE, stdout=subprocess.PIPE, shell=False)
output, err= process.communicate(bash_commands.encode('utf-8'))
print(output, err)
output, _ = process.communicate('r'.encode('utf-8'))
print(output)
It's not working as I expected, there is nothing printed on the screen.

Use process.stdin.write() instead of process.communicate()
process.stdin.write(bash_commands)
process.stdin.flush()
But why you ask
Popen.communicate(input=None, timeout=None)
Interact with process:
Send data to stdin. Read data from stdout and stderr, until
end-of-file is reached
https://docs.python.org/3/library/subprocess.html#subprocess.Popen.communicate
communicate(...) doesn't return until the pipe is closed which typically happens when the subprocess closes. Great for ls -l not so good for long running subprocesses.

subprocess.Popen(): change stderr during child's execution

Goal: I'm trying to put a Python script together that captures the network traffic that occurs as a result of the execution of a block of code. For simplicity, let's assume I want to log the network traffic resulting from a call to socket.gethostbyname('example.com'). Note: I can't just simply terminate tcpdump when gethostbyname() returns as the actual code block that I want to measure triggers other external code, and I have no way to determine when this external code finishes execution (so I have to leave tcpdump running "long enough" for it to be highly probable that I logged all traffic generated by this external code).
Approach: I'm using subprocess to start tcpdump, telling tcpdump to terminate itself after duration seconds using its -G and -W options, e.g.:
duration = 15
nif = 'en0'
pcap = 'dns.pcap'
cmd = ['tcpdump', '-G', str(duration), '-W', '1', '-i', nif, '-w', pcap]
tcpdump_proc = subprocess.Popen(cmd)
socket.gethostbyname('example.com')
time.sleep(duration + 5) # sleep longer than tcpdump is running
The problem with this is that Popen() returns before tcpdump is fully up and running, thus some/all of the traffic resulting from the call to gethostbyname() will not be captured. I could obviously add a time.sleep(x) before calling gethostbyname() to give tcpdump a bit of time to spin up, but that's not a portable solution (I can't just pick some arbitrary x < duration as a powerful system would start capturing packets earlier than a less powerful system).
To deal with this, my idea is to parse tcpdump's output to look for when the following is written to its stderr as that appears to indicate that the capture is up and running fully:
tcpdump: listening on en0, link-type EN10MB (Ethernet), capture size 262144 bytes
Thus I need to attach to stderr, but the problem is that I don't want to commit to reading all of its output as I need my code to move on to actually execute the code block I want to measure (gethostbyname() in this example) instead of being stuck in a loop reading from stderr.
I could solve this by adding a semaphore that blocks the main thread from proceeding onto the gethostbyname() call, and have a background thread read from stderr and decrement the semaphore (to let the main thread move on) when it reads the string above from stderr, but I'd like to keep the code single-threaded if possible.
From my understanding, it's a big NONO to use subprocess.PIPE for stderr and stdout without committing to reading all of the output as the child will end up blocking when the buffer fills up. But can you "detach" (destroy?) the pipe mid execution if you're only interested in reading the first part of the output? Essentially I'd like to end up with something like this:
duration = 15
nif = 'en0'
pcap = 'dns.pcap'
cmd = ['tcpdump', '-G', str(duration), '-W', '1', '-i', nif, '-w', pcap]
tcpdump_proc = subprocess.Popen(cmd, stderr=subprocess.PIPE, text=True)
for l in tcpdump_proc.stderr:
if 'tcpdump: listening on' in l:
break
socket.gethostbyname('example.com')
time.sleep(duration) # sleep at least as long as tcpdump is running
What else do I need to add within the if block to "reassign" who's in charge of reading stderr? Can I just set stderr back to None (tcpdump_proc.stderr = None)? Or should I call tcpdump_proc.stderr.close() (and will tcpdump terminate early if I do so)?
It could also very well be that I missed something obvious and that there is a much better approach to achieve what I want - if so, please enlighten me :).
Thanks in advance :)

You could use detach() or close() on stderr after recieving the listening on message:
import subprocess
import time
duration = 10
nif = 'eth0'
pcap = 'dns.pcap'
cmd = ['tcpdump', '-G', str(duration), '-W', '1', '-i', nif, '-w', pcap]
proc = subprocess.Popen(
cmd, shell=False, stderr=subprocess.PIPE, bufsize=1, text=True
)
for i, line in enumerate(proc.stderr):
print('read %d lines from stderr' % i)
if 'listening on' in line:
print('detach stderr!')
proc.stderr.detach()
break
while proc.poll() is None:
print("doing something else while tcpdump is runnning!")
time.sleep(2)
print(proc.returncode)
print(proc.stderr.read())
Out:
read 0 lines from stderr
detach stderr!
doing something else while tcpdump is runnning!
doing something else while tcpdump is runnning!
doing something else while tcpdump is runnning!
doing something else while tcpdump is runnning!
doing something else while tcpdump is runnning!
doing something else while tcpdump is runnning!
0
Traceback (most recent call last):
File "x.py", line 24, in <module>
print(proc.stderr.read())
ValueError: underlying buffer has been detached
Note:
I haven't checked what is really happening to the stderr data, but detaching stderr doesn't seem to have any impact on tcpdump.

Can't close an SSH connection opened with Popen

I created a class method (this will only run on Linux) that sends a list of commands to a remote computer over SSH and returns the output using subprocess.Popen:
def remoteConnection(self, list_of_remote_commands):
ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
# send ssh commands to stdin
for command in list_of_remote_commands:
ssh.stdin.write(command + "\n")
ssh.stdin.close()
output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}
return output_dict
Whilst I'm still getting to grips with the subprocess module I'd read quite a bit about Popen and no one ever mentioned closing it (SSH Connection with Python 3.0, Proper way to close all files after subprocess Popen and communicate, https://docs.python.org/2/library/subprocess.html) so I assumed that that wasn't a problem.
However when testing this out in ipython outside of a function I noticed that the variable ssh still seemed active. I tried closing ssh.stdin, ssh.stdout and ssh.stderr and even ssh.close(), ssh.terminate() and ssh.kill() but nothing seemed to close it. I thought perhaps it doesn't matter but my function will be called many times for months or even years so I don't want it to spawn a new process everytime it is run otherwise I'm going to quickly use up my maximum processes limit. So I use ssh.pid to find the PID and look it up using ps aux | grep PID and it's still there even after doing all of the above.
I also tried:
with subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0) as shh:
instead of:
ssh = subprocess.Popen(["ssh", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
I also remember solving a similar problem a while back using ssh -T but even:
ssh = subprocess.Popen(["ssh", "-T", self.ssh_connection_string], stdin=subprocess.PIPE, stdout=subprocess.PIPE, stderr=subprocess.PIPE,universal_newlines=True, bufsize=0)
Didn't work.
I'm sure I would have found something about closing Popen if I needed to but then why is the process still open on my computer - can anyone help me understand what's going on here?

In your case, you have a deadlock here:
output_dict = {'stdin': list(ssh.stdin), 'stdout': list(ssh.stdout), 'stderr': list(ssh.stderr)}
Mostly because list(ssh.stdin) blocks forever: trying to read standard input of a process doesn't work (there's also an extra risk because you redirected both standard output & error to different pipes without using threading to consume them)
You mean to use ssh.communicate, passing the whole input as argument. Simply do:
command_input = "".join(["{}\n".format(x) for x in list_of_remote_commands])
output,error = ssh.communicate(command_input) # may need .encode() for python 3
return_code = ssh.wait()
then
output_dict = {'stdin': list_of_commands, 'stdout': output.splitlines(), 'stderr': error.splitlines()}
I may add that in the particular ssh case, using paramiko module is better (python paramiko ssh) and avoids using subprocess completely.

Python subprocesses (ffmpeg) only start once I Ctrl-C the program?

I'm trying to run a few ffmpeg commands in parallel, using Cygwin and Python 2.7.
This is roughly what I have:
import subprocess
processes = set()
commands = ["ffmpeg -i input.mp4 output.avi", "ffmpeg -i input2.mp4 output2.avi"]
for cmd in commands:
processes.add(
subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
)
for process in processes:
if process.poll() is None:
process.wait()
Now, once I am at the end of this code, the whole program waits. All the ffmpeg processes are created, but they're idle, i.e., using 0% CPU. And the Python program just keeps waiting. Only when I hit Ctrl-C, it suddenly starts encoding.
What am I doing wrong? Do I have to "send" something to the processes to start them?

This is only a guess, but ffmpeg usually produces a lot of status messages and output on stderr or stdout. You're using subprocess.PIPE to redirect stdout and stderr to a pipe, but you never read from those, so if the pipe buffer is full, the ffmpeg process will block when trying to write data to it.
When you kill the parent process the pipe is closed on its end, and probably (i haven't checked) ffmpeg handles the error by just not writing to the pipe anymore and is therefore unblocked and starts working.
So eiter consume the process.stdout and process.stderr pipes in your parent process, or redirect the output to os.devnull if you don't care about it.

In addition to what #mata says, ffmpeg may also be asking you if you want to overwrite output.avi and waiting on you to type "y". To force-overwrite, use the "-y" command-line option (ffmpeg -i $input -y $output).

Popen subprocessing problems

I'm trying to learn about the subprocessing module and am therefore making a hlds server administrator.
My goal is to be able to start server instances and send all commands through dispatcher.py to administrate multiple servers, e.g. send commands to subprocesses stdin.
what I've got so far for some initial testing, but got stuck already :]
#dispatcher.py
import subprocess
RUN = '/home/daniel/hlds/hlds_run -game cstrike -map de_dust2 -maxplayers 11'
#RUN = "ls -l"
hlds = subprocess.Popen(RUN.split(), stdout=subprocess.PIPE, stdin=subprocess.PIPE, stderr=subprocess.PIPE)
print hlds.communicate()[0]
print hlds.communicate()[1]
hlds.communicate('quit')
I am not getting any stdout from the hlds server, but it works fine if i dont set stdout to PIPE. And the hlds.communicate('quit') does not seem to be sent to the hlds process stdin either. The ls -l command returns stdout correctly but not hlds.
All help appreciated! :)

See the Popen.communicate docs (emphasis mine):
Interact with process: Send data to stdin. Read data from stdout and stderr, until end-of-file is reached. Wait for process to terminate. The optional input argument should be a string to be sent to the child process, or None, if no data should be sent to the child.
So you can only call communicate once per run of a process, since it waits for the process to terminate. That's why ls -l seems to work -- it terminates immediately, while hlds doesn't.
You'd need to do:
out, error = hlds.communicate('quit')
if you want to send in quit and get all output until it terminates.
If you need more interactivity, you'll need to use hlds.stdout, hlds.stdin, and hlds.stderr directly.