I am trying to write a little program in Python (version 3.7.3) with which I can get the out stream of another program while it is running. To emulate this condition I write a very trivial program in python that print a string every 10 seconds.
writecycle.py
import time
while(1):
print("test process")
time.sleep(10)
In the program that I am trying to write I run this process and I try to get the output
mainproc.py
import time
import subprocess
proc=subprocess.Popen(["python","writecycle.py","&"],stdout=subprocess.PIPE,encoding='UTF-8')
print("start reading output")
while(1):
strout=proc.stdout.read()
print("_"+strout)
time.sleep(10)
but I cannot get further the "start reading output" message. The program get "stuck" on the proc.stdout.read() command.
I red some solution that suggest to use subprocess.communicate() but I think that this command does not fit my needs since it wait the process to be terminate for reading the out stream.
Someone else suggest to use subprocess.poll() but I still get stuck on the proc.stdout.read() command.
I tried using bufsize=1 or 0 in the Popen command with no results, or using readline() but nothing.
I don't know if this helps but I am using a Raspberry Pi4 with Raspbian Buster.
I have to the conclusion that this problem in unsolvable. I give myself an explenation of this but I don't know if is the right answer.
The idea comes to me when I tried to redirect the out stream into a file and then read the file. The problem with this approach is that you cannot read the file if it is still open and I cannot close the file if the process is still running. If I understood correctly Linux (and so Raspbian) is a file-based OS so reading from an open "stdout" is like to reading from file opened from another process.
Again, this is the explenation that I give to myself and I do not know if is correct. Maybe one that have more knowledge about Linux OS can tell if this explenation make sense or if it is wrong.
Related
I'm new in this world of python. Recently I have been asked to make an interface between XFoil (an aerodynamics program) and python. After researching a little bit, I found the subprocess module. As the documentation says it's used to "Spawn new processes, connect to their input/output/error pipes, and obtain their return codes."
The problem is that I need some output archives that XFoil creates while its running. If I close the program, the archives are accesible, but if I try to open or read them while the subprocess is still opened it gives me the following error (Although I can see the archive in the folder):
OSError: save not found.
Here the code:
import subprocess
import numpy as np
import os
process = subprocess.Popen(['<xfoil_path>'], stdin=subprocess.PIPE, universal_newlines=True, creationflags = subprocess.CREATE_NEW_PROCESS_GROUP)
airfoil_path = '<path to airfoil>'
process.stdin.write(f'\nload\n{airfoil_path}')
process.stdin.write('\n\n\noper\nalfa\n2\ncpwr\nsave\n')
process.stdin.tell()
print(os.listdir())
c = np.loadtxt('save', skiprows=1)
print(c)
process.stdin.write('\n\n\noper\nalfa\n3\ncpwr\nsave2\n')
stdin.tell is used to get this output archives, but they are not accesible.
Someone knows why this could be happening?
Why do you imagine process.stdin.tell() should "get this output archives"? It retrieves the file pointer's position.
I'm imagining that the actual problem here is that the subprocess doesn't write the files immediately. Maybe just time.sleep(1) before you try to open them, or figure out a way for it to tell you when it's done writing (some OSes let you tell whether another process has a file open for writing, but I have no idea whether this is possible on Windows, let alone reliable).
Sleeping for an arbitrary time is obviously not very robust; you can't predict how long it takes for the subprocess to write out the files. But if that solves your immediate problem, at least you have some idea of what caused it and maybe where to look next.
As an aside, maybe look into the input= keyword parameter for subprocess.run(). If having the subprocess run in parallel is not crucial, that might be more pleasant as well as more robust.
(Converted into an answer from a comment thread.)
I need to run a Python script in a terminal, several times. This script requires me to import some libraries. So every time I call the script in the terminal, the libraries are loaded again, which results in a loss of time. Is there any way I can import the libraries once and for all at the beginning?
(If I try the "naive" way, calling first a script just to import libraries then running my code, it doesn't work).
EDIT: I need to run the script in a terminal because actually it is made to serve in another program developed in Java. The Java code calls the Pythin script in the terminal, reads its result and processes it, then calls it again.
One solution is that you can leave the python script always running and use a pipe to communicate between processes like the code below taken from this answer.
import os, time
pipe_path = "/tmp/mypipe"
if not os.path.exists(pipe_path):
os.mkfifo(pipe_path)
# Open the fifo. We need to open in non-blocking mode or it will stalls until
# someone opens it for writting
pipe_fd = os.open(pipe_path, os.O_RDONLY | os.O_NONBLOCK)
with os.fdopen(pipe_fd) as pipe:
while True:
message = pipe.read()
if message:
print("Received: '%s'" % message)
print("Doing other stuff")
time.sleep(0.5)
The libraries will be unloaded once the script finishes, so the best way you can handle this is to write the script so it can iterate however many times you want, rather than running the whole script multiple times. I would likely use input() (or raw_input() if you're running Python2) to read in however many times you want to iterate over it, or use a library like click to create a command line argument for it.
I'm using subprocess.Popen to launch an external program with arguments, but when I've opened it the script is hanging, waiting for the program to finish and if I close the script the program immediately quits.
I thought I was just using a similar process before without issue, so I'm unsure if I've actually done it wrong or I'm misremembering what Popen can do. This is how I'm calling my command:
subprocess.Popen(["rv", rvFile, '-nc'])
raw_input("= Opened file")
The raw_input part is only there so the user has a chance to see the message and know that the file should be opened now. But what I end up getting is all the information that the process itself is spitting back out as if it were called in the command line directly. My understanding was that Popen made it an independent child process that would allow me to close the script and leave the other process open.
The linked duplicate question does have a useful answer for my purposes, though it's still not working as I want it.
This is the answer. And this is how I changed my code:
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen(["rv", rvFile, '-nc'], creationflags=DETACHED_PROCESS).pid
raw_input("= Opened file")
It works from IDLE but not when I run the py file through the command prompt style interface. It's still tied to that window, printing the output and quitting the program as soon as I've run the script.
The stackoverflow question Calling an external command in python has a lot of useful answers which are related.
Take a look at os.spawnl, it can take a number of mode flags which include NOWAIT, WAIT.
import os
os.spawnl(os.P_NOWAIT, 'some command')
The NOWAIT option will return the process ID of the spawned task.
Sorry for such a short answer but I have not earned enough points to leave comments yet. Anyhow, put the raw_input("= Opened file") inside the file you are actually opening, rather than the program you are opening it from.
If the file you are opening is not a python file, then it will close upon finishing,regardless of what you declare from within python. If that is the case you could always try detaching it from it's parent using:
from subprocess import Popen, CREATE_NEW_PROCESS_GROUP
subprocess.Popen(["rv", rvFile, '-nc'], close_fds = True | CREATE_NEW_PROCESS_GROUP)
This is specifically for running the python script as a commandline process, but I eventually got this working by combining two answers that people suggested.
Using the combination of DETACHED_PROCESS suggested in this answer worked for running it through IDLE, but the commandline interface. But using shell=True (as ajsp suggested) and the DETACHED_PROCESS parameter it allows me to close the python script window and leave the other program still running.
DETACHED_PROCESS = 0x00000008
pid = subprocess.Popen(["rv", rvFile, '-nc'], creationflags=DETACHED_PROCESS, shell=True).pid
raw_input("= Opened file")
Question: Is there a way, using Python, to access the stdout of a running process? This process has not been started by Python.
Context: There is a program called mayabatch, that renders out images from 3D Maya scene files. If I were to run the program from the command line I would see progress messages from mayabatch. Sometimes, artists close these windows, leaving the progress untracable until the program finishes. That led me along this route of trying to read its stdout after it's been spawned by a foreign process.
Background:
OS: Windows 7 64-bit
My research so far: I have only found questions and answers of how to do this if it was a subprocess, using the subprocess module. I also looked briefly into psutil, but I could not find any way to read a process' stdout.
Any help would be really appreciated. Thank you.
I don't think you can get to the stdout of a process outside of the code that created it
The lazy way to is just to pipe the output of mayabatch to a text file, and then poll the text file periodically in your own code so it's under your control, rather than forcing you to wait on the pipe (which is especially hard on Windows, since Windows select doesn't work with the pipes used by subprocess.
I think this is what maya does internally too: by default mayaBatch logs its results to a file called mayaRenderLog.txt in the user's Maya directory.
If you're running mayabatch from the command line or a bat file, you can funnel stdout to a file with a > character:
mayabatch.exe "file.ma" > log.txt
You should be able to poll that text file from the outside using standard python as long as you only open it for reading. The advantage of doing it this way is that you control the frequency at which you check the file.
OTOH If you're doing it from python, it's a little tougher unless you don't mind having your python script idled until the mayabatch completes. The usual subprocess recipe, which uses popen.communicate() is going to wait for an end-of-process return code:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
print test.communicate()[0]
works but won't report until the process dies. But you calling readlines on the process's stdout will trigger the process and report it one line at a time:
test = subprocess.Popen(["mayabatch.exe","filename.mb"], stdout=subprocess.PIPE)
reader = iter(test.subprocess.readlines, "")
for line in reader:
print line
More discussion here
I'm trying to read lines from a pipe and process them, but I'm doing something silly and I can't figure out what. The producer is going to keep producing lines indefinitely, like this:
producer.py
import time
while True:
print 'Data'
time.sleep(1)
The consumer just needs to check for lines periodically:
consumer.py
import sys, time
while True:
line = sys.stdin.readline()
if line:
print 'Got data:', line
else:
time.sleep(1)
When I run this in the Windows shell as python producer.py | python consumer.py, it just sleeps forever (never seems to get data?) It seems that maybe the problem is that the producer never terminates, since if I send a finite amount of data then it works fine.
How can I get the data to be received and show up for the consumer? In the real application, the producer is a C++ program I have no control over.
Some old versions of Windows simulated pipes through files (so they were prone to such problems), but that hasn't been a problem in 10+ years. Try adding a
sys.stdout.flush()
to the producer after the print, and also try to make the producer's stdout unbuffered (by using python -u).
Of course this doesn't help if you have no control over the producer -- if it buffers too much of its output you're still going to wait a long time.
Unfortunately - while there are many approaches to solve that problem on Unix-like operating systems, such as pyexpect, pexpect, exscript, and paramiko, I doubt any of them works on Windows; if that's indeed the case, I'd try Cygwin, which puts enough of a Linux-like veneer on Windows as to often enable the use of Linux-like approaches on a Windows box.
This is about I/O that is bufferized by default with Python. Pass -u option to the interpreter to disable this behavior:
python -u producer.py | python consumer.py
It fixes the problem for me.