I'm trying to parse some data and make graphs with python and there's an odd issue coming up. A call to os.system() seems to get lost somewhere.
The following three lines:
os.system('echo foo bar')
os.system('gnuplot test.gnuplot')
os.system('gnuplot --version')
Should print:
foo bar
Warning: empty x range [2012:2012], adjusting to [1991.88:2032.12]
gnuplot 4.4 patchlevel 2
But the only significant command in the middle seems to get dropped. The script still runs the echo and version check, and running gnuplot by itself (the gnuplot shell) works too, but there is no warning and no file output from gnuplot.
Why is this command dropped, and why completely silently?
In case it's helpful, the invocation should start gnuplot, it should open a couple of files (the instructions and a data file indicated therein) and write out to an SVG file. I tried deleting the target file so it wouldn't have to overwrite, but to no avail.
This is python 3.2 on Ubuntu Natty x86_64 virtual machine with the 2.6.38-8-virtual kernel.
Is the warning printed to stderr, and that is intercepted somehow?
Try using subprocess instead, for example using
subprocess.check_output(cmd, stderr=subprocess.STDOUT)
and checking the output.
(or plaing subprocess.call might work better than os.system)
So, it turned out the issue was something I failed to mention. Earlier in the script test.gnuplot and test.data were written, but I neglected to call the file objects' close() and verify that they got closed (still don't know how to do that last part so for now it cycles for a bit). So there was some unexpected behaviour going on there causing gnuplot to see two unreadable files, take no action, produce no output, and return 0.
I guess nobody gets points for this one.
Edit: I finally figured it out with the help of strace. Don't know how I did things before I learned how to use it.
don't use os.system. Use subprocess module.
os.system documentation says:
The subprocess module provides more powerful facilities for spawning
new processes and retrieving their results; using that module is
preferable to using this function.
Try this:
subprocess.check_call(['gnuplot', 'test.gnuplot'])
Related
I am using a 3rd-party python module which is normally called through terminal commands. When called through terminal commands it has a verbose option which prints to terminal in real time.
I then have another python program which calls the 3rd-party program through subprocess. Unfortunately, when called through subprocess the terminal output no longer flushes, and is only returned on completion (the process takes many hours so I would like real-time progress).
I can see the source code of the 3rd-party module and it does not set printing to be flushed such as print('example', flush=True). Is there a way to force the flushing through my module without editing the 3rd-party source code? Furthermore, can I send this output to a log file (again in real time)?
Thanks for any help.
The issue is most likely that many programs work differently if run interactively in a terminal or as part of a pipe line (i.e. called using subprocess). It has very little to do with Python itself, but more with the Unix/Linux architecture.
As you have noted, it is possible to force a program to flush stdout even when run in a pipe line, but it requires changes to the source code, by manually applying stdout.flush calls.
Another way to print to screen, is to "trick" the program to think it is working with an interactive terminal, using a so called pseudo-terminal. There is a supporting module for this in the Python standard library, namely pty. Using, that, you will not explicitly call subprocess.run (or Popen or ...). Instead you have to use the pty.spawn call:
def prout(fd):
data = os.read(fd, 1024)
while(data):
print(data.decode(), end="")
data = os.read(fd, 1024)
pty.spawn("./callee.py", prout)
As can be seen, this requires a special function for handling stdout. Here above, I just print it to the terminal, but of course it is possible to do other thing with the text as well (such as log or parse...)
Another way to trick the program, is to use an external program, called unbuffer. Unbuffer will take your script as input, and make the program think (as for the pty call) that is called from a terminal. This is arguably simpler if unbuffer is installed or you are allowed to install it on your system (it is part of the expect package). All you have to do then, is to change your subprocess call as
p=subprocess.Popen(["unbuffer", "./callee.py"], stdout=subprocess.PIPE)
and then of course handle the output as usual, e.g. with some code like
for line in p.stdout:
print(line.decode(), end="")
print(p.communicate()[0].decode(), end="")
or similar. But this last part I think you have already covered, as you seem to be doing something with the output.
I have a build.sh script that my automated build server executes as part of a build. A big portion of logic of the build is calculating and building a version number. All of this logic is in a python script such as calculate-version.py.
Typically what I would do in this case is setup the python script to ONLY print the version number, from which I would read stdout from the bash script, and assign that to an environment variable. However, the python script is becoming sufficiently complex that I'd like to start adding logs to it.
I need to be able to output (stdout) logs from the Python script (via print()) while at the same time when it is done, propagate a "return value" from the python script back to the parent shell script.
What is the best way of doing this? I thought of doing this through environment variables, but my understanding is those won't be available to the parent process.
Short answer: you can't. The return value of a *nix-style executable is an unsigned integer from 0-255. That usually indicates if it failed or not, but you could co-opt it for your own uses.
In this case, I don't think a single unsigned byte is enough. Thus, you need to output it some other way. You have a few options
The simplest (and probably best in this case) is to continue outputting your output data on stdout, and send your logs/debugging information somewhere else. That could be to a file, or (it's sort-of what it's for) stderr
Output your data to a file (such as one given in a command line parameter)
Arrange some kind of named pipe scheme. In practice, this is pretty much the same thing as sending it to a file.
Create an executable python script that prints a variable print 99.
#!/usr/bin/python
print 99
chmod a+x test.py to set it as executable.
From bash do this a=$(./test.py) and if you print a(echo $a) you should get 99.
To get only the version number, you should print only the version number.
I am using Python 2.7.3 in Ubuntu 12.04 OS. I have an external program say 'xyz' whose input is a single file and two files say 'abc.dat' and 'gef.dat' are its output.
When I used os.system or subprocess.check_output or os.popen none of them printed the output files in the working directory.
I need these output files for further calculations.
Plus I've to keep calling the 'xyz' program 'n' times and have to keep getting the output 'abc.dat' and 'gef.dat' every time from it. Please help.
Thank you
I can not comment on your question because my reputation is too low.
If you use os.system or subprocess.check_output or os.popen, you will just get the standard output of your xyz program (if it is printing something in the screen). To see the files in some directory, you can use os.listdir(). Then you can use these files in your script afterwards. It may also be worth using subprocess.check_call.
There may be other better and more efficient solutions.
First, run the program which you invoked in python script directly and see if it generates those two files.
Assume it does, the problem is in your python script. Try using subprocess.Popen then call communicate().
Here's an example:
from subprocess import Popen
p = Popen(["xyz",])
p.communicate()
communicate waits for process to terminate. you should be able to get output files when executing code after p.communicate().
Thank you for answering my question but the answer to my question is this -
import subprocess
subprocess.call("/path/to/software/xyz abc.dat", shell=True)
which gave me the desired the output.
I tried the subprocess-related commands but they returned error " No such file or directory". The 'shell=True' worked like a charm.
Thank you all again for taking your time to answer my question.
I'm writing a python script that performs a series of operations in a loop, by making subprocess calls, like so:
os.system('./svm_learn -z p -t 2 trial-input model')
os.system('./svm_classify test-input model pred')
os.system('python read-svm-rank.py')
score = os.popen('python scorer.py -g gold-test -i out').readline()
When I make the calls individually one after the other in the shell they work fine. But within the script they always break. I've traced the source of the error and it seems that the output files are getting truncated towards the end (leading me to believe that calls are being made without previous ones being completed).
I tried with subprocess.Popen and then using the wait() method of the Popen object, but to no avail. The script still breaks.
Any ideas what's going on here?
I'd probably first rewrite a little to use the subprocess module instead of the os module.
Then I'd probably scrutinize what's going wrong by studying a system call trace:
http://stromberg.dnsalias.org/~strombrg/debugging-with-syscall-tracers.html
Hopefully there'll be an "E" error code near the end of the file that'll tell you what error is being encountered.
Another option would be to comment out subsets of your subprocesses (assuming the n+1th doesn't depend heavily on the output of the nth), to pin down which one of them is having problems. After that, you could sprinkle some extra error reporting in the offending script to see what it's doing.
But if you're not put off by C-ish syscall traces, that might be easier.
From my understanding, os.popen() opens a pipe within Python and initiates a new sub process. I have a problem when I run a for loop in conjunction with os.popen(). I can't seem to CTRL+C out of the loop. Here is my code:
for FILE in os.popen("ls $MY_DIR/"):
os.system("./processFile " + FILE)
Whenever I try to CTRL+C, Python will stop the ./processFile program but NOT the python program itself!
I have Google'd around and couldn't seem to find the correct answer. Some people recommend using SIGNALS (I tried... it didn't work). Another tried to use PIDs and killing child PIDs but I couldn't seem to get it.
Can someone lead me to a better example so I can stop the programming when I use CTRL+C (SIGINT) ?
I see some answer correctly recommended subprocess.check_call and the OP in a comment said
I'm getting this error:
AttributeError: 'module' object has no
attribute 'check_call'
Per the docs I just linked to, check_call is marked as:
New in version 2.5.
so it looks like the OP is using some ancient version of Python -- 2.4 or earlier -- without mentioning the fact (the current production-ready version is 2.7, and 2.4 is many years old).
The best one can recommend, therefore, is to upgrade! If 2.7 is "too new" for your tastes (as it might be considered in a conservative "shop"), 2.6's latest microrelease should at least be fine -- and it won't just give you subprocess.check_call, but also many additional feautures, bug fixes, and optimizations!-)
The behavior is correct. Ctrl+C stops the foreground process and not its parent process. Calling the shell and using ls is inappropriate here, your code should better be written as follows (untested):
import os
import subprocess
for fname in os.listdir(directory):
path = os.path.join(directory, fname)
subprocess.check_call(["./processFile", path])