We Keep Coding

Python Subprocess: Fortran runtime error: End of file

I'm trying to run a few commands in a Athena Vortex Lattice using Python subprocess, but it keeps throwing errors:
C:\Users\Myself\AppData\Local\Programs\Python\Python35\python.exe C:/Users/Myself/Documents/aerodynamics/analyze_cases.py
Root: C:\Users\Myself\Documents\aerodynamics
At line 145 of file ../src/userio.f (unit = 5, file = 'stdin')
Fortran runtime error: End of file
Traceback (most recent call last):
File "C:/Users/Myself/Documents/aerodynamics/analyze_cases.py", line 31, in <module>
process.communicate(b'\n')
File "C:\Users\Myself\AppData\Local\Programs\Python\Python35\lib\subprocess.py", line 1043, in communicate
Loaded
raise ValueError("Cannot send input after starting communication")
ValueError: Cannot send input after starting communication
Process finished with exit code 1
This is the code used:
import time
import subprocess
import os
root = os.getcwd()
print("Root: ", root)
# Start AVL Program
process = subprocess.Popen([root+r"/avl.exe "], shell = True, stdin=subprocess.PIPE, bufsize=1, stdout=subprocess.PIPE)
time.sleep(2)
# Start program with LOAD and filename:
process.communicate(input=b"LOAD "+root.encode()+b"\input_cases\sample.avl \n")
time.sleep(2)
print("Loaded")
process.communicate(b'\n')
time.sleep(5)
print("Leaving")
# process.communicate(b'\n')
process.communicate(b'\n')
time.sleep(0.5)
process.communicate(b'QUIT')
process.kill()
My thoughts: It appears at the first communicate statement already (before Loaded) and crashes when it tries to fire off the second command to a now nonexistent process.
My Theory: Judging from the log there may be something going on with unit = 5, file = 'stdin' (why is file equal to stdin?) but I don't have a clue how to solve that.
There are a few similar questions around here, and I've tried the following hacks:
shell true/false
encode() and bitstring stuff
subprocess communicate instead of stdin.write
Same issue appears with wine on Mac. Program runs nominally with identical commands directly command line outside Python.

Here's a code example where some issues in your code are fixed. You should consider whether you could get rid of time.sleep() too:
#!/usr/bin/env python3
import os
import time
from subprocess import Popen, PIPE, DEVNULL
# start AVL Program
with Popen(os.path.abspath("avl.exe"), stdin=PIPE, stdout=DEVNULL, bufsize=1,
universal_newlines=True) as process:
time.sleep(2)
# start program with LOAD and filename:
print("LOAD " + os.path.abspath(r"input_cases\sample.avl"), file=process.stdin)
time.sleep(2)
print(file=process.stdin) # send newline
time.sleep(5)
print(file=process.stdin) # send newline
time.sleep(0.5)
print("QUIT", file=process.stdin)

python script using subprocess, redirect ALL output to file

I am writing something for static analysis of source code in different languages. As anything has to be open source and callable from command line I now have downloaded one tool per language. So I decided to write a python script listing all source files in a project folder and calling the respective tool.
So part of my code looks like this:
import os
import sys
import subprocess
from subprocess import call
from pylint.lint import Run as pylint
class Analyser:
def __init__(self, source=os.getcwd(), logfilename=None):
# doing initialization stuff
self.logfilename = logfilename or 'CodeAnalysisReport.log'
self.listFiles()
self.analyseFiles()
def listFiles(self):
# lists all source files in the specified directory
def analyseFiles(self):
self.analysePythons()
self.analyseCpps()
self.analyseJss()
self.analyseJavas()
self.analyseCs()
if __name__ == '__main__':
Analyser()
Let's have at a look at the C++ files part (I use Cppcheck to analyse those):
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile])
The console output for one of the files (it's just a random downloaded file) is:
**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
(information) Cppcheck cannot find all the include files (use --check-config for details)
Line 1 and 2 coming from my script, lines 3 to 7 coming from Cppcheck.
And this is what I want to save to my log file, for all the other files too. Everything in one single file.
Of course I have searched SO and found some methods. But none is working completely.
First try:
Adding sys.stdout = open(self.logfilename, 'w') to my constructor. This makes line 1 and 2 of the above showed output be written to my log file. The rest is still shown on console.
Second try:
Additionaly, in analyseCpps I use:
call(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout)
This makes my log file to be:
Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc...
**********************************************************************
C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc
and the console output is:
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:18]: (style) The scope of the variable 'oldi' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:43]: (style) The scope of the variable 'lastbit' can be reduced.
[C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc:44]: (style) The scope of the variable 'two_to_power_i' can be reduced.
Not what I want.
Third try:
Using Popen with pipe. sys.stdout is back to default.
As preliminary work analyseCpps now is:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=subprocess.PIPE)
p.stdout.read()
p.stdout.read() shows only the last line of my desired output (line 7 in code box 3)
Fourth try:
Using subprocess.Popen(['C:\CodeAnalysis\cppcheck\cppcheck', '--enable=all', sourcefile], stdout=open(self.logfilename, 'a+')) just writes the one line Checking C:\CodeAnalysis\testproject\cpp\BiggestUnInt.cc... to my logfile, the rest is shown on the console.
Fifth try:
Instead of subprocess.Popen I use os.system, so my calling command is:
os.system('C:\CodeAnalysis\cppcheck\cppcheck --enable=all %s >> %s' % (sourcefile, self.logfilename))
This results in the same log file as my fourth try. If I type the same command directly in the windows console the result is the same. So I guess it it is not exactly a python problem but still:
If it is on the console there must be a way to put it in a file. Any ideas?
E D I T
Foolish me. I'm still a noob so I forgot about the stderr. That's where the decisive messages are going to.
So now I have:
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stderr=subprocess.PIPE)
with open(self.logfilename, 'a+') as logfile:
logfile.write('%s\n%s\n' % ('*'*70, sourcefile))
for line in p.stderr.readlines():
logfile.write('%s\n' % line.strip())
and it's working fine.
ANOTHER EDIT
according to Didier's answer:
with sys.stdout = open(self.logfilename, 'w', 0) in my constructor:
def analyseCpps(self):
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
p = subprocess.Popen(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile], stdout=sys.stdout, stderr=sys.stdout)

There are several problems:
you should redirect both stdout and stderr
you should use unbuffered files if you want to mix normal print and the output of launched commands.
Something like this:
import sys, subprocess
# Note the 0 here (unbuffered file)
sys.stdout = open("mylog","w",0)
print "Hello"
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
subprocess.call(["./prog"],stdout=sys.stdout, stderr=sys.stdout)
print "-----"
print "End"

You need to redirect stderr too, you can use STDOUT or pass the file object to stderr=:
from subprocess import check_call,STDOUT
with open("log.txt","w") as f:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
check_call(['C:\\CodeAnalysis\\cppcheck\\cppcheck', '--enable=all', sourcefile],
stdout=f, stderr=STDOUT)

Try to redirect stdout and stderr to a logfile:
import subprocess
def analyseCpps(self):
with open("logfile.txt", "w") as logfile:
for sourcefile in self.files['.cc'] + self.files['.cpp']:
print '\n'*2, '*'*70, '\n', sourcefile
call(['C:\\CodeAnalysis\\cppcheck\\cppcheck',
'--enable=all', sourcefile], stdout=logfile,
stderr=subprocess.STDOUT)
In this example the filename is hardcoded, but you should be able to change that easily (to your self.logfilename or similar).

opening git diff in sublime using python for custom command

The core.editor of my git is sublime, but I am launching sublime in my custom git command made in python, so how can i pass git diff HEAD^ HEAD to sublime as argument in python
I have stored the value of core.editor in configdb['core.editor'] which i can launch using subprocess.Popen but passing git diff HEAD^ HEAD as argument opens 4 tabs with title git, diff, HEAD^, HEAD... how should I make any sublime launched with git diff into which i can add my own message that I can store in a variable using python.
# Read git config file
configFile, _ = execGitCommand('git config --list')
configDb = {}
for line in filter(None, configFile.split("\n")):
configDb[line.split("=")[0]] = line.split("=")[-1]
now configDb['core.editor'] = /Applications/Sublime_Text.app/Contents/SharedSupport/bin/subl -n -w
and then
diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
msg, err, = subprocess.Popen(configDb['core.editor'].split(" ")[0], stdin=diff.stdout)
but executing the last line above does opens the diff in sublime but gives below error
Traceback (most recent call last):
File "/Users/san/Development/executables//git-ipush", line 186, in <module>
sys.exit(main())
File "/Users/san/Development/executables//git-ipush", line 43, in main
preCheck(args)
File "/Users/san/Development/executables//git-ipush", line 55, in preCheck
msg, err, = subprocess.Popen(configDb['core.editor'].split(" ")[0], stdin=diff.stdout)
TypeError: 'Popen' object is not utterable
and terminal is now not waiting for sublime to finish editing, but it should as I am passing -w flag as you can see above. The code is a part of this git-command

The following worked for me in the Python console of Sublime Text 3 (after I had already set my current working directory to my git working copy):
>>> import subprocess
>>> diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
>>> subprocess.Popen(['/usr/local/bin/subl'], stdin=diff.stdout)
edit:
OK, I think I understand what you want now. You want to edit the text of the diff before you send it into Sublime, right? The following worked for me:
import subprocess, tempfile
diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
with tempfile.TemporaryFile() as f:
f.write('Hello, World\n')
f.write(diff.stdout.read())
f.seek(0)
subprocess.Popen(['/usr/local/bin/subl'], stdin=f)
This is just a sample. If you actually want to modify the contents of the diff itself, you can read it into a string first. Note that you can't use StringIO (which I think would be better), because somewhere in the second Popen somebody needs a fileno.
edit 2:
Here's how you get the text from Sublime into a variable
import sublime
window = sublime.active_window()
view = window.active_view()
region = sublime.Region(0, view.size())
text = view.substr(region)

python subprocess.Popen

I am having a difficult time understanding how to get python to call a system command with the subprocess.Popen function.
the_file = ('logs/consolidated.log.gz')
webstuff = subprocess.Popen(['/usr/bin/zgrep', '/meatsauce/', the_file ],stdout=subprocess.PIPE)
for line in webstuff.stdout:
print line
Trying to get python to build another file with my search string.

The problem is in how you're constructing your arguments. The way you have it now, you're running:
/usr/bin/zgrep /meatsauce/ logs/consolidated.log.gz
Note the space between /meatsauce/ and logs...
To do what I think you're intending, use os.path.join a la:
import os
the_file = 'logs/consolidated.log.gz'
webstuff = subprocess.Popen(['/usr/bin/zgrep', os.path.join('/meatsauce/', the_file)],stdout=subprocess.PIPE) % dpt_search
for line in webstuff.stdout:
print line

Not exactly sure about your question, but the following snippet will call zgrep with two arguments, a searchterm and a filename - and print the result (stdout) line by line:
#!/usr/bin/env python
# -*- coding: utf-8 -*-
import subprocess
# filename and searchterm
fn, term = 'access_log.gz', 'hello'
p = subprocess.Popen(['/usr/bin/zgrep', term, fn], stdout=subprocess.PIPE)
for line in p.stdout:
print line
In the code you posted the string interpolation (% dpt_search) does not work out, since there is not pure string in front of the modulo sign - in fact it should fail with something like:
TypeError: "unsupported operand type(s) for %: 'Popen' and 'str'"

the_file = ('webalizerlogs/consolidated.log.gz')
output_f = open('output.txt','w')
webstuff = subprocess.Popen(['/usr/bin/zgrep', dpt_search, the_file ],stdout=output_f)

I think you are simply trying to grep the a content in the a file. Is it?
import os
import subprocess
the_file = os.path.join(os.getcwd(),'logs/consolidated.log.gz')
proc = subprocess.Popen(['/usr/bin/zgrep', dpt_search, the_file], stdout=subprocess.PIPE)
out, err = proc.communicate()
with open('resultoutput','w') as f:
f.write(out)
subprocess.call(['/usr/bin/zip',os.path.join(os.getcwd(),'resultoutput'])
Check the docs as well.

How to redirect stderr in Python?

I would like to log all the output of a Python script. I tried:
import sys
log = []
class writer(object):
def write(self, data):
log.append(data)
sys.stdout = writer()
sys.stderr = writer()
Now, if I "print 'something' " it gets logged. But if I make for instance some syntax error, say "print 'something# ", it wont get logged - it will go into the console instead.
How do I capture also the errors from Python interpreter?
I saw a possible solution here:
http://www.velocityreviews.com/forums/showpost.php?p=1868822&postcount=3
but the second example logs into /dev/null - this is not what I want. I would like to log it into a list like my example above or StringIO or such...
Also, preferably I don't want to create a subprocess (and read its stdout and stderr in separate thread).

I have a piece of software I wrote for work that captures stderr to a file like so:
import sys
sys.stderr = open('C:\\err.txt', 'w')
so it's definitely possible.
I believe your problem is that you are creating two instances of writer.
Maybe something more like:
import sys
class writer(object):
log = []
def write(self, data):
self.log.append(data)
logger = writer()
sys.stdout = logger
sys.stderr = logger

You can't do anything in Python code that can capture errors during the compilation of that same code. How could it? If the compiler can't finish compiling the code, it won't run the code, so your redirection hasn't even taken effect yet.
That's where your (undesired) subprocess comes in. You can write Python code that redirects the stdout, then invokes the Python interpreter to compile some other piece of code.

I can't think of an easy way. The python process's standard error is living on a lower level than a python file object (C vs. python).
You could wrap the python script in a second python script and use subprocess.Popen. It's also possible you could pull some magic like this in a single script:
import os
import subprocess
import sys
cat = subprocess.Popen("/bin/cat", stdin=subprocess.PIPE, stdout=subprocess.PIPE)
os.close(sys.stderr.fileno())
os.dup2(cat.stdin.fileno(), sys.stderr.fileno())
And then use select.poll() to check cat.stdout regularly to find output.
Yes, that seems to work.
The problem I foresee is that most of the time, something printed to stderr by python indicates it's about to exit. The more usual way to handle this would be via exceptions.
---------Edit
Somehow I missed the os.pipe() function.
import os, sys
r, w = os.pipe()
os.close(sys.stderr.fileno())
os.dup2(w, sys.stderr.fileno())
Then read from r

To route the output and errors from Windows, you can use the following code outside of your Python file:
python a.py 1> a.out 2>&1
Source: https://support.microsoft.com/en-us/help/110930/redirecting-error-messages-from-command-prompt-stderr-stdout

Since python 3.5 you can use contextlib.redirect_stderr
with open('help.txt', 'w') as f:
with redirect_stdout(f):
help(pow)

For such a request, usually it would be much easier to do it in the OS instead of in Python.
For example, if you're going to run "a.py" and record all the messages it will generate into file "a.out", it would just be
python a.py 2>&1 > a.out
The first part 2>&1 redirects stderr to stdout (0: stdin, 1:stdout, 2:stderr), and the second redirects that to a file called a.out.
And as far as I know, this command works in Windows, Linux or MacOS! For other file redirection techniques, just search the os plus "file redirection"

I found this approach to redirecting stderr particularly helpful. Essentially, it is necessary to understand if your output is stdout or stderr. The difference? Stdout is any output posted by a shell command (think an 'ls' list) while sterr is any error output.
It may be that you want to take a shell commands output and redirect it to a log file only if it is normal output. Using ls as an example here, with an all files flag:
# Imports
import sys
import subprocess
# Open file
log = open("output.txt", "w+")
# Declare command
cmd = 'ls -a'
# Run shell command piping to stdout
result = subprocess.run(cmd, stdout=subprocess.PIPE, shell=True)
# Assuming utf-8 encoding
txt = result.stdout.decode('utf-8')
# Write and close file
log.write(txt)
log.close()
If you wanted to make this an error log, you could do the same with stderr. It's exactly the same code as stdout with stderr in its place. This pipes an error messages that get sent to the console to the log. Doing so actually keeps it from flooding your terminal window as well!
Saw this was a post from a while ago, but figured this could save someone some time :)

import sys
import tkinter
# ********************************************
def mklistenconsswitch(*printf: callable) -> callable:
def wrapper(*fcs: callable) -> callable:
def newf(data):
[prf(data) for prf in fcs]
return newf
stdoutw, stderrw = sys.stdout.write, sys.stderr.write
funcs = [(wrapper(sys.stdout.write, *printf), wrapper(sys.stderr.write, *printf)), (stdoutw, stderrw)]
def switch():
sys.stdout.write, sys.stderr.write = dummy = funcs[0]
funcs[0] = funcs[1]
funcs[1] = dummy
return switch
# ********************************************
def datasupplier():
i = 5.5
while i > 0:
yield i
i -= .5
def testloop():
print(supplier.__next__())
svvitch()
root.after(500, testloop)
root = tkinter.Tk()
cons = tkinter.Text(root)
cons.pack(fill='both', expand=True)
supplier = datasupplier()
svvitch = mklistenconsswitch(lambda text: cons.insert('end', text))
testloop()
root.mainloop()

Python will not execute your code if there is an error. But you can import your script in another script an catch exceptions. Example:
Script.py
print 'something#
FinalScript.py
from importlib.machinery import SourceFileLoader
try:
SourceFileLoader("main", "<SCRIPT PATH>").load_module()
except Exception as e:
# Handle the exception here

To add to Ned's answer, it is difficult to capture the errors on the fly during the compilation.
You can write several print statements in your script and you can stdout to a file, it will stop writing to the file when the error occurs. To debug the code you could check the last logged output and check your script after that point.
Something like this:
# Add to the beginning of the script execution(eg: if __name__ == "__main__":).
from datetime import datetime
dt = datetime.now()
script_dir = os.path.dirname(os.path.abspath(__file__)) # gets the path of the script
stdout_file = script_dir+r'\logs\log'+('').join(str(dt.date()).split("-"))+r'.log'
sys.stdout = open(stdout_file, 'w')
This will create a log file and stream the print statements to the file.
Note: Watch out for escape characters in your filepath while concatenating with script_dir in the second line from the last in the code. You might want something similar to raw string. You can check this thread for this.