We Keep Coding

No such file or directory when attempting to execute subprocess call

Preface: I understand this question has been asked before, but I cannot find a solution to my error from looking at those previous answers.
All I want to do is call diff for the output of two different commands on the same file.
import os, sys
from subprocess import check_call
import shlex
ourCompiler = 'espressoc';
checkCompiler = 'espressocr';
indir = 'Tests/Espresso/GoodTests';
check_call(["pwd"]);
for root, dirs, filenames in os.walk(indir):
for f in filenames:
if len(sys.argv) == 2 and sys.argv[1] == f:
str1 = "<(./%s ./%s) " % (ourCompiler, os.path.join(root, f))
str2 = "<(./%s ./%s) " % (checkCompiler, os.path.join(root, f))
check_call(["diff", str1, str2])
Why is it that I receive following error?
diff: <(./espressoc ./Tests/Espresso/GoodTests/Init.java) : No such file or directory
diff: <(./espressocr ./Tests/Espresso/GoodTests/Init.java) : No such file or directory
Traceback (most recent call last):
File "runTest.py", line 21, in <module>
check_call(["diff", str1, str2])
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['diff', '<(./espressoc ./Tests/Espresso/GoodTests/Init.java) ', '<(./espressocr ./Tests/Espresso/GoodTests/Init.java) ']' returned non-zero exit status 2
If I were to run this command from my shell it works fine.

diff is complaining that it can't find the file with the strange name <(./espressoc ./Tests/Espresso/GoodTests/Init.java) because that's the argument you fed it.
subprocess.Popen (check_call is a convenience function for it) is directly calling what you give it, there isn't a shell to interpret redirections or anything, unless you tell it shell=True, which will then call the command via /bin/sh (on POSIX). Note the security considerations before using it.
So basically:
subprocess.check_call(['diff', '<this', '<that'])` # strange files.
subprocess.check_call('diff <this <that', shell=True)` # /bin/sh does some redirection
If you wanted to be "pure" (probably more effort than it's worth), I think you could subprocess all three processes (diff, compiler 1 and 2) and handle the piping yourself. Does diff wait for 2 EOFs or something before closing stdin? Not sure how it actually deals with the double input redirection like your line has...

Python subprocess.check_call() not recognising pushd and popd

I'm on Ubuntu 15.04 (not by choice obviously) and Python 3.4.3 and I'm trying to execute something like the following.
subprocess.check_call("pushd /tmp", shell=True)
I need the shell=True because the actual code I'm trying to execute contains wildcards that need to be interpreted. However, this gives me the following error.
/usr/lib/python3.4/subprocess.py in check_call(*popenargs, **kwargs)
559 if cmd is None:
560 cmd = popenargs[0]
--> 561 raise CalledProcessError(retcode, cmd)
562 return 0
563
CalledProcessError: Command 'pushd /tmp' returned non-zero exit status 127
I've tried doing the same thing on my Mac (El Capitan and Python 3.5.1) and it works perfectly. I've also tried executing subprocess.check_call("ls", shell=True) on the Ubuntu 15.04 with Python 3.4.3 (for sanity check), and it works fine. As a final sanity check, I've tried the command pushd /tmp && popd in Bash on the Ubuntu 15.04 and that works fine too. So somehow, on (my) Ubuntu 15.04 with Python 3.4.3, subprocess.check_call() does not recognise pushd and popd! Why?

You have two problems with your code. The first one is that the shell used by default is /bin/sh which doesn't support pushd and popd.
In your question you failed to provide the whole error output, and at the top of it you should see the line:
/bin/sh: 1: popd: not found
The next time rememeber to post the whole error message, and not just the portion that you (incorrectly) think is relevant.
You can fix this by telling the subprocess module which shell to use via the executable argument:
>>> subprocess.check_call('pushd ~', shell=True, executable='/bin/bash')
~ ~
0
The second problem is that even with this you will get an error if you use multiple check_call calls:
>>> subprocess.check_call('pushd ~', shell=True, executable='/bin/bash')
~ ~
0
>>> subprocess.check_call('popd', shell=True, executable='/bin/bash')
/bin/bash: riga 0: popd: stack delle directory vuoto
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.5/subprocess.py", line 581, in check_call
raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command 'popd' returned non-zero exit status 1
This is because every call to check_call starts a new subshells, and thus it doesn't matter whether you previously called pushd because the directory stack will always be empty.
Note that if you try to combine pushd and popd in a single call they do work:
>>> subprocess.check_call('pushd ~ && popd', shell=True, executable='/bin/bash')
~ ~
~
0
Now fact is, if you were thinking of using pushd and popd in that way from python... they are useless. That's because you can specify the current working directory via the cwd argument, and so you can keep track of the stack of working directories from python without having to rely on pushd and popd:
current_working_dirs = []
def pushd(dir):
current_working_dirs.append(os.path.realpath(os.path.expanduser(dir)))
def popd():
current_working_dirs.pop()
def run_command(cmdline, **kwargs):
return subprocess.check_call(cmdline, cwd=current_working_dirs[-1], **kwargs)
Replace check_call('pushd xxx') with pushd('xxx') and check_call('popd') with popd and use run_command(...) instead of check_call(...).
As you suggest a more elegant solution would be to use a context manager:
class Pwd:
dir_stack = []
def __init__(self, dirname):
self.dirname = os.path.realpath(os.path.expanduser(self.dirname))
def __enter__(self):
Pwd.dir_stack.append(self.dirname)
return self
def __exit__(self, type, value, traceback):
Pwd.dir_stack.pop()
def run(self, cmdline, **kwargs):
return subprocess.check_call(cmdline, cwd=Pwd.dir_stack[-1], **kwargs)
used as:
with Pwd('~') as shell:
shell.run(command)
with Pwd('/other/directory') as shell:
shell.run(command2) # runs in '/other/directory'
shell.run(command3) # runs in '~'

How to use several subprocess.check_output and avoid non-zero exit status 67

I'm at the beginning of learning python, so it may be something obvious. I'm trying to create a script that will change the desktop pictures in OS X 10.9 according to group membership.
If I only check for one thing, the script works. It's when I try to expend on it that I fail.
I've tried putting all the declarations before the if else, also tried it with if elif else.
How does one get python to use several subprocess.check_output in a row?
The script dies with this traceback:
Traceback (most recent call last):
File "/Library/SetDesktopImages/SetDesktopImages.py", line 36, in module
CheckStudentUser = subprocess.check_output(['dseditgroup', '-o' , 'checkmember', '-m', username, 'students'])
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['dseditgroup', '-o', 'checkmember', '-m', 'root', 'students']' returned non-zero exit status 67
If I run the command itself, it is fine:
$ dseditgroup -o checkmember -m root students
no root is NOT a member of students
Here is my script so far:
#!/usr/bin/python
# This script checks the user's membership to groups in order to set the Desktop Picture.
from AppKit import NSWorkspace, NSScreen
from Foundation import NSURL
import getpass
import subprocess
network_user_path = "/Library/Desktop Pictures/Abstract.jpg"
ladmin_path = "/Library/Desktop Pictures/Desert.jpg"
luser_path = "/Library/Desktop Pictures/Flamingos.jpg"
# generate a fileURL
nu_file_url = NSURL.fileURLWithPath_(network_user_path)
la_file_url = NSURL.fileURLWithPath_(ladmin_path)
lu_file_url = NSURL.fileURLWithPath_(luser_path)
username=getpass.getuser()
options = {}
ws = NSWorkspace.sharedWorkspace()
CheckAdminUser = subprocess.check_output(['dseditgroup', '-o' , 'checkmember', '-m', username, 'admin'])
IsAdmin = subprocess.check_output(['echo', CheckAdminUser, '|', 'cut', '-d', '-f1', '|', 'awk', '{print $1}'])
if IsAdmin == 'yes':
for screen in NSScreen.screens():
(result, error) = ws.setDesktopImageURL_forScreen_options_error_(
la_file_url, screen, options, None)
else:
CheckStudentUser = subprocess.check_output(['dseditgroup', '-o' , 'checkmember', '-m', username, 'students'])
IsStudent = subprocess.check_output(['echo', CheckStudentUser, '|', 'cut', '-d', '-f1', '|', 'awk', '{print $1}'])
if IsStudent == 'yes':
for screen in NSScreen.screens():
(result, error) = ws.setDesktopImageURL_forScreen_options_error_(
nu_file_url, screen, options, None)
else:
CheckLocalUser = subprocess.check_output(['dseditgroup', '-o' , 'checkmember', '-m', username, 'localaccounts'])
IsLocal = subprocess.check_output(['echo', CheckLocalUser, '|', 'cut', '-d', '-f1', '|', 'awk', '{print $1}'])
if IsLocal == 'yes':
for screen in NSScreen.screens():
(result, error) = ws.setDesktopImageURL_forScreen_options_error_(
lu_file_url, screen, options, None)
# not sure about this ending
else:
exit0

A non-zero exit status means there was an error, or other abnormal exit.
Its clear from your command output that there was an error, in fact the error message is shown as a result, the difference is your shell suppresses the error code. To see what the exit code was, type echo "$?", here is an example:
$ dseditgroup -o checkmember -m root students
Group not found.
$ echo "$?"
64
Here the command returned the exit code 64. The output was written to stderr the standard error buffer. In the shell it looks the same because it will print both stdout (standard output) and stderr, but notice if I redirect stderr to a file, you don't see any output:
$ dseditgroup -o checkmember -m root students 2&> ~/out.txt
$ echo "$?"
64
$ cat ~/out.txt
Group not found.
You can avoid the exception by capturing any output that is written to stderr in your script. This is also detailed in the documentation:
check_admin_user = subprocess.check_output(
['dseditgroup', '-o' , 'checkmember', '-m', username, 'admin'],
stderr=subprocess.STDOUT
)
I also changed your variable names as per the Python style guide.

subprocess module in python

I have to connect to a sybase database and run a simple select query using python script
On my server isql command can run only from sybase bin directory, so i have to cd to that directory before firing the query.
---------------------------Edited-----------------------------
Uptill now i'm able to do this:-
#!/usr/bin/python
import subprocess
path = "path/to/sybase/bin"
os.chdir(path)
arguments = ['./isql',"-S server_name", "-U user", "-P password", "-D database","""<<EOF
SELECT * FROM sometable
go
EOF"""]
ps = subprocess.Popen(arguments)
out = ps.communicate()
print out
The errors are just out of my understanding capability :(
Traceback (most recent call last):
File "./test_db.py", line 8, in ?
ps = subprocess.Popen(arguments)
File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 975, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm able to do this outside my python script using isql command on my unix terminal
how can i use isql in python subprocess module?

There is a particular Popen argument for that: cwd, as mentioned here. Provide your command using an array and then the cwd parameter with where the command must be executed:
subprocess.Popen(['ls', '-l'], cwd="/path/to/folder")

Popen only takes one args argument, for the command to run. You could try calling a shell with both the cd and isql commands as arguments, but changing the working directory from python is probably simpler
For the former approach:
subprocess.Popen('/bin/sh -c "cd /path/to/... && isql -arg1..'...)
for the latter:
os.chdir('/path/to...')
subprocess.Popen('isql -arg1..'...)

Try:
import os
import subprocess
os.chdir('/path/to/sybase/bin')
if os.path.exists('isql') or os.path.exists(os.path.join('/path/to/sybase/bin', 'isql')):
ps = subprocess.Popen('isql -S %s -U %s -P %s -D %s <<EOF SELECT * FROM sometable EOF' % (server,user,passwd,database), stdout=subprocess.PIPE, shell=True)
out, err = ps.communicate()
else:
print "isql does not exists in this folder"
I am not super experienced with subprocess but this is how I generally use it on the odd occasion. Hopefully someone else can give a better answer/explanation.
Edit: removed the square brackets to remove confusion.

i know it's been long but just wanted to close this question
from subprocess import Popen, PIPE
from textwrap import dedent
isql = Popen(['./isql', '-I', '/app/sybase/...',
'-S', mdbserver,
'-U', muserid,
'-P', password, ...,
'-w', '99999'], stdin=PIPE, stdout=PIPE, cwd=sybase_path)
output = isql.communicate(dedent("""\
SET NOCOUNT ON
{}
go
""".format(User_Query)))[0]

call IPtables command within python script

I am trying to call the following command in my python script. I am trying to insert the rule into IP tables. I am using sub process to call it and inserting variables into it where needed, but am getting a large error. Any suggestions?
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
Error:
Traceback (most recent call last):
File "./port_forward.py", line 42, in <module>
iptables = subprocess.call('iptables -I FORWARD -i eth0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Your problem is very common for Python beginners. Instead of formatting the string command,
you are trying to build a complex string by concatenating many strings and variables. Instead, use a
a string format, it will allow you to test your command and make your code more readable and flexible.
Your original string lacks spaces between the options and the arguments, e.g. --to-destination1.2.3.4.
Hence, you should format your string (This works also for python 2.7):
opts = {'iptables': '/sbin/iptables', 'protocol': 'tcp', 'port': 80, 'ipAddress': '0.0.0.0'}
ipcmd = '{iptables} -I FORWARD -eth 0 -m {protocol} -t {protocol} \
--dport {port} -j DNAT --to-destination {ipAddress}'.format(**opts)
if DEBUG:
print ipcmd
iptables = subprocess.call(ipcmd)
This is much easier to modify later, and also, when you do more Python programming, you will see that it is more readable.
Also, to call IPTables, you should be root, as stated in the comments:
In the beginning of your script add:
import sys
import os
if not os.getuid() == 0:
print "You must be root to change IPTables."
sys.exit(2)
update after seeing your error trace:
You are trying to call a command iptables but it is not in your path.
You should call the full path of iptables , e.g. /sbin/iptables

I wrote a simple firewall the same way and realized, "why not just write it in bash?". Anyway I discovered the python-iptables library and rewrote my code using that. I recommend checking it out. I think it will give you a more robust and structured way of writing iptables rules.

Your command is full of syntax errors due to missing spaces, as shown below:
iptables = subprocess.call(
'iptables -I FORWARD -eth 0 -m '
+ protocol
+ ' -t'+protocol
^---here
+ '--dport '
^-- here
+ port
+ '-j DNAT --to-destination'
^--here
+ ipAddress)
^---here
As generated, your iptables line will look like
-ttcp--dport 80-j DNAT --to-destination1.2.3.4
-ttcp--dport is parsed as a SINGLE argument, ditto for 80-j and --to-destination1.2.3.4

Just pass the argument shell=True along with the command.
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress, shell=True)