I am having hard time parsing the arguments to subprocess.Popen. I am trying to execute a script on my Unix server. The script syntax when running on shell prompt is as follows:
/usr/local/bin/script hostname = <hostname> -p LONGLIST. No matter how I try, the script is not running inside subprocess.Popen
The space before and after "=" is mandatory.
import subprocess
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname = ', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
The above does not work.
And when I use shell=False, I get OSError: [Errno 8] Exec format error
OSError: [Errno 8] Exec format error can happen if there is no shebang line at the top of the shell script and you are trying to execute the script directly. Here's an example that reproduces the issue:
>>> with open('a','w') as f: f.write('exit 0') # create the script
...
>>> import os
>>> os.chmod('a', 0b111101101) # rwxr-xr-x make it executable
>>> os.execl('./a', './a') # execute it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/os.py", line 312, in execl
execv(file, args)
OSError: [Errno 8] Exec format error
To fix it, just add the shebang e.g., if it is a shell script; prepend #!/bin/sh at the top of your script:
>>> with open('a','w') as f: f.write('#!/bin/sh\nexit 0')
...
>>> os.execl('./a', './a')
It executes exit 0 without any errors.
On POSIX systems, shell parses the command line i.e., your script won't see spaces around = e.g., if script is:
#!/usr/bin/env python
import sys
print(sys.argv)
then running it in the shell:
$ /usr/local/bin/script hostname = '<hostname>' -p LONGLIST
produces:
['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
Note: no spaces around '='. I've added quotes around <hostname> to escape the redirection metacharacters <>.
To emulate the shell command in Python, run:
from subprocess import check_call
cmd = ['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
check_call(cmd)
Note: no shell=True. And you don't need to escape <> because no shell is run.
"Exec format error" might indicate that your script has invalid format, run:
$ file /usr/local/bin/script
to find out what it is. Compare the architecture with the output of:
$ uname -m
I will hijack this thread to point out that this error may also happen when target of Popen is not executable. Learnt it hard way when by accident I have had override a perfectly executable binary file with zip file.
Have you tried this?
Out = subprocess.Popen('/usr/local/bin/script hostname = actual_server_name -p LONGLIST'.split(), shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Edited per the apt comment from #J.F.Sebastian
It wouldn't be wrong to mention that Pexpect does throw a similar error
#python -c "import pexpect; p=pexpect.spawn('/usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/pexpect.py", line 430, in __init__
self._spawn (command, args)
File "/usr/lib/python2.7/site-packages/pexpect.py", line 560, in _spawn
os.execv(self.command, self.args)
OSError: [Errno 8] Exec format error
Over here, the openssl_1.1.0f file at the specified path has exec command specified in it and is running the actual openssl binary when called.
Usually, I wouldn't mention this unless I have the root cause, but this problem was not there earlier. Unable to find the similar problem, the closest explanation to make it work is the same as the one provided by #jfs above.
what worked for me is both
adding /bin/bash at the beginning of the command or file you are
facing the problem with, or
adding shebang #!/bin/sh as the first line.
for ex.
#python -c "import pexpect; p=pexpect.spawn('/bin/bash /usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
OpenSSL 1.1.0f 25 May 2017
If you think the space before and after "=" is mandatory, try it as separate item in the list.
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname', '=', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Related
I have to connect to a sybase database and run a simple select query using python script
On my server isql command can run only from sybase bin directory, so i have to cd to that directory before firing the query.
---------------------------Edited-----------------------------
Uptill now i'm able to do this:-
#!/usr/bin/python
import subprocess
path = "path/to/sybase/bin"
os.chdir(path)
arguments = ['./isql',"-S server_name", "-U user", "-P password", "-D database","""<<EOF
SELECT * FROM sometable
go
EOF"""]
ps = subprocess.Popen(arguments)
out = ps.communicate()
print out
The errors are just out of my understanding capability :(
Traceback (most recent call last):
File "./test_db.py", line 8, in ?
ps = subprocess.Popen(arguments)
File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 975, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm able to do this outside my python script using isql command on my unix terminal
how can i use isql in python subprocess module?
There is a particular Popen argument for that: cwd, as mentioned here. Provide your command using an array and then the cwd parameter with where the command must be executed:
subprocess.Popen(['ls', '-l'], cwd="/path/to/folder")
Popen only takes one args argument, for the command to run. You could try calling a shell with both the cd and isql commands as arguments, but changing the working directory from python is probably simpler
For the former approach:
subprocess.Popen('/bin/sh -c "cd /path/to/... && isql -arg1..'...)
for the latter:
os.chdir('/path/to...')
subprocess.Popen('isql -arg1..'...)
Try:
import os
import subprocess
os.chdir('/path/to/sybase/bin')
if os.path.exists('isql') or os.path.exists(os.path.join('/path/to/sybase/bin', 'isql')):
ps = subprocess.Popen('isql -S %s -U %s -P %s -D %s <<EOF SELECT * FROM sometable EOF' % (server,user,passwd,database), stdout=subprocess.PIPE, shell=True)
out, err = ps.communicate()
else:
print "isql does not exists in this folder"
I am not super experienced with subprocess but this is how I generally use it on the odd occasion. Hopefully someone else can give a better answer/explanation.
Edit: removed the square brackets to remove confusion.
i know it's been long but just wanted to close this question
from subprocess import Popen, PIPE
from textwrap import dedent
isql = Popen(['./isql', '-I', '/app/sybase/...',
'-S', mdbserver,
'-U', muserid,
'-P', password, ...,
'-w', '99999'], stdin=PIPE, stdout=PIPE, cwd=sybase_path)
output = isql.communicate(dedent("""\
SET NOCOUNT ON
{}
go
""".format(User_Query)))[0]
The core.editor of my git is sublime, but I am launching sublime in my custom git command made in python, so how can i pass git diff HEAD^ HEAD to sublime as argument in python
I have stored the value of core.editor in configdb['core.editor'] which i can launch using subprocess.Popen but passing git diff HEAD^ HEAD as argument opens 4 tabs with title git, diff, HEAD^, HEAD... how should I make any sublime launched with git diff into which i can add my own message that I can store in a variable using python.
# Read git config file
configFile, _ = execGitCommand('git config --list')
configDb = {}
for line in filter(None, configFile.split("\n")):
configDb[line.split("=")[0]] = line.split("=")[-1]
now configDb['core.editor'] = /Applications/Sublime_Text.app/Contents/SharedSupport/bin/subl -n -w
and then
diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
msg, err, = subprocess.Popen(configDb['core.editor'].split(" ")[0], stdin=diff.stdout)
but executing the last line above does opens the diff in sublime but gives below error
Traceback (most recent call last):
File "/Users/san/Development/executables//git-ipush", line 186, in <module>
sys.exit(main())
File "/Users/san/Development/executables//git-ipush", line 43, in main
preCheck(args)
File "/Users/san/Development/executables//git-ipush", line 55, in preCheck
msg, err, = subprocess.Popen(configDb['core.editor'].split(" ")[0], stdin=diff.stdout)
TypeError: 'Popen' object is not utterable
and terminal is now not waiting for sublime to finish editing, but it should as I am passing -w flag as you can see above. The code is a part of this git-command
The following worked for me in the Python console of Sublime Text 3 (after I had already set my current working directory to my git working copy):
>>> import subprocess
>>> diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
>>> subprocess.Popen(['/usr/local/bin/subl'], stdin=diff.stdout)
edit:
OK, I think I understand what you want now. You want to edit the text of the diff before you send it into Sublime, right? The following worked for me:
import subprocess, tempfile
diff = subprocess.Popen(['git', 'diff', 'HEAD^', 'HEAD'], stdout=subprocess.PIPE)
with tempfile.TemporaryFile() as f:
f.write('Hello, World\n')
f.write(diff.stdout.read())
f.seek(0)
subprocess.Popen(['/usr/local/bin/subl'], stdin=f)
This is just a sample. If you actually want to modify the contents of the diff itself, you can read it into a string first. Note that you can't use StringIO (which I think would be better), because somewhere in the second Popen somebody needs a fileno.
edit 2:
Here's how you get the text from Sublime into a variable
import sublime
window = sublime.active_window()
view = window.active_view()
region = sublime.Region(0, view.size())
text = view.substr(region)
I am trying to call the following command in my python script. I am trying to insert the rule into IP tables. I am using sub process to call it and inserting variables into it where needed, but am getting a large error. Any suggestions?
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
Error:
Traceback (most recent call last):
File "./port_forward.py", line 42, in <module>
iptables = subprocess.call('iptables -I FORWARD -i eth0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
Your problem is very common for Python beginners. Instead of formatting the string command,
you are trying to build a complex string by concatenating many strings and variables. Instead, use a
a string format, it will allow you to test your command and make your code more readable and flexible.
Your original string lacks spaces between the options and the arguments, e.g. --to-destination1.2.3.4.
Hence, you should format your string (This works also for python 2.7):
opts = {'iptables': '/sbin/iptables', 'protocol': 'tcp', 'port': 80, 'ipAddress': '0.0.0.0'}
ipcmd = '{iptables} -I FORWARD -eth 0 -m {protocol} -t {protocol} \
--dport {port} -j DNAT --to-destination {ipAddress}'.format(**opts)
if DEBUG:
print ipcmd
iptables = subprocess.call(ipcmd)
This is much easier to modify later, and also, when you do more Python programming, you will see that it is more readable.
Also, to call IPTables, you should be root, as stated in the comments:
In the beginning of your script add:
import sys
import os
if not os.getuid() == 0:
print "You must be root to change IPTables."
sys.exit(2)
update after seeing your error trace:
You are trying to call a command iptables but it is not in your path.
You should call the full path of iptables , e.g. /sbin/iptables
I wrote a simple firewall the same way and realized, "why not just write it in bash?". Anyway I discovered the python-iptables library and rewrote my code using that. I recommend checking it out. I think it will give you a more robust and structured way of writing iptables rules.
Your command is full of syntax errors due to missing spaces, as shown below:
iptables = subprocess.call(
'iptables -I FORWARD -eth 0 -m '
+ protocol
+ ' -t'+protocol
^---here
+ '--dport '
^-- here
+ port
+ '-j DNAT --to-destination'
^--here
+ ipAddress)
^---here
As generated, your iptables line will look like
-ttcp--dport 80-j DNAT --to-destination1.2.3.4
-ttcp--dport is parsed as a SINGLE argument, ditto for 80-j and --to-destination1.2.3.4
Just pass the argument shell=True along with the command.
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress, shell=True)
I want to call a python script, script1.py, which takes a regex string as argument. This script then connects to another server and calls a script2.py.
script1.py:
regex = sys.argv[1]
log = os.popen('ssh otherserver python /home/log/scripts/script2.py \"%s\"' % regex)
for line in log:
print line.strip()
log.close()
script2.py:
regex = re.compile(sys.argv[1])
for line in ['[1','[ 2]',' a ']:
if regex.search(line):
print line.strip()
The problem I'm running into is that parentheses are breaking the script when run through SSH.
This works:
[foo#bar scripts]$ python script1.py ".*"
[1
[ 2]
a
This doesn't:
[foo#bar scripts]$ python script1.py "(.*)"
bash: -c: line 0: syntax error near unexpected token ('
bash: -c: line 0:python /home/log/scripts/script2.py (.*)'
Why isn't the argument in the second example escaped? This does not occur if I call script2.py locally without SSH.
Update with solution:
Turns out the solution was using subprocess, but passing arguments didn't work, so I had to format the last argument to include the escaped regex:
log = subprocess.check_output(['ssh', 'eu1', 'nice', 'python', '/home/log/scripts/script2.py \"%s\"' % regex])
print log.strip()
Try this:
import subprocess
output = subprocess.check_output(['ssh', 'otherserver', 'python', '/home/log/scripts/script2.py', regex])
I have a Python script that needs to invoke another Python script in the same directory. I did this:
from subprocess import call
call('somescript.py')
I get the following error:
call('somescript.py')
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I have the script somescript.py in the same folder though. Am I missing something here?
If 'somescript.py' isn't something you could normally execute directly from the command line (I.e., $: somescript.py works), then you can't call it directly using call.
Remember that the way Popen works is that the first argument is the program that it executes, and the rest are the arguments passed to that program. In this case, the program is actually python, not your script. So the following will work as you expect:
subprocess.call(['python', 'somescript.py', somescript_arg1, somescript_val1,...]).
This correctly calls the Python interpreter and tells it to execute your script with the given arguments.
Note that this is different from the above suggestion:
subprocess.call(['python somescript.py'])
That will try to execute the program called python somscript.py, which clearly doesn't exist.
call('python somescript.py', shell=True)
Will also work, but using strings as input to call is not cross platform, is dangerous if you aren't the one building the string, and should generally be avoided if at all possible.
Windows? Unix?
Unix will need a shebang and exec attribute to work:
#!/usr/bin/env python
as the first line of script and:
chmod u+x script.py
at command-line or
call('python script.py'.split())
as mentioned previously.
Windows should work if you add the shell=True parameter to the "call" call.
Check out this.
from subprocess import call
with open('directory_of_logfile/logfile.txt', 'w') as f:
call(['python', 'directory_of_called_python_file/called_python_file.py'], stdout=f)
import subprocess
command = 'home/project/python_files/run_file.py {} {} {}'.format(
arg1, arg2, arg3) # if you want to pass any arguments
p = subprocess.Popen(
[command],
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True)
out, err = p.communicate()
subprocess.call expects the same arguments as subprocess.Popen - that is a list of strings (the argv in C) rather than a single string.
It's quite possible that your child process attempted to run "s" with the parameters "o", "m", "e", ...
If you're on Linux/Unix you could avoid call() altogether and not execute an entirely new instance of the Python executable and its environment.
import os
cpid = os.fork()
if not cpid:
import somescript
os._exit(0)
os.waitpid(cpid, 0)
For what it's worth.
What's wrong with
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
or better still wrap the functionality in a function, e.g. baz, then do this.
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
somescript.baz()
There seem to be a lot of scripts starting python processes or forking, is that a requirement?
First, check if somescript.py is executable and starts with something along the lines of #!/usr/bin/python.
If this is done, then you can use subprocess.call('./somescript.py').
Or as another answer points out, you could do subprocess.call(['python', 'somescript.py']).
def main(argv):
host = argv[0]
type = argv[1]
val = argv[2]
ping = subprocess.Popen(['python ftp.py %s %s %s'%(host,type,val)],stdout = subprocess.PIPE,stderr = subprocess.PIPE,shell=True)
out = ping.communicate()[0]
output = str(out)
print output
The subprocess call is a very literal-minded system call. it can be used for any generic process...hence does not know what to do with a Python script automatically.
Try
call ('python somescript.py')
If that doesn't work, you might want to try an absolute path, and/or check permissions on your Python script...the typical fun stuff.