We Keep Coding

Python Error using subprocess without shell=false and variable from an array

I have the code snippet that works with shell=True which isn't secure and when i attempt to remove shell=True and include shell=False the program errors out
The code is below:
cmd = "git clone https://github.com/{} &"
#define a worker function
def worker():
while True:
item = q.get()
subprocess.Popen(cmd.format(item))
q.task_done()
I get the error below:
File "rapid.py", line 56, in worker
subprocess.Popen(cmd.format(item))
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 775, in __init__
restore_signals, start_new_session)
File "/Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/subprocess.py", line 1522, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'git clone https://github.com/laramies/theHarvester.git &': 'git clone https://github.com/laramies/theHarvester.git &'
if i add shell = True to the subprocess line it runs just fine (see below) but then code factor flags it as insecure code. Any way to do this without shell = true?
cmd = "git clone https://github.com/{} &"
#define a worker function
def worker():
while True:
item = q.get()
subprocess.Popen(cmd.format(item), shell = True)
q.task_done()

The command is being passed to subprocess.Popen as a string. This works when using shell=True because the shell can accept a command as a single string. But when shell=False, Popen expects the command as a list of arguments, the first one being the full path of the program to run. (This is assuming you're on a POSIX machine, not Windows.)
Essentially, the code says "Run a program called git clone https://github.com/laramies/theHarvester.git with no arguments" instead of "Run git with arguments clone and https://github.com/laramies/theHarvester.git".
The & should also be removed because that is a shell feature and with shell=False it would be passed to git as an argument it won't understand. You don't need it though, since the process will run in the background anyway.
Something like this should work:
subprocess.Popen(["/usr/bin/git", "clone", "https://github.com/{}".format(item)])
More info here: https://docs.python.org/3.7/library/subprocess.html#popen-constructor

OSError: [Errno 8] Exec format error

I am having hard time parsing the arguments to subprocess.Popen. I am trying to execute a script on my Unix server. The script syntax when running on shell prompt is as follows:
/usr/local/bin/script hostname = <hostname> -p LONGLIST. No matter how I try, the script is not running inside subprocess.Popen
The space before and after "=" is mandatory.
import subprocess
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname = ', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
The above does not work.
And when I use shell=False, I get OSError: [Errno 8] Exec format error

OSError: [Errno 8] Exec format error can happen if there is no shebang line at the top of the shell script and you are trying to execute the script directly. Here's an example that reproduces the issue:
>>> with open('a','w') as f: f.write('exit 0') # create the script
...
>>> import os
>>> os.chmod('a', 0b111101101) # rwxr-xr-x make it executable
>>> os.execl('./a', './a') # execute it
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.7/os.py", line 312, in execl
execv(file, args)
OSError: [Errno 8] Exec format error
To fix it, just add the shebang e.g., if it is a shell script; prepend #!/bin/sh at the top of your script:
>>> with open('a','w') as f: f.write('#!/bin/sh\nexit 0')
...
>>> os.execl('./a', './a')
It executes exit 0 without any errors.
On POSIX systems, shell parses the command line i.e., your script won't see spaces around = e.g., if script is:
#!/usr/bin/env python
import sys
print(sys.argv)
then running it in the shell:
$ /usr/local/bin/script hostname = '<hostname>' -p LONGLIST
produces:
['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
Note: no spaces around '='. I've added quotes around <hostname> to escape the redirection metacharacters <>.
To emulate the shell command in Python, run:
from subprocess import check_call
cmd = ['/usr/local/bin/script', 'hostname', '=', '<hostname>', '-p', 'LONGLIST']
check_call(cmd)
Note: no shell=True. And you don't need to escape <> because no shell is run.
"Exec format error" might indicate that your script has invalid format, run:
$ file /usr/local/bin/script
to find out what it is. Compare the architecture with the output of:
$ uname -m

I will hijack this thread to point out that this error may also happen when target of Popen is not executable. Learnt it hard way when by accident I have had override a perfectly executable binary file with zip file.

Have you tried this?
Out = subprocess.Popen('/usr/local/bin/script hostname = actual_server_name -p LONGLIST'.split(), shell=False,stdout=subprocess.PIPE,stderr=subprocess.PIPE)
Edited per the apt comment from #J.F.Sebastian

It wouldn't be wrong to mention that Pexpect does throw a similar error
#python -c "import pexpect; p=pexpect.spawn('/usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/usr/lib/python2.7/site-packages/pexpect.py", line 430, in __init__
self._spawn (command, args)
File "/usr/lib/python2.7/site-packages/pexpect.py", line 560, in _spawn
os.execv(self.command, self.args)
OSError: [Errno 8] Exec format error
Over here, the openssl_1.1.0f file at the specified path has exec command specified in it and is running the actual openssl binary when called.
Usually, I wouldn't mention this unless I have the root cause, but this problem was not there earlier. Unable to find the similar problem, the closest explanation to make it work is the same as the one provided by #jfs above.
what worked for me is both
adding /bin/bash at the beginning of the command or file you are
facing the problem with, or
adding shebang #!/bin/sh as the first line.
for ex.
#python -c "import pexpect; p=pexpect.spawn('/bin/bash /usr/local/ssl/bin/openssl_1.1.0f version'); p.interact()"
OpenSSL 1.1.0f 25 May 2017

If you think the space before and after "=" is mandatory, try it as separate item in the list.
Out = subprocess.Popen(['/usr/local/bin/script', 'hostname', '=', 'actual server name', '-p', 'LONGLIST'],shell=True,stdout=subprocess.PIPE,stderr=subprocess.PIPE)

subprocess module in python

I have to connect to a sybase database and run a simple select query using python script
On my server isql command can run only from sybase bin directory, so i have to cd to that directory before firing the query.
---------------------------Edited-----------------------------
Uptill now i'm able to do this:-
#!/usr/bin/python
import subprocess
path = "path/to/sybase/bin"
os.chdir(path)
arguments = ['./isql',"-S server_name", "-U user", "-P password", "-D database","""<<EOF
SELECT * FROM sometable
go
EOF"""]
ps = subprocess.Popen(arguments)
out = ps.communicate()
print out
The errors are just out of my understanding capability :(
Traceback (most recent call last):
File "./test_db.py", line 8, in ?
ps = subprocess.Popen(arguments)
File "/usr/lib64/python2.4/subprocess.py", line 542, in __init__
errread, errwrite)
File "/usr/lib64/python2.4/subprocess.py", line 975, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I'm able to do this outside my python script using isql command on my unix terminal
how can i use isql in python subprocess module?

There is a particular Popen argument for that: cwd, as mentioned here. Provide your command using an array and then the cwd parameter with where the command must be executed:
subprocess.Popen(['ls', '-l'], cwd="/path/to/folder")

Popen only takes one args argument, for the command to run. You could try calling a shell with both the cd and isql commands as arguments, but changing the working directory from python is probably simpler
For the former approach:
subprocess.Popen('/bin/sh -c "cd /path/to/... && isql -arg1..'...)
for the latter:
os.chdir('/path/to...')
subprocess.Popen('isql -arg1..'...)

Try:
import os
import subprocess
os.chdir('/path/to/sybase/bin')
if os.path.exists('isql') or os.path.exists(os.path.join('/path/to/sybase/bin', 'isql')):
ps = subprocess.Popen('isql -S %s -U %s -P %s -D %s <<EOF SELECT * FROM sometable EOF' % (server,user,passwd,database), stdout=subprocess.PIPE, shell=True)
out, err = ps.communicate()
else:
print "isql does not exists in this folder"
I am not super experienced with subprocess but this is how I generally use it on the odd occasion. Hopefully someone else can give a better answer/explanation.
Edit: removed the square brackets to remove confusion.

i know it's been long but just wanted to close this question
from subprocess import Popen, PIPE
from textwrap import dedent
isql = Popen(['./isql', '-I', '/app/sybase/...',
'-S', mdbserver,
'-U', muserid,
'-P', password, ...,
'-w', '99999'], stdin=PIPE, stdout=PIPE, cwd=sybase_path)
output = isql.communicate(dedent("""\
SET NOCOUNT ON
{}
go
""".format(User_Query)))[0]

call IPtables command within python script

I am trying to call the following command in my python script. I am trying to insert the rule into IP tables. I am using sub process to call it and inserting variables into it where needed, but am getting a large error. Any suggestions?
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
Error:
Traceback (most recent call last):
File "./port_forward.py", line 42, in <module>
iptables = subprocess.call('iptables -I FORWARD -i eth0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress)
File "/usr/lib/python2.7/subprocess.py", line 493, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.7/subprocess.py", line 679, in __init__
errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1259, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory

Your problem is very common for Python beginners. Instead of formatting the string command,
you are trying to build a complex string by concatenating many strings and variables. Instead, use a
a string format, it will allow you to test your command and make your code more readable and flexible.
Your original string lacks spaces between the options and the arguments, e.g. --to-destination1.2.3.4.
Hence, you should format your string (This works also for python 2.7):
opts = {'iptables': '/sbin/iptables', 'protocol': 'tcp', 'port': 80, 'ipAddress': '0.0.0.0'}
ipcmd = '{iptables} -I FORWARD -eth 0 -m {protocol} -t {protocol} \
--dport {port} -j DNAT --to-destination {ipAddress}'.format(**opts)
if DEBUG:
print ipcmd
iptables = subprocess.call(ipcmd)
This is much easier to modify later, and also, when you do more Python programming, you will see that it is more readable.
Also, to call IPTables, you should be root, as stated in the comments:
In the beginning of your script add:
import sys
import os
if not os.getuid() == 0:
print "You must be root to change IPTables."
sys.exit(2)
update after seeing your error trace:
You are trying to call a command iptables but it is not in your path.
You should call the full path of iptables , e.g. /sbin/iptables

I wrote a simple firewall the same way and realized, "why not just write it in bash?". Anyway I discovered the python-iptables library and rewrote my code using that. I recommend checking it out. I think it will give you a more robust and structured way of writing iptables rules.

Your command is full of syntax errors due to missing spaces, as shown below:
iptables = subprocess.call(
'iptables -I FORWARD -eth 0 -m '
+ protocol
+ ' -t'+protocol
^---here
+ '--dport '
^-- here
+ port
+ '-j DNAT --to-destination'
^--here
+ ipAddress)
^---here
As generated, your iptables line will look like
-ttcp--dport 80-j DNAT --to-destination1.2.3.4
-ttcp--dport is parsed as a SINGLE argument, ditto for 80-j and --to-destination1.2.3.4

Just pass the argument shell=True along with the command.
iptables = subprocess.call('iptables -I FORWARD -eth 0 -m '+protocol+' -t'+protocol+'--dport '+port+'-j DNAT --to-destination'+ipAddress, shell=True)

Using a Python subprocess call to invoke a Python script

I have a Python script that needs to invoke another Python script in the same directory. I did this:
from subprocess import call
call('somescript.py')
I get the following error:
call('somescript.py')
File "/usr/lib/python2.6/subprocess.py", line 480, in call
return Popen(*popenargs, **kwargs).wait()
File "/usr/lib/python2.6/subprocess.py", line 633, in __init__
errread, errwrite)
File "/usr/lib/python2.6/subprocess.py", line 1139, in _execute_child
raise child_exception
OSError: [Errno 2] No such file or directory
I have the script somescript.py in the same folder though. Am I missing something here?

If 'somescript.py' isn't something you could normally execute directly from the command line (I.e., $: somescript.py works), then you can't call it directly using call.
Remember that the way Popen works is that the first argument is the program that it executes, and the rest are the arguments passed to that program. In this case, the program is actually python, not your script. So the following will work as you expect:
subprocess.call(['python', 'somescript.py', somescript_arg1, somescript_val1,...]).
This correctly calls the Python interpreter and tells it to execute your script with the given arguments.
Note that this is different from the above suggestion:
subprocess.call(['python somescript.py'])
That will try to execute the program called python somscript.py, which clearly doesn't exist.
call('python somescript.py', shell=True)
Will also work, but using strings as input to call is not cross platform, is dangerous if you aren't the one building the string, and should generally be avoided if at all possible.

Windows? Unix?
Unix will need a shebang and exec attribute to work:
#!/usr/bin/env python
as the first line of script and:
chmod u+x script.py
at command-line or
call('python script.py'.split())
as mentioned previously.
Windows should work if you add the shell=True parameter to the "call" call.

Check out this.
from subprocess import call
with open('directory_of_logfile/logfile.txt', 'w') as f:
call(['python', 'directory_of_called_python_file/called_python_file.py'], stdout=f)

import subprocess
command = 'home/project/python_files/run_file.py {} {} {}'.format(
arg1, arg2, arg3) # if you want to pass any arguments
p = subprocess.Popen(
[command],
shell=True,
stdin=None,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE,
close_fds=True)
out, err = p.communicate()

subprocess.call expects the same arguments as subprocess.Popen - that is a list of strings (the argv in C) rather than a single string.
It's quite possible that your child process attempted to run "s" with the parameters "o", "m", "e", ...

If you're on Linux/Unix you could avoid call() altogether and not execute an entirely new instance of the Python executable and its environment.
import os
cpid = os.fork()
if not cpid:
import somescript
os._exit(0)
os.waitpid(cpid, 0)
For what it's worth.

What's wrong with
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
or better still wrap the functionality in a function, e.g. baz, then do this.
import sys
from os.path import dirname, abspath
local_dir = abspath(dirname(__file__))
sys.path.append(local_dir)
import somescript
somescript.baz()
There seem to be a lot of scripts starting python processes or forking, is that a requirement?

First, check if somescript.py is executable and starts with something along the lines of #!/usr/bin/python.
If this is done, then you can use subprocess.call('./somescript.py').
Or as another answer points out, you could do subprocess.call(['python', 'somescript.py']).

def main(argv):
host = argv[0]
type = argv[1]
val = argv[2]
ping = subprocess.Popen(['python ftp.py %s %s %s'%(host,type,val)],stdout = subprocess.PIPE,stderr = subprocess.PIPE,shell=True)
out = ping.communicate()[0]
output = str(out)
print output

The subprocess call is a very literal-minded system call. it can be used for any generic process...hence does not know what to do with a Python script automatically.
Try
call ('python somescript.py')
If that doesn't work, you might want to try an absolute path, and/or check permissions on your Python script...the typical fun stuff.