We Keep Coding

How to fix/get awk command called by subprocess working in script (works in python shell)?

I'm trying to get the following command to be called from a python script and to get the output of the command: awk -F':' '$1 == "VideoEdge" {print $2, $3, $8}' /etc/shadow
I've got the function working using subprocess.check_output and .Popen in a python shell but when called from a script it doesn't work and causes an exception of which has no apparent output or message.
How can I get this command working from a script?
I've tried using check_output, Popen and shlex to help with possible issues I thought were causing the issue. Code works fine in a shell.
temp = "User"
cmd = "awk -F':' '$1 == \"" + temp + "\" {print $2, $3, $8}' /etc/shadow"
cmdOutput = subprocess.check_output(shlex.split(cmd))
print cmdOutput
temp = "User"
cmd = "awk -F':' '$1 == \"" + temp + "\" {print $2, $3, $8}' /etc/shadow"
cmdOutput = subprocess.Popen(cmd, stdin=PIPE, stdout=PIPE, stderr=PIPE)
print cmdOutput.communicate()[0]

I'd just do. [Just for user consideration, (it will be cryptic, in the comments)]
user = "someuser"
with open('/etc/shadow') as f:
for line in f:
if line.startswith(user):
data = line.split(':')
break
print(data)

Make the same shlex.split(cmd) and it will work:
cmdOutput = subprocess.Popen(shlex.split(cmd), stdin=PIPE, stdout=PIPE, stderr=PIPE)

I had a permission issue with the file i was performing the command. bangs head against desk

How to execute python subprocess.Popen with many Arguments?

I need to execute the same command on a local and remote server. So I'm using subprocess.Popen to execute, and local command work as expected, but when I execute on remote it gives me some error like command not found. I appreciate your support as I am new to this.
Local execution function
def topic_Offset_lz(self):
CMD = "/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s | grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.topic,self.envr,self.partition)
t_out_lz, t_error_lz = subprocess.Popen(CMD, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_out_lz
Remote server execution
def topic_offset_sl(self):
CMD = "/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s | grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.topic, self.envr, self.partition)
t_out_sl, t_error_sl = subprocess.Popen(["ssh", "-q", CMD], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_error_sl
Error I'm getting for the remote execution
Landing Zone Offset: 0
SoftLayer Zone Offset: /bin/sh: ^# |sed 1: command not found
/bin/sh: d: command not found

I came up with below solution, might be there will easy way rather than this.
def topic_offset_sl(self):
CMD_SL1 = "ssh -q %s '/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s'" % (KEY_SERVER,self.topic, self.envr)
CMD_SL2 = "| grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.partition)
t_out_sl, t_error_sl = subprocess.Popen(CMD_SL1 + CMD_SL2 , stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_out_sl

The ssh command passes its argument vector as a single command line string, not an array. To do this, it simply concatenates the arguments, without performing shell quoting:
$ ssh target "python -c 'import sys;print(sys.argv)'" 1 2 3
['-c', '1', '2', '3']
$ ssh target "python -c 'import sys;print(sys.argv)'" "1 2 3"
['-c', '1', '2', '3']
If there was proper shell quoting, the distinction between 1 2 3 and "1 2 3" would have been preserved, and the first argument would not need double-quoting.
Anyway, in your case, the following might work:
def topic_offset_sl(self):
CMD = "ssh -q " + pipes.quote("/dsapps/admin/edp/scripts/edp-admin.sh"
+ " kafka-topic offset %s -e %s" % (self.topic, self.envr)) \
+ "grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#'"
+ " | awk -F\: '{print $3}'|sed '%sq;d'" % self.partition
t_out_sl, t_error_sl = subprocess.Popen(CMD], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_error_sl
This assumes you only want to run the /dsapps/admin/edp/scripts/edp-admin.sh script remotely and not the rest.
Note that the way you use string splicing to construct command lines likely introduces shell command injection vulnerabilities (both locally and on the remote server).

Handle result of os.system

I'm using python to script a functional script and I can't handler the result of this command line:
os.system("ps aux -u %s | grep %s | grep -v 'grep' | awk '{print $2}'" % (username, process_name)
It shows me pids but I can't use it as List.
If I test:
pids = os.system("ps aux -u %s | grep %s | grep -v 'grep' | awk '{print $2}'" % (username, process_name)
print type(pids)
#Results
29719
30205
31037
31612
<type 'int'>
Why is pids an int? How can I handle this result as List?
Stranger part:
print type(os.system("ps aux -u %s | grep %s | grep -v 'grep' | awk '{print $2}'" % (username, process_name))
There is nothing. Not any type written on my console..

os.system does not capture the output of the command it runs. To do so you need to use subprocess.
from subprocess import check_output
out = check_output("your command goes here", shell=true)
The above will work in Python 2.7. For older Pythons, use:
import subprocess
p = subprocess.Popen("your command goes here", stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()

os module documentation
os.system(command)
Execute the command (a string) in a subshell. This is implemented by calling the Standard C function system(), and has the same limitations. Changes to sys.stdin, etc. are not reflected in the environment of the executed command.
On Unix, the return value is the exit status of the process encoded in the format specified for wait(). Note that POSIX does not specify the meaning of the return value of the C system() function, so the return value of the Python function is system-dependent.
If you want access to the output of the command, use the subprocess module instead, e.g. check_output:
subprocess.check_output(args, *, stdin=None, stderr=None, shell=False, universal_newlines=False)
Run command with arguments and return its output as a byte string.

Setting Variable to Gateway IP

I have a code so far that filters out everything except the gateway IP (route -n | awk '{if($4=="UG")print $2}'), but I'm trying to figure out how to pipe this to a variable in Python. Here's what I got:
import shlex;
from subprocess import Popen, PIPE;
cmd = "route -n | grep 'UG[ \t]' | awk '{print $2}'";
gateway = Popen(shlex.split(cmd), stdout=PIPE);
gateway.communicate();
exit_code = gateway.wait();
Any ideas?
NOTE: I'm new at this.

For better or worse, your cmd uses a shell pipeline. To use shell features in subprocess, one must set shell=True:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n | grep 'UG[ \t]' | awk '{print $2}'"
gateway = Popen(cmd, shell=True, stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
Alternatively, one could keep shell=False, eliminate the pipeline, and do all the string processing in python:
from subprocess import Popen, PIPE
cmd = "/sbin/route -n"
gateway = Popen(cmd.split(), stdout=PIPE)
stdout, stderr = gateway.communicate()
exit_code = gateway.wait()
gw = [line.split()[1] for line in stdout.decode().split('\n') if 'UG' in line][0]
Because of the vagaries of shell processing, and unless there is a specific need, it is probably best to avoid shell=True.

How to split up the command here for using subprocess.Popen()

ip = subprocess.Popen(["/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') | grep \"inet addr\" | awk -F: '{print $2}' | awk \'{print $1}\'"], stdout=subprocess.PIPE)
I am not sure where to put the commas to separate them to use this command using subprocess.Popen. Does anyone know?

You are using shell features (the pipe) so instead of splitting the command, you should pass it as a single string (not a list) with shell=True
ip = subprocess.Popen("/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') | grep \"inet addr\" | awk -F: '{print $2}' | awk \'{print $1}\'",
shell=True,
stdout=subprocess.PIPE)

Here's what I would recommend.
Create a file with this contents - call it 'route-info' and make it executable:
#!/bin/sh
/sbin/ifconfig $(/sbin/route | awk '/default/ {print $8}') |
grep "inet addr" |
awk -F: '{print $2}' |
awk '{print $1}'
In your python program, use:
ip = subprocess.Popen(["/path/to/route-info"], stdout=subprocess.PIPE)
Then you don't have to worry about quoting characters and you can independently test the route-info script to make sure it is working correctly.
The script route-info doesn't take any command line arguments, but if it did this is how you would pass them:
ip = subprocess.Popen(["/path/to/route-info", arg1, arg2, ...], stdout=subprocess.PIPE)

Quoting the official documentation of subprocess.Popen here
It may not be obvious how to break a shell command into a sequence of
arguments, especially in complex cases. shlex.split() can illustrate
how to determine the correct tokenization for args:
import shlex, subprocess
command_line = input()
args = shlex.split(command_line)
print(args)
p = subprocess.Popen(args) # Success!
shlex is included in standard library so you need not to install it.
Writing it in a single line like str.split() should look like:
import shlex
import subprocess
command = "ls -l"
proc = subprocess.Popen(shlex.split(command) , stdout = subprocess.PIPE , stderr = subprocess.PIPE)
output , errors = proc.communicate()
print(output , errors)