This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 1 year ago.
I want to write a python script to execute to following line:
ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done
I tried:
def loop(jobname):
"""
Loop over gen_fcl output and store as tarbull
"""
loop = ["ls", "-1", "-d", "???", "|", "cut", "-c", "1-2", "|", "sort", "|", "uniq", "|", "while", "read", "NN;", "do", "echo", "$NN;", "tar", "-cjf", "fcllist_"+str(jobname)+".bz2", "${NN}?;", "done"]
subproc = subprocess.run(loop)
return subproc
But I am seeing this error:
ls: invalid option -- 'j'
Try 'ls --help' for more information.
Does anyone have any ideas?
in your code you have -cjf. each letter is an "option" for a command. this is causing the error. if you remove it it will run
also if you want it to run a line of code then run that line of code.
codeToRun = "ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done"
subproc = subprocess.run(codeToRun)
Related
I'm having an issue running a simple python script that reads a helm command from a .sh script and outputs it.
When I run the command directly in the terminal, it runs fine:
helm list | grep prod- | cut -f5
# OUTPUT: prod-L2.0.3.258
But when I run python test.py (see below for whole source code of test.py), I get an error as if the command I'm running is helm list -f5 and not helm list | grep prod- | cut -f5:
user#node1:$ python test.py
# OUTPUT:
# Opening file 'helm_chart_version.sh' for reading...
# Running command 'helm list | grep prod- | cut -f5'...
# Error: unknown shorthand flag: 'f' in -f5
The test.py script:
import subprocess
# Open file for reading
file = "helm_chart_version.sh"
print("Opening file '" + file + "' for reading...")
bashCommand = ""
with open (file) as fh:
next(fh)
bashCommand = next(fh)
print("Running command '" + bashCommand + "'...")
process = subprocess.Popen(bashCommand.split(), stdout=subprocess.PIPE)
output, error = process.communicate()
if error is None:
print output
else:
print error
Contents of helm_chart_version.sh:
cat helm_chart_version.sh
# OUTPUT:
## !/bin/bash
## helm list | grep prod- | cut -f5
Try to avoid running complex shell pipelines from higher-level languages. Given the command you show, you can run helm list as a subprocess, and then do the post-processing on it in Python.
process = subprocess.run(["helm", "list"], capture_output=True, text=True, check=True)
for line in process.stdout.splitlines():
if 'prod-' not in line:
continue
words = line.split()
print(words[4])
The actual Python script you show doesn't seem to be semantically different from just directly running the shell script. You can use the sh -x option or the shell set -x command to cause it to print out each line as it executes.
I am using below python script to filter data from /var/log/messages file of target machine by date & time .
But getting syntax error
I am using python version 2.7 & will not be able to upgrade python version
#!/usr/bin/python
import cgi, cgitb
import os
from subprocess import PIPE, Popen
def cmdline(command):
process = Popen(
args=command,
stdout=PIPE,
shell=True
)
return process.communicate()[0]
out4=cmdline('sshpass -p redhat ssh -o ConnectTimeout=6 -o NumberOfPasswordPrompts=2 -o StrictHostKeyChecking=no -tt ricky#192.168.0.50 "echo redhat | sudo -S zless /var/log/messages* | grep \'^Sep 9\' | awk \' \$3 > "09:30" && \$3 < "23:50" \' "')
print(out4)
Getting below output while executing this script
Connection to 192.168.0.50 closed.
awk: $3 > 09:30 && $3 < 23:50
awk: ^ syntax error
awk: $3 > 09:30 && $3 < 23:50
awk: ^ syntax error
[sudo] password for ricky:
Anyone please help me to correct it
You are falling into quote hell due to multiple levels of special char handling, by the multiple shells you're running one inside the other.
You can reduce your doom, by avoiding using shell=True so it is one less shell to escape for:
p = subprocess.run([
'sshpass', '-p', 'redhat'
'ssh',
'-o', 'ConnectTimeout=6',
'-o', 'NumberOfPasswordPrompts=2',
'-o', 'StrictHostKeyChecking=no',
'-tt', 'ricky#192.168.0.50'
"""sudo -S zless /var/log/messages* | grep '^Sep 9' | awk ' \$3 > "09:30" && \$3 < "23:50" '"""
], input='redhat\n', stdout=subprocess.PIPE)
out4 = p.stdout
I need to execute the same command on a local and remote server. So I'm using subprocess.Popen to execute, and local command work as expected, but when I execute on remote it gives me some error like command not found. I appreciate your support as I am new to this.
Local execution function
def topic_Offset_lz(self):
CMD = "/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s | grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.topic,self.envr,self.partition)
t_out_lz, t_error_lz = subprocess.Popen(CMD, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_out_lz
Remote server execution
def topic_offset_sl(self):
CMD = "/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s | grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.topic, self.envr, self.partition)
t_out_sl, t_error_sl = subprocess.Popen(["ssh", "-q", CMD], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_error_sl
Error I'm getting for the remote execution
Landing Zone Offset: 0
SoftLayer Zone Offset: /bin/sh: ^# |sed 1: command not found
/bin/sh: d: command not found
I came up with below solution, might be there will easy way rather than this.
def topic_offset_sl(self):
CMD_SL1 = "ssh -q %s '/dsapps/admin/edp/scripts/edp-admin.sh kafka-topic offset %s -e %s'" % (KEY_SERVER,self.topic, self.envr)
CMD_SL2 = "| grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#' | awk -F\: '{print $3}'|sed '%sq;d'" % (self.partition)
t_out_sl, t_error_sl = subprocess.Popen(CMD_SL1 + CMD_SL2 , stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_out_sl
The ssh command passes its argument vector as a single command line string, not an array. To do this, it simply concatenates the arguments, without performing shell quoting:
$ ssh target "python -c 'import sys;print(sys.argv)'" 1 2 3
['-c', '1', '2', '3']
$ ssh target "python -c 'import sys;print(sys.argv)'" "1 2 3"
['-c', '1', '2', '3']
If there was proper shell quoting, the distinction between 1 2 3 and "1 2 3" would have been preserved, and the first argument would not need double-quoting.
Anyway, in your case, the following might work:
def topic_offset_sl(self):
CMD = "ssh -q " + pipes.quote("/dsapps/admin/edp/scripts/edp-admin.sh"
+ " kafka-topic offset %s -e %s" % (self.topic, self.envr)) \
+ "grep -v Getting |grep -v Verifying | egrep -v '^[[:space:]]*$|^#'"
+ " | awk -F\: '{print $3}'|sed '%sq;d'" % self.partition
t_out_sl, t_error_sl = subprocess.Popen(CMD], stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True).communicate()
return t_error_sl
This assumes you only want to run the /dsapps/admin/edp/scripts/edp-admin.sh script remotely and not the rest.
Note that the way you use string splicing to construct command lines likely introduces shell command injection vulnerabilities (both locally and on the remote server).
This is a bash command that I run in python and get the expected result:
count = subprocess.Popen("ps -ef | grep app | wc -l", stdout=subprocess.PIPE, shell=True)
but when I'd like to pass an argument (count in this case) cannot figure out how to do it.
I tried:
pid = subprocess.call("ps -ef | grep app | awk -v n=' + str(count), 'NR==n | awk \'{print $2}\'", shell=True)
and
args = shlex.split('ps -ef | grep app | awk -v n=' + str(count), 'NR==n | awk \'{print $2}\'')
pid = subprocess.Popen(args, stdout=subprocess.PIPE, shell=True)
among other attempts, from various posts here, but still cannot make it.
You're mixing opening and closing quotations and you pass a colon by mistake on your other attempts among other things.
Try this for a fix:
pid = subprocess.call("ps -ef | grep app | awk -v n=" + str(count) + " NR==n | awk '{print $2}'", shell=True)
You opened the command parameter with " and there for you need to close it before you do + str() with a " and not a '. Further more i swapped the , 'NR= with + "NR= since you want to append more to your command and not pass a argument to subprocess.call().
As pointed out in the comments, there's no point in splitting the command with shlex since piping commands isn't implemented in subprocess, I would however like to point out that using shell=True is usually not recommended because for instance one of the examples given here.
An other vay is using format:
pid = subprocess.call("ps -ef | grep app | awk -v n={} NR==n | awk '{{print $2}}'".format(str(count)), shell=True)
Your Awk pipeline could be simplified a great deal - if the goal is to print the last match, ps -ef | awk '/app/ { p=$2 } END { print p }' does that. But many times, running Awk from Python is just silly, and performing the filtering in Python is convenient and easy, as well as obviously more efficient (you save not only the Awk process, but also the pesky shell=True).
for p in subprocess.check_output(['ps', '-ef']).split('\n'):
if 'app' in p:
pid = p.split()[1]
I am trying to use os.system to invoke an external (piped) shell command:
srcFile = os.path.abspath(sys.argv[1])
srcFileIdCmd = "echo -n '%s' | cksum | cut -d' ' -f1" % srcFile
print "ID command: %s" % srcFileIdCmd
srcFileID = os.system(srcFileIdCmd)
print "File ID: %s" % srcFileID
outputs
ID command: echo -n '/my/path/filename' | cksum | cut -d' ' -f1
File ID: 0
But when I run
echo -n '/my/path/filename' | cksum | cut -d' ' -f1
manually on a command line, I get 2379496500, not 0.
What do I need to change to get the correct value out of the shell command?
Use
sp = subprocess.Popen(["program", "arg"], stdout=subprocess.PIPE)
instead, and then read from the file sp.stdout. The pogram in question can be a shell, and you can pass complex shell commands to it as parameters (["/usr/bin/bash", "-c", "my-complex-command"]).