We Keep Coding

subprocess for long string of commands python error message [duplicate]

This question already has answers here:
How to use `subprocess` command with pipes
(7 answers)
Closed 1 year ago.
I want to write a python script to execute to following line:
ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done
I tried:
def loop(jobname):
"""
Loop over gen_fcl output and store as tarbull
"""
loop = ["ls", "-1", "-d", "???", "|", "cut", "-c", "1-2", "|", "sort", "|", "uniq", "|", "while", "read", "NN;", "do", "echo", "$NN;", "tar", "-cjf", "fcllist_"+str(jobname)+".bz2", "${NN}?;", "done"]
subproc = subprocess.run(loop)
return subproc
But I am seeing this error:
ls: invalid option -- 'j'
Try 'ls --help' for more information.
Does anyone have any ideas?

in your code you have -cjf. each letter is an "option" for a command. this is causing the error. if you remove it it will run
also if you want it to run a line of code then run that line of code.
codeToRun = "ls -1 -d ??? | cut -c 1-2 | sort | uniq | while read NN; do echo $NN; tar -cjf fcllistRPC_${NN}.bz2 ${NN}?; done"
subproc = subprocess.run(codeToRun)

The term 'Select-String' is not recognized

I am trying to get only the PID by running this python script.
My current error:
Select string is not recognized as an internal or external command
To deal with this error, I thought that I needed to escape | by adding ^ --> But apparently it doesn't work
I added in some \ to escape ", hopefully it's correct?
cmd = "netstat -ano | findstr " + str(o)
print (cmd)
cmd += " | Select-String \"TCP\s+(.+)\:(.+)\s+(.+)\:(\d+)\s+(\w+)\s+(\d+)\" | ForEach-Object { Write-Output $_.matches[0].Groups[6].value }"
print (cmd)
pid = run_command(cmd)
The run_command method does this:
def run_command(cmd_array,os_name='posix'):
p = subprocess.Popen(cmd_array,shell=True,cwd=os.getcwd())
output,err = p.communicate()
print('output=%s'%output)
print('err=%s'%err)
return output
Expected Result
When I run solely the command in command prompt, it gives me the PID --> which is 7556 in this case.
Not too sure why it is not working for the script but working in command prompt by itself.

This question is specific to windows OS
Answer to my own question
With help from the comments, i did not use my run_command method since it was using shell = True.
shell = True refers to cmd.exe on Windows, not powershell.
The commands i have written are powershell commands.
Directly use subprocess.call to run powershell commands
Python scripts
cmd = "netstat -ano | findstr 8080"
cmd += " | Select-String \"TCP\s+(.+)\:(.+)\s+(.+)\:(\d+)\s+(\w+)\s+(\d+)\" | ForEach-Object { Write-Output $_.matches[0].Groups[6].value }"
subprocess.call(["powershell.exe", cmd])
#this does the job but the code will print extra zeros along with PID. It was not what i was looking for.
Result:
6492 (Prints out the PID Along with some additional zeros)
What worked for me - For those who are trying to get only PID and kill port using PID in python script
cmd = "for /f \"tokens=5\" %a in ('netstat -aon ^| find \":8080"
cmd += "\" ^| find \"LISTENING\"\') do taskkill /f /pid %a"
#I added in some \ to escape the "
run_command(cmd)
Result:
SUCCESS: The process with PID 2072 has been terminated

How can I specify quotes(" ") in my string text in Python?

The command I want to run from my python script is:
tool-manager-cli -a "somexml.xml" -c "another.xml"
My python code is:
command1 = """ tool-manager-cli -a "somexml.xml" -c "another.xml" """
subprocess.Popen(command1)
However when I run my py script, I get SyntaxError: invalid syntax as my error for this line.
How can I specify quotes(" ") in my string text without closing it or invalid syntax?

Try using single quotes to differentiate like so
command1 = """ tool-manager-cli -a 'somexml.xml' -c 'another.xml' """

You can use format method to structure your command:
command1= "{} {} {} {} {}".format("tool-managel-cli","-a", "somexml.xml","-c","another.xml")
subprocess.Popen(command1)
You can also concat by + , but its tedious ofcourse
command1 ="tool-manager-cli"+" -a"+ " somexml.xml" + " -c "+" another.xml"

Awk in Python's subprocess giving Invalid Expressions "'" error

I am trying to read the filename and filestamp for the most recent files of each of the two naming schemes as seen in the code. I have the following code, roughly:
#!/usr/bin/env python
import string, subprocess, sys, os
mypath = "/path/to/file"
my_cmd = (["ls -lt --full-time " + mypath + "*DAI*.txt",
"ls -lt --full-time " + mypath + "*CA*.txt"]
)
getmostrecent_cmd = "head -n 1"
getcols_cmd = "awk '{ print $6, $7, $9 }'"
for cmd in my_cmd:
p1 = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
p2 = subprocess.Popen(getmostrecent_cmd.split(), stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(getcols_cmd.split(), stdin=p2.stdout, stdout=subprocess.PIPE)
output = p3.communicate()[0]
print output
which give me the following error(s):
ls: cannot access /path/to/file/*DAI*.txt: No such file or directory
awk: '{
awk: ^ invalid char ''' in expression
ls: cannot access /path/to/file/*CA*.txt: No such file or directory
awk: '{
awk: ^ invalid char ''' in expression
But:
I can use "ls -lt --full-time /path/to/file/*DAI*.txt" and get a result in the terminal. Why is it causing an issue with the same path?
The awk command, when put in to subprocess directly, works fine; E.g. subprocess.Popen(["awk", ....], stdin=...., stdout=....) worked okay. But now I am getting an issue with the single quote. I tried triple quoting the string and escaping the single-quote.

I can use "ls -lt --full-time /path/to/file/DAI.txt" and get a
result in the terminal. Why is it causing an issue with the same path?
Glob expansion is performed by the shell. By default, shell is not involved in starting a new subprocess via Popen(). To this end you must pass the shell=True argument to it:
p1 = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, shell=True)
# ^^^^^^^^^^
The awk command, when put in to subprocess directly, works fine; E.g.
subprocess.Popen(["awk", ....], stdin=...., stdout=....) worked okay. But now I am getting an issue with the single quote. I tried
triple quoting the string and escaping the single-quote.
On the shell command line the single quotes in awk '{ print $6, $7, $9 }' are needed to make the string { print $6, $7, $9 } treated as a single argument (as well as to prevent the variable expansion). The single quotes are removed by the shell, and awk only sees the string { print $6, $7, $9 }. Since Popen() by default doesn't involve shell when executing the subprocess command and passes the arguments to the command verbatim, you don't need the single quotes:
subprocess.Popen(["awk", "{ print $6, $7, $9 }"], stdin=...., stdout=....)

Python subprocess for queries with quotes

I need to fire this query which runs perfectly on the terminal:
sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' <PATH_TO_FILE>
This adds a line below where I find "default]" string.
Using the python code:
query = r""" sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' %s """ % (PATH_T)_FILE)
proc = subprocess.Popen(query.split(), stdout=subprocess.PIPE, stderr=subprocess.PIPE)
output, err = proc.communicate()
However, the command fails in python with error:
Error executing the query sed -i '' '/default\]/a\'$'\n'' Hello world'$'\n' /Users/hshah/tmpFile . output = , error = sed: 1: "'/default\]/a\'$'\n''": invalid command code '
What could be the problem here?

You split on every whitespace. This causes query.split() to be
['sed',
'-i',
"''",
"'/default\\]/a\\'$'\\n''",
'Hello',
"world'$'\\n'",
'/tmp/foo']
which is not what you want. Build up the parameters for subprocess.Popen by hand, not by splitting a string.