I am trying to read the filename and filestamp for the most recent files of each of the two naming schemes as seen in the code. I have the following code, roughly:
#!/usr/bin/env python
import string, subprocess, sys, os
mypath = "/path/to/file"
my_cmd = (["ls -lt --full-time " + mypath + "*DAI*.txt",
"ls -lt --full-time " + mypath + "*CA*.txt"]
)
getmostrecent_cmd = "head -n 1"
getcols_cmd = "awk '{ print $6, $7, $9 }'"
for cmd in my_cmd:
p1 = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE)
p2 = subprocess.Popen(getmostrecent_cmd.split(), stdin=p1.stdout, stdout=subprocess.PIPE)
p3 = subprocess.Popen(getcols_cmd.split(), stdin=p2.stdout, stdout=subprocess.PIPE)
output = p3.communicate()[0]
print output
which give me the following error(s):
ls: cannot access /path/to/file/*DAI*.txt: No such file or directory
awk: '{
awk: ^ invalid char ''' in expression
ls: cannot access /path/to/file/*CA*.txt: No such file or directory
awk: '{
awk: ^ invalid char ''' in expression
But:
I can use "ls -lt --full-time /path/to/file/*DAI*.txt" and get a result in the terminal. Why is it causing an issue with the same path?
The awk command, when put in to subprocess directly, works fine; E.g. subprocess.Popen(["awk", ....], stdin=...., stdout=....) worked okay. But now I am getting an issue with the single quote. I tried triple quoting the string and escaping the single-quote.
I can use "ls -lt --full-time /path/to/file/DAI.txt" and get a
result in the terminal. Why is it causing an issue with the same path?
Glob expansion is performed by the shell. By default, shell is not involved in starting a new subprocess via Popen(). To this end you must pass the shell=True argument to it:
p1 = subprocess.Popen(cmd.split(), stdout=subprocess.PIPE, shell=True)
# ^^^^^^^^^^
The awk command, when put in to subprocess directly, works fine; E.g.
subprocess.Popen(["awk", ....], stdin=...., stdout=....) worked okay. But now I am getting an issue with the single quote. I tried
triple quoting the string and escaping the single-quote.
On the shell command line the single quotes in awk '{ print $6, $7, $9 }' are needed to make the string { print $6, $7, $9 } treated as a single argument (as well as to prevent the variable expansion). The single quotes are removed by the shell, and awk only sees the string { print $6, $7, $9 }. Since Popen() by default doesn't involve shell when executing the subprocess command and passes the arguments to the command verbatim, you don't need the single quotes:
subprocess.Popen(["awk", "{ print $6, $7, $9 }"], stdin=...., stdout=....)
Related
I need to run a couple of complex bash commands from Python. One of them is:
top -n1 -b | awk -vtime="$(date +%Y-%m-%d-%T)" 'NR==7{printf("Time\t%s\t%s\t%s\t%s\t%s\t%s\t%s\n",$1,$5,$6,$9,$10,$11,$NF)}' > ~/logs/perfmon.log
This command prints a header to a file, using the top command, and formats the string using awk.
The second command finds the process named process_name and continuously records its performance with the top command to the created file. The command is NOT awaited in Python.
The problem is that the command itself contains both double quotes " and apostrophe ', as well as the escape characters \t and \n. And it's a long string.
I've tried to define the command string as f-string and put double \ before the escape characters, I've tried to use the r-string, but with no success.
Either the header line is printed without tabs between words, or the command is misinterpreted by the awk command.
I attach my function below. Both commands need some help :)
I'd appreciate you checking the solution before posting here.
def execute_performance_monitoring(sample_frequency: int = default_top_frequency) -> None:
today = datetime.today()
date = today.strftime('%Y-%m-%d-%H-%M')
process_name = "find_and_log_me"
logs_dir = Path(logs_path)
full_file_name = logs_dir / f'{process_name}_{date}.log'
print_header_command = (
'top -n1 -b'
' | awk -vtime="$(date +%Y-%m-%d-%T)" \'NR==7{{printf("Time %s %s %s %s %s %s %s\\n",$1,$5,$6,$9,$10,$11,$NF)}}\''
f' > {full_file_name}'
)
print_perfmon_command = (
f'top -b -o +%CPU -d {sample_frequency}'
'| awk -vtime="$(date +%m-%d-%Y-%T)" \'NR>8{{printf("%s %s %s %s %s %s %s %s\\n"'
',\'time,$1,$5,$6,$9,$10,$11,$NF)}}\''
f' | grep --line-buffered {process_name} >> {full_file_name} &'
)
subprocess.run(print_header_command)
subprocess.run(
print_perfmon_command,
shell=True,
stdin=None,
stdout=None,
stderr=None,
close_fds=True,
)
After many tries and days without a success, I've moved the commands to a bash script that takes the relevant parameters, and Python script does the validation job, and calls the bash script with parameters.
I want to start the testrunner.bat of ReadyApi with parameters. I try to pass some XML parts (PeriodEnd in the code below) as arguments to subprocess.Popen:
argslist = ['C:/Program Files/SmartBear/ReadyAPI-1.9.0/bin/testrunner.bat',
'-a', '-s', 'TestSuite', '-c', 'TestCase', '-f', 'C:/temp/', '-P',
'PeriodEnd=<PeriodEnd>2017-04-11T00:00:00.000Z</PeriodEnd>',
'C:/temp/soapui-project.xml']
proc = Popen(argslist, stdout=PIPE, stderr=PIPE)
This produces the following error:
The system cannot find the file specified.
I found out, that the "<" and ">" are the problems. How can I escape them or pass them to Popen?
The escape character in CMD is ^.
C:\> echo asdf<123>
The syntax of the command is incorrect.
C:\> echo asdf^<123^>
asdf<123>
\ is used to escape character, try \<
https://docs.python.org/2.0/ref/strings.html
possibly use " instead of the '
I am using Python to do sed delete / replace with the help of the subprocess. Somehow I am not getting the number of escapes correct. Here is my code:
from subprocess import call
remover = ["sed", "-i", "'1d'", "file"]
call(remover)
removeq = ["sed", "-i", "'s/\"//g'", "file"]
call(removeq)
Both of these tasks produce the error message:
sed: -e expression #1, char 1: unknown command: `''
How many times does the ' sign need to be escaped in the first list and how many times does the " have to be escaped in the last one? I tried once, twice and three times, but to no avail. There are also no hidden characters that can potentially mess something up.
Any suggestions?
Fix the quoting mechanism:
sed -i 's/\"//g' file
Should be just:
sed -i 's/"//g' file
You can also take adventage of shlex library.
Example from interpreter:
>>> import shlex
>>> cmd = "sed -i '1d' file"
>>> shlex.split(cmd)
['sed', '-i', '1d', 'file']
>>> cmd = """sed -i 's/"//g' file"""
>>> shlex.split(cmd)
['sed', '-i', 's/"//g', 'file']
I want to run a bash command from python shell.
my bash is:
grep -Po "(?<=<cite>).*?(?=</cite>)" /tmp/file1.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > /tmp/file2.txt
what I tried is:
#!/usr/bin/python
import commands
commands.getoutput('grep ' + '-Po ' + '\"\(?<=<dev>\).*?\(?=</dev>\)\" ' + '/tmp/file.txt ' + '| ' + 'awk \'!x[$0]++\' ' + '> ' + '/tmp/file2.txt')
But I don't have any result.
Thank you
If you want to avoid splitting your arguments and worrying about pipes, you can use the shell=True option:
cmd = "grep -Po \"(?<=<dev>).*?(?=</dev>)\" /tmp/file.txt | awk -F/ '{print $1}' | awk '!x[$0]++' > file2.txt"
out = subprocess.check_output(cmd, shell=True)
This will run a subshell which will understands all your directives, including "|" for piping, ">" for redirection. If you do not do this, these symbols normally parsed by the shell will just be passed to grep program.
Otherwise, you have to create the pipes yourself. For example (untested code below):
grep_p = subprocess.Popen(["grep", "-Po", "(?<=<dev>).*?(?=</dev>)", "/tmp/file.txt"], stdout=subprocess.PIPE)
awk_p = subprocess.Popen(["awk", "-F/", "'{print $1}'"], stdin = grep_p.stdout)
file2_fh = open("file2.txt", "w")
awk_p_2 = subprocess.Popen(["awk", "!x[$0]++", stdout = file2_fh, stdin = awk_p.stdout)
awk_p_2.communicate()
However, you're missing the point of python if you are doing this. You should instead look into the re module: re.match, re.sub, re.search, though I'm not familiar enough with awk to translate your commands.
The recommend way to run system commands in python is to use the module subprocess.
import subprocess
a=['grep' ,'-Po', '"(?<=<dev>).*?(?=</dev>)"','/tmp/file.txt']
b=['awk', '-F/', '"{print $1}"']
c=["awk", '"!x[$0]++"']
p1 = subprocess.Popen(a,stdout=subprocess.PIPE)
p2 = subprocess.Popen(b,stdin=p1.stdout,stdout=subprocess.PIPE)
p3 = subprocess.Popen(c,stdin=p2.stdout,stdout=subprocess.PIPE)
p1.stdout.close()
p2.stdout.close()
out,err=p3.communicate()
print out
The point of creating pipes between each subprocess is for security and debugging reasons. Also it makes the code much clearer in terms, which process gets input and sends output to.
Let us write a simple function to easily deal with these messy pipes for us:
def subprocess_pipes (pipes, last_pipe_out = None):
import subprocess
from subprocess import PIPE
last_p = None
for cmd in pipes:
out_pipe = PIPE if not (cmd==pipes[-1] and last_pipe_out) else open(last_pipe_out, "w")
cmd = cmd if isinstance(cmd, list) else cmd.split(" ")
in_pipe = last_p.stdout if last_p else None
p = subprocess.Popen(cmd, stdout = out_pipe, stdin = in_pipe)
last_p = p
comm = last_p.communicate()
return comm
Then we run,
subprocess_pipes(("ps ax", "grep python"), last_pipe_out = "test.out.2")
The result is a "test.out.2" file with the contents of piping "ps ax" into "grep python".
In your case,
a = ["grep", "-Po", "(?<=<cite>).*?(?=</cite>)", "/tmp/file1.txt"]
b = ["awk", "-F/", "{print $1}"]
c = ["awk", "!x[$0]++"]
subprocess_pipes((a, b, c), last_pipe_out = "/tmp/file2.txt")
The commands module is obsolete now.
If you don't actually need the output of your command you can use
import os
exit_status = os.system("your-command")
Otherwise you can use
import suproccess
out, err = subprocess.Popen("your | commands", stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell = True).communicate()
Note: for your command you send stdout to file2.txt so I wouldn't expect to see anything in out you will however still see error messages on stderr which will go into err
you must use
import os
os.system(command)
I think what you are looking for is something like:
ubprocess.check_output(same as popen arguments, **kwargs) , use it the same way you would use a popen command , it should show you the output of the program that's being called.
For more details here is a link: http://freefilesdl.com/how-to-call-a-shell-command-from-python/
I have this commands in bash:
ACTIVE_MGMT_1=ssh -n ${MGMT_IP_1} ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;
I was trying to do it in Python like this:
active_mgmgt_1 = os.popen("""ssh -n MGMT_IP_1 ". .bash_profile; xms sho proc TRAF.*" 2>/dev/null |egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2""") ACTIVE_MGMT_1 = active_mgmgt_1.read().replace('\n', '')
It doesn't work; any advice please?
Your popen call needs be set to communicate via a pipe.
Also stop trying to put everything on one line - python doesn't require it and places a lot of empasis on readable code.
I would strongly suggest doing the string processing in python rather than egrep, (use find or re in python), awk (find or egrep) and cut (string split).
It is also recommended to use subprocess.Popen rather than os.popen functions. There is a suggestion to use shlex.spilt to clear up this sort of issue.
untested code
import subprocess
import re
import os
MGMT_IP_1 = os.getenv('MGMT_IP_1')
sp = subprocess.Popen(
['ssh', '-n', MGMT_IP_1, '. .bash_profile; xms sho proc TRAF.*'],
stdout=PIPE, stderr=None)
(result, outtext) = sp.communicate()
# Proceed to process outtext from here using re, find and split
# to the equivalent of egrep " A " |awk '/TRAF/{print $1}' |cut -d "." -f2;