We Keep Coding

Python - how to grep using a variable over ssh to remote machine

I'm trying to execute remote command over ssh.
all working well from the ssh connection and also with an other simple cmd command like 'ls'.
but, i would like to using ls and grep for a value stored in a variable(ptrn)
but i always failed.
i tried using:
cmd="ls -ltrd /export/home| grep %s ptrn"
cmd="ls -ltrd /export/home| grep" + ptrn
cmd="ls -ltrd /export/home| grep", ptrn
but no luck ):
my code:
BuildServer = 1.2.3.4
ptrn = "abc"
cmd="ls -ltrd /export/home| grep ptrn"
ssh = subprocess.Popen(["ssh", "%s" % BuildServer, cmd],shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print (ssh)
Can anyone please help me with that?
Thank you all.

If I read you correctly, you want to grep for the contents of the ptrn Python variable, so you need to pass that to grep, rather than its name, which doesn't mean anything in that context.
#!/usr/bin/env python
import subprocess
from pipes import quote
BuildServer = "1.2.3.4" # you pass it as a string to ssh
ptrn = "abc"
cmd="ls -ltrd /export/home| grep " + quote(ptrn) # Don't forget the space after grep.
ssh = subprocess.Popen(["ssh", "%s" % BuildServer, cmd], shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
print ssh.communicate()[0]
As for quote, this answer mentions it for quoting shell arguments, so that grep will consider it a single argument.
communicate() reads output from the process and waits for it to end.

You can try these:
cmd="ls -ltrd /export/home| grep %s"%ptrn
cmd="ls -ltrd /export/home| grep " + ptrn #notice the space

Combining multiple shell script, sed and user input in a single script

At the moment I am using two separate shell script to get the job done.
1) Listing the current directory and saving it as a .html file (first listing only the root directory followed by complete listing)
tree -L 1 -dH ./ >> /Volumes/BD/BD-V1.html && tree -H ./ >> /Volumes/BD/BD-V1.html
2) Using sed to remove unwanted lines (I'm on mac)
sed -i '' '/by Francesc Rocher/d' /Volumes/BD/BD-V1.html && sed -i '' '/by Steve Baker/d' /Volumes/BD/BD-V1.html && sed -i '' '/by Florian Sesser/d' /Volumes/BD/BD-V1.html
Now I want to combine them as a single script with user input for the file path. I was trying to do with python but no success
import subprocess
subprocess.call(["tree", "-d", "-L", "1"])
The above one can list the directory but I couldn't save the output (I have to do this inside of python), I tried something like this but didn't get work.
file = open('out.txt', 'w')
import subprocess
variation_string = subprocess.call(["tree", "-d", "-L", "1"])
file.write(variation_string)
file.close()
Also I'm not sure how to implement sed :(
edit: I'm a beginner

You can simply redirect the stdout to a file object:
from subprocess import check_call
with open("out.txt","w") as f:
check_call(["tree", "-d", "-L", "1"],stdout=f)
In your code you are basically trying to write the return code as that is what call returns to the file which would raise an error as write expects a string. If you wanted to store the output of running a command you would use check_output.

You can do this with the subprocess module. You can create another process that runs your command, and then communicate with it. This will give you the output.
import subprocess
file = open('out.txt', 'w')
...
command = "tree -d -L 1"
process = subprocess.Popen(command.split(), stdout=subprocess.PIPE)
output = process.communicate()[0]
...
file.write(output)
file.close()

How to use subprocess Popen?

I'm trying to execute a command using Popen.
The command uses some PostGIS/Postgresql utility programs to upload a raster file to a database and works when executed from the command line. It uses unix style pipes to chain 2 commands and looks like this:
"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres
When using within Python, I make it a string with the ' codes:
command = '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe" -d -I -C -e -Y -F -t 128x128 "C:\\temp\\SampleDTM\\SampleDTM.tif" test | "C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe" -h localhost -p 5432 -d adr_hazard -U postgres'
attempting to execute it results in an error:
p = subprocess.Popen(command)
ERROR: Unable to read raster file: test
The error seems like the command was not parsed correctly (it is interpreting the wrong argument as the raster file)
Am I using Popen wrong?

Your command uses pipe |. It requires a shell:
p = subprocess.Popen(command, shell=True)
The command itself as far as I can tell looks ok.

It's not necessary to use shell=True to achieve this with pipes. This can be done programmatically with pipes even where concern about insecure input is an issue. Here, conn_params is a dictionary with PASSWORD, NAME (database name), USER, and HOST keys.
raster2pgsql_ps = subprocess.Popen([
'raster2pgsql', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128',
'C:\\temp\\SampleDTM\\SampleDTM.tif',
'test'
], stdout=subprocess.PIPE)
# Connection made using conninfo parameters
# http://www.postgresql.org/docs/9.0/static/libpq-connect.html
psql_ps = subprocess.check_output([
'psql',
'password={PASSWORD} dbname={NAME} user={USER} host={HOST}'.format(**conn_params),
], stdin=raster2pgsql_ps.stdout)

The following worked for me on Windows, while avoiding shell=True
One can make use of Python's fstring formatting to make sure the commands will work in windows.
Please note that I used shp2pgsql but it should be a very similar process for raster2pgsql.
Parameters for the shp2pgsql: srid is the coordinate system of the shape file, filename is the path to the shape file to be imported, tablename is the name you'd like to give your table.
import os
import subprocess
shp2pgsql_binary = os.path.join(pgsql_dir, "bin", "shp2pgsql")
psql_binary = os.path.join(pgsql_dir, "bin", "psql")
command0 = f'\"{shp2pgsql_binary}\" -s {srid} \"{filename}\" {tablename}'
command1 = f'\"{psql_binary}\" \"dbname={databasename} user={username} password={password} host={hostname}\"'
try:
shp2pgsql_ps = subprocess.Popen(command0, stdout=subprocess.PIPE)
psql_ps = subprocess.check_output(command1, stdin=shp2pgsql_ps.stdout)
except:
sys.stderr.write("An error occurred while importing data into the database, you might want to \
check the SQL command below:")
sys.stderr.write(command)
raise
To adpat to raster2pgsql, you just need to modify the string in command0, e.g. -s {srid} becomes -d -I -C -e -Y -F -t 128x128. The string for command1 can remain the same.

PIPE = subprocess.PIPE
pd = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe", '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', "C:\\temp\\SampleDTM\\SampleDTM.tif", 'test'],
stdout=PIPE, stderr=PIPE)
stdout, stderr = pd.communicate()

It will be better to use subprocess.Popen in this way:
proc = subprocess.Popen(['"C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe"', '-d', '-I', '-C', '-e', '-Y', '-F', '-t', '128x128', '"C:\\temp\\SampleDTM\\SampleDTM.tif"', 'test', '|', '"C:\\Program Files\\PostgreSQL\\9.2\\bin\\psql.exe"', '-h', 'localhost', '-p', '5432', '-d', 'adr_hazard', '-U', 'postgres'], shell = True, stdout = subprocess.pipe, stderr = subprocess.STDOUT)
proc.wait()
result = proc.stdout.readlines()#if you want to process the result of your command
proc.kill()
B.T.W, it's good to format the path first, use:
path = os.path.normalpath("C:\\Program Files\\PostgreSQL\\9.2\\bin\\raster2pgsql.exe")
this will avoid some path problems for different OS platform.
The shell = True is important if you want to execute your command just like executing it in local shell.
Hope will help you.

python subprocess.popen tshark works with some args, but not others

When I execute a python script with this:
tsharkCall = ["tshark", "-a", "duration:6", "-i", "2", "-w", "thsark.pcap"]
tsharkProc = subprocess.Popen(tsharkCall, bufsize=0, executable="C:\\Program Files\\Wireshark\\tshark.exe")
A pcap file with the expected contents duly appears in the same folder as the script.
A second procedure to create a text file from the pcap does not work:
tsharkCall = ["tshark", "-i", "-", "<", "tshark.pcap", ">", "tshark.txt", "-V"]
tsharkProc = subprocess.Popen(tsharkCall, bufsize=0, executable="C:\\Program Files\\Wireshark\\tshark.exe")
I see "Capturing on Standard input" in the cmd window, but no "x packets captured", and no tshark.txt file appears in the folder.
From a command prompt in the same location, this does the job I am hoping for from the script:
>"C:\Program Files\Wireshark\tshark.exe" -i - < "tshark.pcap" > "tshark.txt" -V
It seems odd that one call works and the other doesn't. Any ideas as to what I'm missing?

subprocess.Popen by default bypasses CMD.EXE / sh, therefore command line I/O redirections (<, >) will not work. You can get a similar effect like this:
tsharkCall = ["tshark", "-i", "-", "-V"]
tsharkIn = open("tshark.pcap", "rb")
tsharkOut = open("tshark.txt", "wb")
tsharkProc = subprocess.Popen(tsharkCall,
stdin=tsharkIn,
stdout=tsharkOut,
executable="C:\\Program Files\\Wireshark\\tshark.exe")

This also works
tsharkCall = ["C:\\Program Files\\Wireshark\\tshark.exe", "-P", "-V", "-x", "-r", "C:\\Data\\PCAP_TEST_FILES\\test.pcap"]
tsharkOut = open("tshark.txt", "wb")
tsharkProc = subprocess.call(tsharkCall, stdout=tsharkOut)

How to execute shell command get the output and pwd after the command in Python

How can I execute a shell command, can be complicated like normal command in bash command line, get the output of that command and pwd after execution?
I used function like this:
import subprocess as sub
def execv(command, path):
p = sub.Popen(['/bin/bash', '-c', command],
stdout=sub.PIPE, stderr=sub.STDOUT, cwd=path)
return p.stdout.read()[:-1]
And I check if user use cd command but that will not work when user use symlink to cd or other wierd way to change directory.
and I need a dictionary that hold {'cwd': '<NEW PATH>', 'result': '<COMMAND OUTPUT>'}

If you use subprocess.Popen, you should get a pipe object that you can communicate() for the command output and use .pid() to get the process id. I'd be really surprised if you can't find a method to get the current working directory of a process by pid...
e.g.: http://www.cyberciti.biz/tips/linux-report-current-working-directory-of-process.html

I redirect stdout to stderr of pwd command. if stdout is empty and stderr is not a path then stderr is error of the command
import subprocess as sub
def execv(command, path):
command = 'cd %s && %s && pwd 1>&2' % (path, command)
proc = sub.Popen(['/bin/bash', '-c', command],
stdout=sub.PIPE, stderr=sub.PIPE)
stderr = proc.stderr.read()[:-1]
stdout = proc.stdout.read()[:-1]
if stdout == '' and not os.path.exists(stderr):
raise Exception(stderr)
return {
"cwd": stderr,
"stdout": stdout
}
UPDATE: here is better implemention (using last line for pwd and don't use stderr)
def execv(command, path):
command = 'cd %s && %s 2>&1;pwd' % (path, command)
proc = sub.Popen(['/bin/bash', '-c', command],
env={'TERM':'linux'},
stdout=sub.PIPE)
stdout = proc.stdout.read()
if len(stdout) > 1 and stdout[-1] == '\n':
stdout = stdout[:-1]
lines = stdout.split('\n')
cwd = lines[-1]
stdout = '\n'.join(lines[:-1])
return {
"cwd": cwd,
"stdout": man_to_ansi(stdout)
}

To get output of an arbitrary shell command with its final cwd (assuming there is no newline in the cwd):
from subprocess import check_output
def command_output_and_cwd(command, path):
lines = check_output(command + "; pwd", shell=True, cwd=path).splitlines()
return dict(cwd=lines[-1], stdout=b"\n".join(lines[:-1]))