I'm attempting to execute a command over SSH, but bash on the other end doesn't think it's escaped properly.
Here, self._client is a paramiko.SSHClient object; args is a list of arguments, the command to execute.
def run(self, args, stdin=None, capture_stdout=False):
"""Runs a command.
On success, returns the output, if requested, or None.
On failure, raises CommandError, with stderr and, if captured, stdout,
as well as the exit code.
"""
command = ' '.join(_shell_escape(arg) for arg in args)
print('About to run command:\n {}'.format(command))
print('About to run command:\n {!r}'.format(command))
channel = self._client.get_transport().open_session()
channel.exec_command(command)
_shell_escape:
_SHELL_SAFE = _re.compile(r'^[-A-Za-z0-9_./]+$')
def _shell_escape(s):
if _SHELL_SAFE.match(s):
return s
return '\'{}\''.format(s.replace('\'', '\'\\\'\''))
I'm attempt to run some Python through this. On stderr, I get back:
bash: -c: line 5: unexpected EOF while looking for matching `''
bash: -c: line 6: syntax error: unexpected end of file
The output from the two print statements:
About to run command:
python -c 'import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
' .
About to run command:
"python -c 'import os, sys\npath = sys.argv[1]\nif sys.version_info.major == 2:\n path = path.decode('\\''utf-8'\\'')\nentries = os.listdir(path)\nout = b'\\'''\\''.join(e.encode('\\''utf-8'\\'') + b'\\''\x00'\\'' for e in entries)\nsys.stdout.write(out)\n' ."
If I copy and paste the output of command, and paste it into bash, it executes, so it really does appear to be properly escaped. My current understanding is that SSH, on the other end, will take command, and run [my_shell, '-c', command].
Why is bash erroring on that command?
The input contains an embedded nul character, which bash appears to treat as the end of the string. (I'm not sure there's any way it couldn't!). This is visible in my question, where I output command:
About to run command:
"python -c 'import os, sys [SNIP…] + b'\\''\x00'\\'' for [SNIP…]"
That's a repr output, but notice the single slash before the x in \x00: that's an actual \x00 that made it through. My original code has this Python embedded as a snippet, which I didn't include (I didn't believe it was relevant):
_LS_CODE = """\
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('utf-8')
entries = os.listdir(path)
out = b''.join(e.encode('utf-8') + b'\x00' for e in entries)
sys.stdout.write(out)
"""
Here, Python's """ is still processing \ as an escape character. I need to double up, or look into raw strings (r""")
You need to escape newlines as well. A better option is to put the program text in a here document.
Make the output of "About to run command:" to look like
python -c << EOF
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
.
EOF
Maybe you wouldn't need to escape anything at all.
Related
i'm building tester for programs in different languages, but I'm not able to get C program working, currently the command is called like this:
codeResult = subprocess.run(self.createRunCommand(currLanguage, file),
input = codeToTest,
shell = True,
timeout = TIMEOUT,
capture_output=True)
and createRunCommand() returns:
def createRunCommand(self, language, file):
if language == '.py':
command = f'python {file}'
elif language == '.c':
if not os.path.exists(f'C:/<myPath>/{file}.out'):
command = f'gcc -std=c11 {file} -o C:/<myPath>/{file}.out \
./C:/<myPath>/{file}.out'
else:
command = f'./C:/<myPath>/{file}.out'
elif language == '.java':
command = f''
elif language == '.cpp':
command = f''
return command
the input and test itself is good, as it runs correctly with a python program, but I cannot figure out how to setup C (and probably other compiled first languages).
You'll need multiple command invocations for (e.g.) C/C++, so have your createRunCommand return multiple.
I also changed things up here to
automatically figure out the language from the extension of the filename
use a list of arguments instead of a string; it's safer
use sys.executable for the current Python interpreter, and shutil.which("gcc") to find gcc.
import os
import shlex
import shutil
import subprocess
import sys
def get_commands(file):
"""
Get commands to (compile and) execute `file`, as a list of subprocess arguments.
"""
ext = os.path.splitext(file)[1].lower()
if ext == ".py":
return [(sys.executable, file)]
if ext in (".c", ".cpp"):
exe_file = f"{file}.exe"
return [
(shutil.which("gcc"), "-std=c11", file, "-o", exe_file),
(exe_file,),
]
raise ValueError(f"Unsupported file type: {ext}")
filename = "foo.py"
for command in get_commands(filename):
print(f"Running: {shlex.join(command)}")
code_result = subprocess.run(command, capture_output=True)
I'm trying to call ffmpeg command using subprocess.call() on linux, but I'm unable to get the arguments right. Before hand, I used os.system and it worked, but this method is not recommended.
Using arguments with a dash such as "-i" gets me this error
Unrecognized option 'i "rtsp://192.168.0.253:554/user=XXX&password=XXX&channel=0&stream=0.sdp?real_stream"'.
Error splitting the argument list: Option not found
Using arguments without dash like "i" gets me this error
[NULL # 0x7680a8b0] Unable to find a suitable output format for 'i rtsp://192.168.0.253:554/user=admin&password=&channel=0&stream=0.sdp?real_stream'
i rtsp://192.168.0.253:554/user=XXX&password=XXX&channel=0&stream=0.sdp?real_stream: Invalid argument
Here's the code
class IPCamera(Camera):
"""
IP Camera implementation
"""
def __init__(self,
path='\"rtsp://192.168.0.253:554/'
'user=XXX&password=XXX&channel=0&stream=0.sdp?real_stream\"'):
"""
Constructor
"""
self.path = path
def __ffmpeg(self, nb_frames=1, filename='capture%003.jpg'):
"""
"""
ffm_input = "-i " + self.path
ffm_rate = "-r 5"
ffm_nb_frames = "-vframes " + str(nb_frames)
ffm_filename = filename
if platform.system() == 'Linux':
ffm_path = 'ffmpeg'
ffm_format = '-f v4l2'
else:
ffm_path = 'C:/Program Files/iSpy/ffmpeg.exe'
ffm_format = '-f image2'
command = [ffm_path, ffm_input, ffm_rate, ffm_format, ffm_nb_frames, ffm_filename]
subprocess.call(command)
print(command)
BTW, I'm running this command on a MT7688.
Thanks
You have to split the options:
command = [ffm_path, '-i', ffm_input, '-r', ffm_rate, '-f', ffm_format, '-vframes', ffm_nb_frames, ffm_filename]
The ffm_input, ffm_rate, ffm_format should only contain the value:
ffm_input = self.path
ffm_rate = '5'
ffm_nd_frames = str(nb_frames)
ffm_format = 'v412' if platform.system() == 'Linux' else 'image2'
When you pass a list no parsing is done so -r 5 is taken as a single argument but the program expects you to provide two separate arguments -r followed by 5.
Basically if you put them as a single element in the list it's as if you quoted them on the command line:
$ echo "-n hello"
-n hello
$ echo -n hello
hello$
In the first example echo sees a single argument -n hello. Since it does not match any option it just prints it. In the second case echo sees two arguments -n and hello, the first is the valid option to suppress end of line and as you can see the prompt is printed right after hello and not on its own line.
Following is the code which I am trying to execute using python
from subprocess import Popen, PIPE
cmd = 'if (-e "../a.txt") then \n ln -s ../a.txt . \n else \n echo "file is not present " \n endif'
ret_val = subprocess.call(cmd,shell="True")
When executed gives following error message
/bin/sh: -c: line 5: syntax error: unexpected end of file
In sh scripts, if is terminated with fi, not endif.
http://tldp.org/LDP/Bash-Beginners-Guide/html/sect_07_01.html
Or just write the darn code in Python:
import os
if os.path.exists('../a.txt'):
print 'Exists'
os.symlink('../a.txt', 'a.txt')
else:
print 'Does not exist'
os.path.exists ()
os.symlink()
If you really want to run tcsh commands, then:
import shlex
import subprocess
args = ['tcsh', '-c'] + shlex.split(some_tcsh_command)
ret = suprocess.call(args)
I write lots of small scripts to manipulate files on a Bash-based server. I would like to have a mechanism by which to log which commands created which files in a given directory. However, I don't just want to capture every input command, all the time.
Approach 1: a wrapper script that uses a Bash builtin (a la history or fc -ln -1) to grab the last command and write it to a log file. I have not been able to figure out any way to do this, as the shell builtin commands do not appear to be recognized outside of the interactive shell.
Approach 2: a wrapper script that pulls from ~/.bash_history to get the last command. This, however, requires setting up the Bash shell to flush every command to history immediately (as per this comment) and seems also to require that the history be allowed to grow inexorably. If this is the only way, so be it, but it would be great to avoid having to edit the ~/.bashrc file on every system where this might be implemented.
Approach 3: use script. My problem with this is that it requires multiple commands to start and stop the logging, and because it launches its own shell it is not callable from within another script (or at least, doing so complicates things significantly).
I am trying to figure out an implementation that's of the form log_this.script other_script other_arg1 other_arg2 > file, where everything after the first argument is logged. The emphasis here is on efficiency and minimizing syntax overhead.
EDIT: iLoveTux and I both came up with similar solutions. For those interested, my own implementation follows. It is somewhat more constrained in its functionality than the accepted answer, but it also auto-updates any existing logfile entries with changes (though not deletions).
Sample usage:
$ cmdlog.py "python3 test_script.py > test_file.txt"
creates a log file in the parent directory of the output file with the following:
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
Additional file changes are added to the log;
$ cmdlog.py "python3 test_script.py > test_file_2.txt"
the log now contains
2015-10-12#10:47:09 test_file.txt "python3 test_script.py > test_file.txt"
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
Running on the original file name again changes the file order in the log, based on modification time of the files:
$ cmdlog.py "python3 test_script.py > test_file.txt"
produces
2015-10-12#10:47:44 test_file_2.txt "python3 test_script.py > test_file_2.txt"
2015-10-12#10:48:01 test_file.txt "python3 test_script.py > test_file.txt"
Full script:
#!/usr/bin/env python3
'''
A wrapper script that will write the command-line
args associated with any files generated to a log
file in the directory where the files were made.
'''
import sys
import os
from os import listdir
from os.path import isfile, join
import subprocess
import time
from datetime import datetime
def listFiles(mypath):
"""
Return relative paths of all files in mypath
"""
return [join(mypath, f) for f in listdir(mypath) if
isfile(join(mypath, f))]
def read_log(log_file):
"""
Reads a file history log and returns a dictionary
of {filename: command} entries.
Expects tab-separated lines of [time, filename, command]
"""
entries = {}
with open(log_file) as log:
for l in log:
l = l.strip()
mod, name, cmd = l.split("\t")
# cmd = cmd.lstrip("\"").rstrip("\"")
entries[name] = [cmd, mod]
return entries
def time_sort(t, fmt):
"""
Turn a strftime-formatted string into a tuple
of time info
"""
parsed = datetime.strptime(t, fmt)
return parsed
ARGS = sys.argv[1]
ARG_LIST = ARGS.split()
# Guess where logfile should be put
if (">" or ">>") in ARG_LIST:
# Get position after redirect in arg list
redirect_index = max(ARG_LIST.index(e) for e in ARG_LIST if e in ">>")
output = ARG_LIST[redirect_index + 1]
output = os.path.abspath(output)
out_dir = os.path.dirname(output)
elif ("cp" or "mv") in ARG_LIST:
output = ARG_LIST[-1]
out_dir = os.path.dirname(output)
else:
out_dir = os.getcwd()
# Set logfile location within the inferred output directory
LOGFILE = out_dir + "/cmdlog_history.log"
# Get file list state prior to running
all_files = listFiles(out_dir)
pre_stats = [os.path.getmtime(f) for f in all_files]
# Run the desired external commands
subprocess.call(ARGS, shell=True)
# Get done time of external commands
TIME_FMT = "%Y-%m-%d#%H:%M:%S"
log_time = time.strftime(TIME_FMT)
# Get existing entries from logfile, if present
if LOGFILE in all_files:
logged = read_log(LOGFILE)
else:
logged = {}
# Get file list state after run is complete
post_stats = [os.path.getmtime(f) for f in all_files]
post_files = listFiles(out_dir)
# Find files whose states have changed since the external command
changed = [e[0] for e in zip(all_files, pre_stats, post_stats) if e[1] != e[2]]
new = [e for e in post_files if e not in all_files]
all_modded = list(set(changed + new))
if not all_modded: # exit early, no need to log
sys.exit(0)
# Replace files that have changed, add those that are new
for f in all_modded:
name = os.path.basename(f)
logged[name] = [ARGS, log_time]
# Write changed files to logfile
with open(LOGFILE, 'w') as log:
for name, info in sorted(logged.items(), key=lambda x: time_sort(x[1][1], TIME_FMT)):
cmd, mod_time = info
if not cmd.startswith("\""):
cmd = "\"{}\"".format(cmd)
log.write("\t".join([mod_time, name, cmd]) + "\n")
sys.exit(0)
You can use the tee command, which stores its standard input to a file and outputs it on standard output. Pipe the command line into tee, and pipe tee's output into a new invocation of your shell:
echo '<command line to be logged and executed>' | \
tee --append /path/to/your/logfile | \
$SHELL
i.e., for your example of other_script other_arg1 other_arg2 > file,
echo 'other_script other_arg1 other_arg2 > file' | \
tee --append /tmp/mylog.log | \
$SHELL
If your command line needs single quotes, they need to be escaped properly.
OK, so you don't mention Python in your question, but it is tagged Python, so I figured I would see what I could do. I came up with this script:
import sys
from os.path import expanduser, join
from subprocess import Popen, PIPE
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
log_file = join(home, "command_log")
command = sys.argv[1:]
with open(log_file, "a") as fout:
fout.write("{}\n".format(" ".join(command)))
out, err = issue_command(command)
which you can call like (if you name it log_this and make it executable):
$ log_this echo hello world
and it will put "echo hello world" in a file ~/command_log, note though that if you want to use pipes or redirection you have to quote your command (this may be a real downfall for your use case or it may not be, but I haven't figured out how to do this just yet without the quotes) like this:
$ log_this "echo hello world | grep h >> /tmp/hello_world"
but since it's not perfect, I thought I would add a little something extra.
The following script allows you to specify a different file to log your commands to as well as record the execution time of the command:
#!/usr/bin/env python
from subprocess import Popen, PIPE
import argparse
from os.path import expanduser, join
from time import time
def issue_command(command):
process = Popen(command, stdout=PIPE, stderr=PIPE, shell=True)
return process.communicate()
home = expanduser("~")
default_file = join(home, "command_log")
parser = argparse.ArgumentParser()
parser.add_argument("-f", "--file", type=argparse.FileType("a"), default=default_file)
parser.add_argument("-p", "--profile", action="store_true")
parser.add_argument("command", nargs=argparse.REMAINDER)
args = parser.parse_args()
if args.profile:
start = time()
out, err = issue_command(args.command)
runtime = time() - start
entry = "{}\t{}\n".format(" ".join(args.command), runtime)
args.file.write(entry)
else:
out, err = issue_command(args.command)
entry = "{}\n".format(" ".join(args.command))
args.file.write(entry)
args.file.close()
You would use this the same way as the other script, but if you wanted to specify a different file to log to just pass -f <FILENAME> before your actual command and your log will go there, and if you wanted to record the execution time just provide the -p (for profile) before your actual command like so:
$ log_this -p -f ~/new_log "echo hello world | grep h >> /tmp/hello_world"
I will try to make this better, but if you can think of anything else this could do for you, I am making a github project for this where you can submit bug reports and feature requests.
I am trying to source a bash script containing some environment variables in python. I followed one other thread to do it. But, there seems that one of the variable is malformed, as can be seen in the given snippet.
COLORTERM=gnome-terminal
mc=() { . /usr/share/mc/mc-wrapper.sh
}
_=/usr/bin/env
I am using the following code to set up the current environment.
import os
import pprint
import subprocess
command = ['bash', '-c', 'source init_env && env']
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for line in proc.stdout:
(key, _, value) = line.partition("=")
os.environ[key] = value
proc.communicate()
'
If I change the above code a little like putting a condition:
for line in proc.stdout:
(key, _, value) = line.partition("=")
if not value:
continue
os.environ[key] = value
then things are working but the environment is corrupted because of one missing bracket as can be seen from the snippet of environment variable that the bracket is appearing on new line. Because of this corruption, If I run some other command like
os.system("ls -l")
it gives me the following error
sh: mc: line 1: syntax error: unexpected end of file
sh: error importing function definition for `mc'
What could be the possible solutions for this problem?
Thanks alot
Probably the best way to do this is to create a separate program that writes out the environment variables in a way that is easily and unambiguously processed by your own program; then call that program instead of env. Using the standard pickle module, that separate program can be as simple as this:
import os
import sys
import pickle
pickle.dump(os.environ, sys.stdout)
which you can either save into its own .py file, or else put directly in a Bash command:
python -c 'import os, sys, pickle; pickle.dump(os.environ, sys.stdout)'
In either case, you can process its output like this:
import os
import pprint
import subprocess
import pickle
command = [
'bash',
'-c',
'source init_env && ' +
'python -c "import os, sys, pickle; ' +
'pickle.dump(os.environ, sys.stdout)"'
]
proc = subprocess.Popen(command, stdout = subprocess.PIPE)
for k, v in pickle.load(proc.stdout).iteritems():
os.environ[k] = v
proc.communicate()