I'm trying to write a python script to copy files from a remote server to a local directory via scp.
Because I'm running this on an OpenELEC distribution (minimal HTPC linux distro, read-only filesystem except for userhome makes it impractical to install python ssh module), I'm doing this ugly and just passing the filename to the scp command via os.system.
SCPCopy = "scp -c blowfish -C user#host:\"" + pipes.quote(file) + "\" /storage/downloads/incoming/"
SCPCopy = SCPCopy.replace('\n','')
os.system(SCPCopy)
This works, except for filenames containing an apostrophe.
Below is an example of what gets passed to os.system in a file with an apostrophe:
scp -c blowfish -C user#host:"'/media/sdi1/home/data/bob'"'"'s file.avi'" /storage/downloads/incoming/
And the error:
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
It looks pipes.quote(x) is escaping the apostrophe (as it should), but obviously the syntax is still incorrect. I've experimented ditching pipes.quote(x) and replacing apostrophes with /' but that isn't getting me anywhere either.
As scp is based on SSH, the filenames you give to it are subject to shell escaping on the remote side as well. Thus you need to escape twice.
A correctly escaped cmdline for the shell:
scp -c blowfish -C user#host:"\"/media/sdi1/home/data/bob's file\"" /storage/.../
To make a python string, we have to add one more level of escaping. To stay sane, we could use triple-quotes:
"""scp -c blowfish -C user#host:"\"/media/sdi1/home/data/bob's file\"" /storage/.../"""
If you do it programmatically (e.g. using the deprecated pipes.quote), then don't touch the filename at all (in your example above, you added apostrophes around the filename).
fp = "/media/sdi1/home/data/bob's file.avi"
fp = "user#host:" + pipes.quote(pipes.quote(fp))
cmdline = "scp -c blowfish -C " + fp + " /storage/downloads/incoming/"
os.system(cmdline)
This is admittedly confusing. For a simple model, the whole point of pipes.quote is to escape the input so that the input will be parsed by the shell as exactly one word, which is equal to the input.
The following is a more generally correct way (and yields the same result):
fp = "/media/sdi1/home/data/bob's file.avi"
# the filepath argument escaped for ssh/scp on the remote side
fp = pipes.quote(fp)
commandargs = ["scp", "-c", "blowfish", "-C", "user#host:"+fp, "/storage/downloads/incoming/"]
# escape all words for the local shell, and then concatenate space-separated
cmdline = " ".join(map(pipes.quote, commandargs))
os.system(cmdline)
It expresses more clearly the intent: Controlling what words exactly the shell will parse.
But why start with a shell in the first place? We don't need one and can save the escaping on the local side. To spawn a process with our args, directly, use commands from the os.exec* family.
fp = pipes.quote("/media/sdi1/home/data/bob's file.avi")
commandargs = ["scp", "-c", "blowfish", "-C", "user#host:"+fp, "/storage/downloads/incoming/"]
if os.fork() == 0:
os.execvp("scp", commandargs)
Related
I am using Inkscape to take an input single page pdf file and to output an svg file. The following works from the command line
c:\progra~1\Inkscape\inkscape -z -f "N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf" -l "N:\pdf_skunkworks\inflation-report-may-2018-page0.svg"
where -z is short for --without-gui, -f is short for input file, -l is short for --export-plain-svg. And that works from command line.
I could not get the equivalent to work from Python, either passing the command line as one long string or as separate arguments. stderr and stdout give no error as they both print None
import subprocess #import call,subprocess
#completed = subprocess.run(["c:\Progra~1\Inkscape\Inkscape.exe",r"-z -f \"N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf\" -l \"N:\pdf_skunkworks\inflation-report-may-2018-page0.svg\""])
completed = subprocess.run(["c:\Progra~1\Inkscape\Inkscape.exe","-z", r"-f \"N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf\"" , r"-l \"N:\pdf_skunkworks\inflation-report-may-2018-page0.svg\""])
print ("stderr:" + str(completed.stderr))
print ("stdout:" + str(completed.stdout))
Just to test OS plumbing I wrote some VBA code (my normal language) and this works
Sub TestShellToInkscape()
'* Tools->References->Windows Script Host Object Model (IWshRuntimeLibrary)
Dim sCmd As String
sCmd = "c:\progra~1\Inkscape\inkscape -z -f ""N:\pdf_skunkworks\inflation-report-may-2018-page0.pdf"" -l ""N:\pdf_skunkworks\inflation-report-may-2018-page0.svg"""
Debug.Print sCmd
Dim oWshShell As IWshRuntimeLibrary.WshShell
Set oWshShell = New IWshRuntimeLibrary.WshShell
Dim lProc As Long
lProc = oWshShell.Run(sCmd, 0, True)
End Sub
So I'm obviously doing something silly in the Python code. I'm sure experienced Python programmer could solve easily.
Swap your slashes:
import subprocess #import call,subprocess
completed = subprocess.run(['c:/Progra~1/Inkscape/Inkscape.exe',
'-z',
'-f', r'N:/pdf_skunkworks/inflation-report-may-2018-page0.pdf' ,
'-l', r'N:/pdf_skunkworks/inflation-report-may-2018-page0.svg'])
print ("stderr:" + str(completed.stderr))
print ("stdout:" + str(completed.stdout))
Python knows to swap forward slashes for back slashes on windows OS, and your back slashes are currently acting as escape prefixes.
I am having some trouble with the subprocess module. I would like the module to run the shell command equivalent to 'ls -l "/path/to/file/with possible space in directory/or with space in name"'. Subprocess works fine when the filename is not a variable. If the filename is a variable that contains the quotes, then it doesn't work.
Code that doesn't work:
import subprocess
archive_file_list = "/var/tmp/list"
archive = open(archive_file_list, "r")
for line in archive:
noreturnline = line[:-1]
quotedline = "\"" + noreturnline + "\""
if extension == "zip":
print quotedline
archivelist = subprocess.check_output(['ls', '-l', quotedline])
print archivelist
Code that works:
archivelist = subprocess.check_output(['ls', '-l', "/path/to/file/with possible space in directory/or with space in name"])
Here is the output for the code that doesn't work:
"/path/to/file/with possible space in directory/or with space in name"
ls: cannot access "/path/to/file/with possible space in directory/or with space in name" No such file or directory
Traceback (most recent call last):
File "./archive_test.py", line 12, in <module>
archivelist = subprocess.check_output(['ls', '-l', quotedline])
File "/usr/lib64/python2.7/subprocess.py", line 575, in check_output
raise CalledProcessError(retcode, cmd, output=output)
subprocess.CalledProcessError: Command '['ls', '-l', '"/path/to/file/with possible space in directory/or with space in name"']' returned non-zero exit status 2
Before you ask - yes, I have already verified that "/path/to/file/with possible space in directory/or with space in name" does in fact exist by running 'ls -l' from the command line.
Any help would be appreciated. Thanks in advance.
in the first command (which is the best option there is):
archivelist = subprocess.check_output(['ls', '-l', "/path/to/file/with possible space in directory/or with space in name"])
the third argument is actually /path/to/file/with possible space in directory/or with space in name (without quotes) which is the filename that exists, and the command works.
Since shell=True isn't even set, the command is directly passed to exec, with the arguments passed as-is: the spaces & other chars are preserved.
If you add more quotes, they're not removed and they're passed literally to ls.
Since there's no such file called "/path/to/file/with possible space in directory/or with space in name" (with quotes), the file/dir isn't found.
There's another (dirty) way of calling a command: passing the full command as a string (not as a list of parameters). In that case, that would work (without shell=True at least on Windows, subprocess seems to handle the argument splitting, shell=True seems to be required on Unix-like systems):
subprocess.check_output('ls -l "/path/to/file/with possible space in directory/or with space in name"')
but your first approach is cleaner, specially if you don't know the directory name because it's a parameter. Let subprocess do the heavy lifting for you.
On Unix-like systems, using this last approach requires shell=True, but then you're exposing your program to malicious attacks like any open system call (appending ;rm -rf / to the filename, evaluating sub-shells for instance)
Final note: if you're really planning to use ls and parse its output, don't do it (http://mywiki.wooledge.org/ParsingLs), use standard os.listdir, os.path.getsize/getmtime & os.stat calls to get the information you need.
I am able to run this properly using os.system. It is writing pcap file into text.
os.system("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r filename.pcap > testfile")
But when I tried to give input file from termimal, I got following error:
tshark: -z invalid argument
host = raw_input("Enter file name: ")
test = subprocess.Popen(["tshark","-z","'proto,colinfo,tcp.srcport,tcp.srcport'","-r",host,">","testfile"], stdout=subprocess.PIPE)
output = test.communicate()[0]
Can anybody please figure out where I am making mistake?
To emulate the os.system command, use the shell=True parameter to subprocess.Popen and provide the same command string (not an array of strings):
subprocess.Popen("tshark -z 'proto,colinfo,tcp.srcport,tcp.srcport' -r "
+ host + "> testfile", stdout=subprocess.PIPE, shell=True)
You need a shell to interpret your command line as you are using output redirection to a file ("> testfile").
In your example, you are passing each element of the string list to the execve() system call and hence as parameters to the tshark command (which gets 'proto,colinfo,tcp.srcport,tcp.srcport' as the argument to the -z option instead of proto,colinfo,tcp.srcport,tcp.srcport and which won't know what to do with the > and testfile arguments).
As wnnmaw points out in his comment, using os.system or subprocess.Popen with shell=True with command lines built from user input (the host variable in your case) allows a user to pass arbitrary data to the shell. This can be used to execute (potentially nasty) commands on your system.
For instance, setting host in your example to ; /bin/rm -rf / would delete every file on a UNIX system (assuming the user running the process had enough privilege).
It is therefore very important to validate an user input before adding it to the command string.
I'm trying to call 'sed' from Python and having troubles passing the command line via either subprocess.check_call() or os.system().
I'm on Windows 7, but using the 'sed' from Cygwin (it's in the path).
If I do this from the Cygwin shell, it works fine:
$ sed 's/ /\ /g' <"C:foobar" >"C:foobar.temp"
In Python, I've got the full pathname I'm working with in "name". I tried:
command = r"sed 's/ /\ /g' " + "<" '\"' + name + '\" >' '\"' + name + '.temp' + '\"'
subprocess.check_call(command, shell=True)
All the concatenation is there to make sure I have double quotes around the input and output filenames (in case there are blank spaces in the Windows file path).
I also tried it replacing the last line with:
os.system(command)
Either way, I get this error:
sed: -e expression #1, char 2: unterminated `s' command
'amp' is not recognized as an internal or external command,
operable program or batch file.
'nbsp' is not recognized as an internal or external command,
operable program or batch file.
Yet, as I said, it works OK from the console. What am I doing wrong?
The shell used by subprocess is probably not the shell you want. You can specify the shell with executable='path/to/executable'. Different shells have different quoting rules.
Even better might be to skip subprocess altogether, and write this as pure Python:
with open("c:foobar") as f_in:
with open("c:foobar.temp", "w") as f_out:
for line in f_in:
f_out.write(line.replace(' ', ' '))
I agree with Ned Batchelder's assessment, but think what you might want to consider using the following code because it likely does what you ultimately want to accomplish which can be done easily with the help of Python's fileinput module:
import fileinput
f = fileinput.input('C:foobar', inplace=1)
for line in f:
line = line.replace(' ', ' ')
print line,
f.close()
print 'done'
This will effectively update the given file in place as use of the keyword suggests. There's also an optional backup= keyword -- not used above -- which will save a copy of the original file if desired.
BTW, a word of caution about using something like C:foobar to specify the file name because on Windows it means a file of that name in whatever the current directory is on drive C:, which might not be what you want.
I think you'll find that, in Windows Python, it's not actually using the CygWin shell to run your command, it's instead using cmd.exe.
And, cmd doesn't play well with single quotes the way bash does.
You only have to do the following to confirm that:
c:\pax> echo hello >hello.txt
c:\pax> type "hello.txt"
hello
c:\pax> type 'hello.txt'
The system cannot find the file specified.
I think the best idea would be to use Python itself to process the file. The Python language is a cross-platform one which is meant to remove all those platform-specific inconsistencies, such as the one you've just found.
I'm trying to write an svn pre-commit hook in python. Part of this involves checking the diff file to see if there are any actual file changes (as opposed to just property changes).
I have a working grep command which I can execute fine on the shell
grep "^\(Added: \|Modified: \|Deleted: \)" diff filename | grep -v 'svn:'
However when I put it through subprocess.POpen it escapes all my backslashes, which knackers the regexp.
Executing command: ['grep', '"^\\Added: \\|Modified: \\|Deleted: \\)", ...]
How do I avoid this?
NB: I'm aware that I can pipe results between subprocesses and I can do the two greps that way. I need help getting the first one working first though :/
NB2: I also tried using filterdiff --clean instead and couldn't get it to work. Searching for Added, Modified or Deleted lines, removing those with 'svn:' in and checking I had some results seemed to work though.
Python code:
command = ['grep', '"^\(Added: \|Modified: \|Deleted: \)"', filename]
sys.stdout.write('Executing command: %s\n' % (command))
p = subprocess.Popen(command,
stdin = subprocess.PIPE
stdout = subprocess.PIPE
stderr = subprocess.STDOUT
shell = True)
data = p.stdout.read()
if len(data) == 0:
sys.stdout.write("Diff does not contain any file modifications./n")
exit(0)
You need to consider what you want grep to see in its command line arguments.
The first argument needs to be the literal string "^\(Added: \|Modified: \|Deleted: \)", so that means that it shouldn't include the double quotes but should include the backslashes.
The way to express this kind of string is to use Python raw strings:
command = ['grep', r'^\(Added: \|Modified: \|Deleted: \)', filename]
A good way to check what you're actually running is to replace grep by echo so you can at least see what you're passing to the command.