I am attempting to create a function in python 2.7.8 to delete files that are no longer needed on a remote PC.
If I run it from the command line I am able to do this:
C:\>del /f /q \\RemotePC\d$\temp\testfile.txt && if exist \\RemotePC\d$\temp\testfile.txt (set errorlevel=1) else (set errorlevel=0)
From this I get the expected result:
C:\>echo %errorlevel%
0
But when I attempt to put this into my python function it does not work. The test variable does not get set.
Here is what I have in my python function:
def DelFile(self, address, del_file):
cmd_line = "del /f /q \\\\" + address + del_file + "\&\& if exist \"\\\\" + address + del_file + "\" (set errorlevel=1) else (set errorlevel=0)"
myP = subprocess.Popen(cmd_line, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
out = timeout(myP.communicate, (), {}, 120, (None, None))
result = out[0]
error = out[1]
error_code = None
error_code = myP.returncode
notes = []
return cmd_line, result, error, error_code, notes
I have verified that the address and del_file variables are formatted correctly in order to get me to the desired file. I have also verified that it deletes or does not delete the intended file based on the circumstances. But for some reason the test variable never gets set.
It is my understanding that the ampersand character needs to be escaped, like so "\&" when being used in a string. Is this not true or am I not escaping it correctly?
Your cmd_line should look like this
cmd_line = "del /f /q \\\\{0}{1} & if exist \\\\{0}{1} (set test=1) else (set test=0)".format(address, del_file)
Try using one &
I figured out the issue, turns out I actually had a few. The first being that the ampersand apparently does not need to be escaped (Thank you FirebladeDan), the second being that I was missing a space before the ampersands (Thank you again to FirebladeDan for the tip with formatting, that is how I noticed the missing space). The last being more of a CMD issue than python, I was trying to directly set the errorlevel variable, when I should have been using the exit /b with the respective 1 or 0.
As a side note, I also added in the use of psexec, it was not necessary for the solution, it just helps with the performance of the overall program.
Here is what I ended up with:
def DelFile(self, address, del_file):
cmd_line = r'psexec \\{0} cmd /c "del /f /q "{1}" && if exist "{1}" (exit /b 1) else (exit /b 0)"'.format(address, del_file)
myP = subprocess.Popen(cmd_line, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
out = timeout(myP.communicate, (), {}, 60, (None, None))
result = out[0]
error = out[1]
error_code = None
error_code = myP.returncode
notes = []
return cmd_line, result, error, error_code, notes
Related
I have one problem with the python subprocess module.
import os, subprocess
BLEU_SCRIPT_PATH = os.path.join(os.path.abspath(os.path.dirname(__file__)), 'multi-bleu.perl')
command = BLEU_SCRIPT_PATH + ' %s < %s'
ref = "ref.en-fr.test.txt"
hyp = "hyp100.en-fr.test.txt"
p = subprocess.Popen(command % (ref, hyp), stdout=subprocess.PIPE, shell=True)
result = p.communicate()[0].decode("utf-8")
# ...
# ...
The multi-bleu.perl file does the evaluation and returns a real number or an error if any; but that's not my concern.
The last line of code automatically opens the multi-bleu.perl file with my default text editor, stops the program execution until I manually close the file.
How can I disable this behavior?
I don't think subprocess.Popen interprets any shebang in the file. You need to specify the executable in the command to be executed. Also, Popen requires a list as first argument, so you need to "lift up" your string formatting into the command-list.
command = [
'/path/to/perl',
BLEU_SCRIPT_PATH + ' %s < %s' % (ref, hyp)
]
p = subprocess.Popen(command, stdout=subprocess.PIPE, shell=True)
You might also want to have a look at subprocess.check_output which will make the code a bit easier.
I am attempting to use paramiko to send powershell commands over ssh to a Windows box with OpenSSH on it. The commands appear to be successful (return code 0) even when they should fail, and I'm not getting any output on the pipes. When I try commands like making a directory, it is not created, which makes it seem as though the commands aren't reaching the remote system, but also aren't throwing errors.
First, here's my code:
version = self.oscall.run_remote(['java', '-version'])
def run_remote(self, command): # Command is a list of command + args
string = ""
self.timeout = 300
for arg in command:
string = string + " " + arg
self.client.connect(self.host, username=self.user, password=self.pw, timeout=self.timeout)
self.transport = self.client.get_transport()
self.transport.set_keepalive(1)
self.channel = self.transport.open_session(timeout=self.timeout) # transport is abstract connection, session is socket
if self.channel.gettimeout() == None: self.channel.settimeout(self.timeout)
self.channel.exec_command(string)
self.out = self.channel.makefile()
self.err = self.channel.makefile_stderr()
self.output = CallOutput(self.out, self.err, None)
self.output.returncode = self.channel.recv_exit_status()
self.channel.close()
return self.output
class CallOutput(object):
def __init__(self, out, err, rc):
self.out = out.readlines()
self.err = err.readlines()
self.outfile = tempfile.TemporaryFile()
for line in self.out:
if isinstance(line, unicode): line = line.encode('utf-8')
self.outfile.write(line + '\n')
self.outfile.seek(0)
self.errfile = tempfile.TemporaryFile()
for line in self.err:
if isinstance(line, unicode): line = line.encode('utf-8')
self.errfile.write(line + '\n')
self.errfile.seek(0)
self.returncode = rc
Sorry for the wall of text, but I went for completeness. This is part of a larger application.
This code works perfectly connecting to Linux, so I don't expect there to be many little bugs. The returncode is always 0, even for garbage, and there is never any output on the pipes. If I run the command just using the terminal, I get the correct output:
$ ssh testuser#testwin.internal.com 'java -version'
Warning: Permanently added 'testwin.internal.com,10.10.10.12' (ECDSA) to the
list of known hosts.
testuser#testwin.internal.com's password:
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)
$ echo $?
0
$ ssh testuser#testwin.internal.com 'foo'
foo : The term 'foo' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path
is correct and try again.
At line:1 char:1
+ foo
+ ~~~
+ CategoryInfo : ObjectNotFound: (foo:String) [], CommandNotFoundException
+ FullyQualifiedErrorId : CommandNotFoundException
$ echo $?
1
The only difference between our Linux and Windows processes that I can think of is that on Windows we have to use a password, as we haven't setup passwordless ssh yet. What weird Windows idiosyncrasy am I missing? Any insights would be greatly appreciated.
As usual, it turns out to be way simpler than I was thinking.
string = ""
for arg in command:
string = string + " " + arg
The above code produces " whatever command was passed in" and it turns out that Windows reacts extremely poorly to the preceding space, while linux didn't care. The new code snippet is:
string = ""
first = True
for arg in command:
if first:
string = string + arg
first = False
else:
string = string + " " + arg
I'm leaving the title and details the same, to hopefully help anyone who makes the exact same error I did.
I'm attempting to execute a command over SSH, but bash on the other end doesn't think it's escaped properly.
Here, self._client is a paramiko.SSHClient object; args is a list of arguments, the command to execute.
def run(self, args, stdin=None, capture_stdout=False):
"""Runs a command.
On success, returns the output, if requested, or None.
On failure, raises CommandError, with stderr and, if captured, stdout,
as well as the exit code.
"""
command = ' '.join(_shell_escape(arg) for arg in args)
print('About to run command:\n {}'.format(command))
print('About to run command:\n {!r}'.format(command))
channel = self._client.get_transport().open_session()
channel.exec_command(command)
_shell_escape:
_SHELL_SAFE = _re.compile(r'^[-A-Za-z0-9_./]+$')
def _shell_escape(s):
if _SHELL_SAFE.match(s):
return s
return '\'{}\''.format(s.replace('\'', '\'\\\'\''))
I'm attempt to run some Python through this. On stderr, I get back:
bash: -c: line 5: unexpected EOF while looking for matching `''
bash: -c: line 6: syntax error: unexpected end of file
The output from the two print statements:
About to run command:
python -c 'import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
' .
About to run command:
"python -c 'import os, sys\npath = sys.argv[1]\nif sys.version_info.major == 2:\n path = path.decode('\\''utf-8'\\'')\nentries = os.listdir(path)\nout = b'\\'''\\''.join(e.encode('\\''utf-8'\\'') + b'\\''\x00'\\'' for e in entries)\nsys.stdout.write(out)\n' ."
If I copy and paste the output of command, and paste it into bash, it executes, so it really does appear to be properly escaped. My current understanding is that SSH, on the other end, will take command, and run [my_shell, '-c', command].
Why is bash erroring on that command?
The input contains an embedded nul character, which bash appears to treat as the end of the string. (I'm not sure there's any way it couldn't!). This is visible in my question, where I output command:
About to run command:
"python -c 'import os, sys [SNIP…] + b'\\''\x00'\\'' for [SNIP…]"
That's a repr output, but notice the single slash before the x in \x00: that's an actual \x00 that made it through. My original code has this Python embedded as a snippet, which I didn't include (I didn't believe it was relevant):
_LS_CODE = """\
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('utf-8')
entries = os.listdir(path)
out = b''.join(e.encode('utf-8') + b'\x00' for e in entries)
sys.stdout.write(out)
"""
Here, Python's """ is still processing \ as an escape character. I need to double up, or look into raw strings (r""")
You need to escape newlines as well. A better option is to put the program text in a here document.
Make the output of "About to run command:" to look like
python -c << EOF
import os, sys
path = sys.argv[1]
if sys.version_info.major == 2:
path = path.decode('\''utf-8'\'')
entries = os.listdir(path)
out = b'\'''\''.join(e.encode('\''utf-8'\'') + b'\'''\'' for e in entries)
sys.stdout.write(out)
.
EOF
Maybe you wouldn't need to escape anything at all.
To begin with, I don't know the first thing about Python ... so I can really use any pointers you have. I do know some Perl, Bash scripting and a bit C++.
I'm running DenyHosts (http://denyhosts.sourceforge.net/) which every now and then sends me a message through email that an IP address was added to /etc/deny.hosts. Eg.:
Added the following hosts to /etc/hosts.deny:
87.215.133.109 (unknown)
----------------------------------------------------------------------
So far so good, but I want to add the country of the IP address to this message. To do this I created a small Perl script that spits out the country:
/usr/local/bin/geo-ip.pl --short 87.215.133.109
Netherlands
So all I want to do is to call this Perl script from Python and then fill the result in the message string. I located the source code which I suspect I need to change, but as announced at the top of this message, I don't know the first thing about Python.
This is a sniplet from the main program calling a subroutine in report.py
deny_hosts.py:
#print deny_hosts
new_denied_hosts, status = self.update_hosts_deny(deny_hosts)
if new_denied_hosts:
if not status:
msg = "WARNING: Could not add the following hosts to %s" % self.__prefs.get('HOSTS_DENY')
else:
msg = "Added the following hosts to %s" % self.__prefs.get('HOSTS_DENY')
self.__report.add_section(msg, new_denied_hosts)
if self.__sync_server: self.sync_add_hosts(new_denied_hosts)
plugin_deny = self.__prefs.get('PLUGIN_DENY')
if plugin_deny: plugin.execute(plugin_deny, new_denied_hosts)
I think the change should go somewhere in here.
report.py defines the add_section:
def add_section(self, message, iterable):
self.report += "%s:\n\n" % message
for i in iterable:
if type(i) in (TupleType, ListType):
extra = ": %d\n" % i[1]
i = i[0]
else:
extra = ""
if self.hostname_lookup:
hostname = self.get_hostname(i)
debug("get_host: %s", hostname)
else: hostname = i
self.report += "%s%s\n" % (hostname, extra)
if self.use_syslog:
syslog.syslog("%s - %s%s" %(message, hostname, extra))
self.report += "\n" + "-" * 70 + "\n"
Please help me change the code in such a way that it'll spit out a message like:
Added the following hosts to /etc/hosts.deny:
87.215.133.109 (Netherlands, unknown)
----------------------------------------------------------------------
EDIT3:
This is how I solved it. The output is identical to the original message. After changing the sources, the daemon needs to be restarted (sudo /etc/init.d/denyhosts restart)
def add_section(self, message, iterable):
# added geo-ip
# moving this from statement to the top of the file makes pycheck generate
# a lot of errors, so I left it here.
from subprocess import Popen, PIPE
# end geo-ip hack import
self.report += "%s:\n\n" % message
for i in iterable:
if type(i) in (TupleType, ListType):
extra = ": %d\n" % i[1]
i = i[0]
else:
extra = ""
if self.hostname_lookup:
hostname = self.get_hostname(i)
debug("get_host: %s", hostname)
else: hostname = i
# self.report += "%s%s\n" % (hostname, extra)
# JPH: added geo-ip
geocmd = "/usr/local/bin/geo-ip.pl --short %s" % i
country = Popen( geocmd, shell=True, stdout=PIPE).communicate()[0]
country = country.strip()
self.report += "%s%s\n%s\n" % (hostname, extra, country)
# end geo-ip hack
if self.use_syslog:
syslog.syslog("%s - %s%s" %(message, hostname, extra))
self.report += "\n" + "-" * 70 + "\n"
Also help me understand what I change, so I can learn a bit Python today too.
EDIT2: For the sake of sharing a link to the geo-ip.pl script http://wirespeed.xs4all.nl/mediawiki/index.php/Geo-ip.pl
EDIT1: Recompilation is done automatically when the source changes, so that answers the question below.
The second problem I have with this is that I found two matching files on my system:
/usr/share/denyhosts/DenyHosts/report.py
/usr/share/denyhosts/DenyHosts/report.pyc
where the .py is the source code and I suspect .pyc actually being executed. So when I change the source code, I wouldn't be surprised nothing changes if I don't somehow compile it afterwards.
I'm only going to answer the specific part of your question about how to call your perl script via python and get the output. The part about where to slot in this info is a little too vague for me to guess from your snippets...
from subprocess import Popen, PIPE
hostIP = "87.215.133.109"
cmd = "/usr/local/bin/geo-ip.pl --short %s" % hostIP
output = Popen(cmd, shell=True, stdout=PIPE).communicate()[0]
## alternate form ##
# cmd = ["/usr/local/bin/geo-ip.pl, "--short", hostIP]
# output = Popen(cmd, stdout=PIPE).communicate()[0]
print output.strip()
# Netherlands
Update
Since I am doing a few things at once on that Popen line, and you are new to python (based on your comments below), I wanted to break down that line a bit for you...
# this call to Popen actually returns a
# Popen object with a number of methods and attributes
# to interact with the process that was just created
p = Popen(cmd, shell=True, stdout=PIPE)
# communicate() is a method of a Popen object which
# allows you to wait for the return output of the pipes
# that you named (or send data to stdin)
# It blocks until data is ready and returns a tuple (stdout, stderr)
stdout, stderr = p.communicate()
# We only wanted the stdout in this case, so we took the first index
output = p.communicate()[0]
# output is a string, and strings have the strip() method to remove
# surrounding whitespace
stripped_output = output.strip()
This could do the trick:
import subprocess
country = subprocess.check_output(
["/usr/local/bin/geo-ip.pl", "--short", "87.215.133.109"])
I have found an example python script and have modified it to do checks for comments in pre-commit. My problem is that the text of the comment as parsed by python always ends up blank.
Env: Windows XP
SVN version: svn, version 1.5.6 (r36142) compiled Mar 6 2009, 14:54:47
Python 2.7
pre-commit.bat
C:\Python27\python %1\hooks\pre-commit.py %1 %2
pre-commit.py
import sys, os, string, re
SVNLOOK='C:\\SVN\\bin\\svnlook.exe'
# return true or false if this passed string is a valid comment
def check(comment):
#define regular expression
print comment
p = re.match("[bB][uU][gG]\s([0-9]+|NONE)+", comment)
print p
return (p != None) #returns false if doesn't match
def result(r, txn, repos, log_msg, log_cmd):
if r == 1:
sys.stderr.write ("File: " + repos + " Comment: " + txn + "\n" +\
"Log Msg: " + log_msg + "\n" +\
"Log Cmd: " + log_cmd + "\n" +\
"Comments must have the format of \n'Bug X' \n" +\
"'Comment text' where X is the issue number.\n" +\
"Use comma to separatemultiple bug id's\n" +\
"Example:\nBug 1234, 1235\nComment: Fixed things\n" +\
"Use 'NONE' if there is no bug ID")
sys.exit(r)
def main(repos, txn):
log_cmd = '%s log %s -t %s' % (SVNLOOK, txn, repos)
log_msg = os.popen(log_cmd, 'r').readline().rstrip('\n')
if check(log_msg):
result(0, txn, repos, log_msg, log_cmd)
else:
result(1, txn, repos, log_msg, log_cmd)
if __name__ == '__main__':
if len(sys.argv) < 3:
sys.stderr.write("Usage: %s REPOS TXN\n" % (sys.argv[0]))
else:
main(sys.argv[1], sys.argv[2])
I Added print outs of the vars into the error message for debugging purposes.
What is anoying is that if I use bat file commands, things seem to work fine:
pre-commit.bat that only checks for blank commit msg:
#echo off
:: Stops commits that have empty log messages.
#echo off
setlocal
rem Subversion sends through the path to the repository and transaction id
set REPOS=%1
set TXN=%2
rem line below ensures at least one character ".", 5 characters require change to "....."
C:\SVN\bin\svnlook.exe log %REPOS% -t %TXN% | findstr . > nul
if %errorlevel% gtr 0 (goto err) else exit 0
:err
echo. 1>&2
echo Your commit has been blocked because you didn't enter a comment. 1>&2
echo Write a log message describing the changes made and try again. 1>&2
echo Thanks 1>&2
exit 1
what am I doing wrong?
When you run it through Python, the error code which is set by Python is ignored in the batch file; you will want to do it something like this:
C:\Python27\python %1\hooks\pre-commit.py %1 %2
exit %ERRORLEVEL%
It's a good idea to quote your vaules, as well, to make sure that they get through to Python intact:
C:\Python27\python "%1\hooks\pre-commit.py" "%1" "%2"
exit %ERRORLEVEL%