We Keep Coding

Command not executed on if statement being true

So I'm writing a simple program in Python that checks active window ID and in this program I have an if statement inside a while True loop that checks if the active window ID matches the expected window ID.
Here's the code:
import os, subprocess
def get_window_id():
output = subprocess.check_output("xprop -root | grep \"window id\"", shell=True)
get_window_id.result = {}
for row in output.split(b"\n"):
if b": " in row:
key, value = row.split(b": ")
get_window_id.result[key.strip(b"window id # ")] = value.strip()
while True:
get_window_id()
print(get_window_id.result[b"_NET_ACTIVE_WINDOW(WINDOW)"].strip(b"b'window id # "))
if get_window_id.result[b"_NET_ACTIVE_WINDOW(WINDOW)"].strip(b"b'window id # ") != "b'0x3000003'":
os.system("kill -STOP $(pgrep mpv)")
else:
os.system("kill -CONT $(pgrep mpv)")
When the active window ID doesn't match the expected one, the block of code that corresponds to that condition executes, but when the active window ID matches the expected one, the block of code that corresponds to that condition doesn't execute. I obviously want that block of code to execute.
As you can see, I'm not the best at Python and asking for help but I hope that this is enough for somebody to help. If not I can always provide more information :)
And sorry if the title and/or body of the question are not clear. It's currently 2AM and I've been trying to do this all day.
Thanks in advance.

YES! I did it! Just needed some sleep and a little bit more searching, but basically what I needed to do is get rid of those bs before the strings and to do that I needed to add encoding=utf8 to the subprocess.check_output line and delete the bs and it worked!
Code after the change:
import os, subprocess
def get_window_id():
output = subprocess.check_output("xprop -root | grep \"window id\"", shell=True, encoding="utf8")
get_window_id.result = {}
for row in output.split("\n"):
if ": " in row:
key, value = row.split(": ")
get_window_id.result[key.strip("window id # ")] = value.strip()
while True:
get_window_id()
print(get_window_id.result["_NET_ACTIVE_WINDOW(WINDOW)"].strip("'window id # "))
if get_window_id.result["_NET_ACTIVE_WINDOW(WINDOW)"].strip("'window id # ") != "0x3000003":
os.system("kill -STOP $(pgrep mpv)")
else:
os.system("kill -CONT $(pgrep mpv)")

RuntimeError dictionary changed size during iteration during dictionary iteration

I am getting following error when I run my code. It started happening with apparently no changes to the code. The code basically tries to rendor the device config using jinja2 template.
Traceback (most recent call last):
File "vdn_ler_generate_config.py", line 380, in <module>
lerConfig = lerTemplate.render(config=copy.copy(config.lerDevList[tLer]))
File "/Users/nileshkhambal/Documents/myansible/lib/python3.8/site-packages/jinja2/environment.py", line 1304, in render
self.environment.handle_exception()
File "/Users/nileshkhambal/Documents/myansible/lib/python3.8/site-packages/jinja2/environment.py", line 925, in handle_exception
raise rewrite_traceback_stack(source=source)
File "./templates/jinja2/JuniperLerConfigTemplate", line 71, in top-level template code
{%- for tBgpGrp in config.vrfs[tvrf].bgpProto %}
RuntimeError: dictionary changed size during iteration
The code snippet below inside "try" and "expect" block shows the problem. There is an exactly same code for second vendor and it works fine. Using print statement in expect statement I see that there is an extra group object (with 'default') added in the dictionary during the iteration. According to my YAML config there should be only one such group object that should be there.
try:
if tVendor == 'VENDOR1':
lerTemplate = ENV.get_template('Vendor1LerConfigTemplate')
lerConfig = lerTemplate.render(config=config.lerDevList[tLer])
pOutputFile = './configs/' + str(tLer) + '.cfg'
with open(pOutputFile, "w+") as opHandle:
#print('Writing configuration for {}'.format(tLer))
opHandle.write(lerConfig)
opHandle.write('\n')
except:
for aKey in config.lerDevList[tLer].vrfs.keys():
#print(config.lerDevList[tLer].vrfs[aKey].bgpProto)
print('What VRF: {} How many BGP Groups: {}'.format(aKey,len(config.lerDevList[tLer].vrfs[aKey].bgpProto.keys())))
for agrp in config.lerDevList[tLer].vrfs[aKey].bgpProto.keys():
print(config.lerDevList[tLer].vrfs[aKey].bgpProto[agrp])
continue
if tVendor == 'VENDOR2':
for aKey in config.lerDevList[tLer].vrfs.keys():
#print(config.lerDevList[tLer].vrfs[aKey].bgpProto)
for agrp in config.lerDevList[tLer].vrfs[aKey].bgpProto.keys():
print(config.lerDevList[tLer].vrfs[aKey].bgpProto[agrp])
lerTemplate = ENV.get_template('Vendor2LerConfigTemplate')
lerConfig = lerTemplate.render(config=config.lerDevList[tLer])
pOutputFile = './configs/' + str(tLer) + '.cfg'
with open(pOutputFile, "w+") as opHandle:
#print('Writing configuration for {}'.format(tLer))
opHandle.write(lerConfig)
opHandle.write('\n')
using some print() statements I can see that, the code creating group using base class object, adds one group but iteration code seems to add or see an extra group with name 'default'. 'default' is the name used in the base class for the group. once the object is instantiated it is assigned a proper group name.
Creating vrf bgp group object for xxx4-bb-pe1 BLUE: AS65YYY-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for yyy6-bb-pe1 RED: AS65YYY-RED-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for zzz2-bb-pe1 BLUE: AS4200XXXXXX-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for zzz2-bb-pe2 RED: AS4200XXXXXX-RED-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for xyxy2-bb-gw1 BLUE: AS4200XXXXXX-BLUE-V4-PEER
Double checking bgp groups: 1
Creating vrf bgp group object for xyxy2-bb-gw2 RED: AS4200XXXXXX-RED-V4-PEER
Double checking bgp groups: 1
Writing configuration for xxx4-bb-pe1
AS65YYY-BLUE-V4-PEER
Writing configuration for yyy6-bb-pe1
AS65YYY-RED-V4-PEER
Writing configuration for zzz2-bb-pe1
AS4200XXXXXX-BLUE-V4-PEER
Writing configuration for zzz2-bb-pe2
AS4200XXXXXX-RED-V4-PEER
Writing configuration for xyxy2-bb-gw1
What VRF: BLUE How many BGP Groups: 2 <<< extra group
AS4200XXXXXX-BLUE-V4-PEER
default << extra group
Writing configuration for xyxy2-bb-gw2
What VRF: RED How many BGP Groups: 2 <<< extra group
AS4200XXXXXX-RED-V4-PEER
default <<<< extra group
here is the default group class definition
class bgpGroup():
def __init__(self):
self.vrf = ''
self.grpName = 'default'
self.grpType = 'internal'
self.grpDescr = ''
self.grpLocalAddr = '0.0.0.0'
self.clusterid = ''
self.gr_disable = False
self.remove_private = False
self.as_override = False
self.peer_as = ''
self.local_as = ''
self.local_as_private = False
self.local_as_noprepend = False
self.holdtime = ''
self.grpFamily = []
self.grpImport = ''
self.grpExport = ''
self.grpNbrList = defaultdict(bgpNbr)
self.grpLoopDetect = False
self.grpMltHopTtl = 0
self.grpInetAsPathPrependReceive = False
self.grpLabelInet6AsPathPrependReceive = False
def __repr__(self):
return self.grpName

I really don't know what the bug is, but have you tried:
for aKey in list(config.lerDevList[tLer].vrfs.keys())
This grabs all the keys from the dictionary at the start, and if another entry is added, it won't matter.
Don't also that you don't need .keys(), though it doesn't hurt. Iterating through a dictionary is iterating through its keys.

Issue was in a jinja2 template code. A wrong variable was referenced inside a statement in for loop. Fixing it fixed the error. Sorry for the noise. Error did point to jinja2 code with the start of the loop line that had error. Did not point to the actual line with an error. All good now.

Behave - Testing using blank Example fields

I am using Behave to automate the testing of a config file, as part of this test I need to populate various fields in the config file with invalid and blank fields. Where I am entering values I can do this using a Scenario Outline entering the values in the Examples. However when I try entering a blank field using this method Behave does not like the fact there is no value.
Is there an easy way to pass a blank value from the Examples file, or will I need to test these conditions using a separate behave test
feature
Scenario Outline:Misconfigured Identity Listener
Given an already stopped Identity Listener
And parameter <parameter> is configured to value <config_value>
When the Identity Listener is started
Then the identity listener process is not present on the system
And the log contains a <message> showing that the parameter is not configured
Examples: Protocols
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | |
|cache_ip_address | cache_ip_address | 123.123.12 |
the step where I define the config value
#given('parameter {parameter} is configured to value {config_value}')
def step_impl(context, parameter, config_value):
context.parameter = parameter
context.config_value = config_value
context.identity_listener.update_config(parameter, config_value)
changing the config file using sed -i (I am interacting with a linux box in this test)
def update_config(self, param, config_value):
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= "+ config_value + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)
Thanks to answer from #Verv i got this working solution below
passed an empty value in for fields where I don't want a value passed
|parameter |message |config_value|
|cache_ip_address | cache_ip_address | empty |
Added an if else statement into my update config step
def update_config(self, param, config_value):
if config_value == "empty":
il_config = ""
else:
il_config = config_value
command = 'sudo sh -c "sed -i'
command = command + " '/" + param + "/c\\" + param + "= " + il_config + " \\' {0}\""
command = command.format(self.config_file)
self.il_ssh.runcmd(command)

You could put something like empty in the field, and tweak your method so that whenever the field's value is empty, you treat it as an actual empty string (i.e. "")

Python Converting tab limited file into csv

I basically want to convert tab delimited text file http://www.linux-usb.org/usb.ids into a csv file.
I tried importing using Excel, but it is not optimal, it turns out like:
8087 Intel Corp.
0020 Integrated Rate Matching Hub
0024 Integrated Rate Matching Hub
How I want it so for easy searching is:
8087 Intel Corp. 0020 Integrated Rate Matching Hub
8087 Intel Corp. 0024 Integrated Rate Matching Hub
Is there any ways I can do this in python?

$ListDirectory = "C:\USB_List.csv"
Invoke-WebRequest 'http://www.linux-usb.org/usb.ids' -OutFile $ListDirectory
$pageContents = Get-Content $ListDirectory | Select-Object -Skip 22
"vendor`tvendor_name`tproduct`tproduct_name`r" > $ListDirectory
#Variables and Flags
$currentVid
$currentVName
$currentPid
$currentPName
$vendorDone = $TRUE
$interfaceFlag = $FALSE
$nextline
$tab = "`t"
foreach($line in $pageContents){
if($line.StartsWith("`#")){
continue
}
elseif($line.length -eq 0){
exit
}
if(!($line.StartsWith($tab)) -and ($vendorDone -eq $TRUE)){
$vendorDone = $FALSE
}
if(!($line.StartsWith($tab)) -and ($vendorDone -eq $FALSE)){
$pos = $line.IndexOf(" ")
$currentVid = $line.Substring(0, $pos)
$currentVName = $line.Substring($pos+2)
"$currentVid`t$currentVName`t`t`r" >> $ListDirectory
$vendorDone = $TRUE
}
elseif ($line.StartsWith($tab)){
if ($interfaceFlag -eq $TRUE){
$interfaceFlag = $FALSE
}
$nextline = $line.TrimStart()
if ($nextline.StartsWith($tab)){
$interfaceFlag = $TRUE
}
if ($interfaceFlag -eq $FALSE){
$pos = $nextline.IndexOf(" ")
$currentPid = $nextline.Substring(0, $pos)
$currentPName = $nextline.Substring($pos+2)
"$currentVid`t$currentVName`t$currentPid`t$currentPName`r" >> $ListDirectory
Write-Host "$currentVid`t$currentVName`t$currentPid`t$currentPName`r"
$interfaceFlag = $FALSE
}
}
}
I know the ask is for python, but I built this PowerShell script to do the job. It takes no parameters. Just run as admin from the directory where you want to store the script. The script collects everything from the http://www.linux-usb.org/usb.ids page, parses the data and writes it to a tab delimited file. You can then open the file in excel as a tab delimited file. Ensure the columns are read as "text" and not "general" and you're go to go. :)
Parsing this page is tricky because the script has to be contextually aware of every VID-Vendor line proceeding a series of PID-Product lines. I also forced the script to ignore the commented description section, the interface-interface_name lines, the random comments that he inserted throughout the USB list (sigh) and everything after and including "#List of known device classes, subclasses and protocols" which is out of scope for this request.
I hope this helps!

You just need to write a little program that scans in the data a line at a time. Then it should check to see if the first character is a tab ('\t'). If not then that value should be stored. If it does start with tab then print out the value that was previously stored followed by the current line. The result will be the list in the format you want.

Something like this would work:
import csv
lines = []
with open("usb.ids.txt") as f:
reader = csv.reader(f, delimiter="\t")
device = ""
for line in reader:
# Ignore empty lines and comments
if len(line) == 0 or (len(line[0]) > 0 and line[0][0] == "#"):
continue
if line[0] != "":
device = line[0]
elif line[1] != "":
lines.append((device, line[1]))
print(lines)
You basically need to loop through each line, and if it's a device line, remember that for the following lines. This will only work for two columns, and you would then need to write them all to a csv file but that's easy enough

How to parse nagios status.dat file?

I'd like to parse status.dat file for nagios3 and output as xml with a python script.
The xml part is the easy one but how do I go about parsing the file? Use multi line regex?
It's possible the file will be large as many hosts and services are monitored, will loading the whole file in memory be wise?
I only need to extract services that have critical state and host they belong to.
Any help and pointing in the right direction will be highly appreciated.
LE Here's how the file looks:
########################################
# NAGIOS STATUS FILE
#
# THIS FILE IS AUTOMATICALLY GENERATED
# BY NAGIOS. DO NOT MODIFY THIS FILE!
########################################
info {
created=1233491098
version=2.11
}
program {
modified_host_attributes=0
modified_service_attributes=0
nagios_pid=15015
daemon_mode=1
program_start=1233490393
last_command_check=0
last_log_rotation=0
enable_notifications=1
active_service_checks_enabled=1
passive_service_checks_enabled=1
active_host_checks_enabled=1
passive_host_checks_enabled=1
enable_event_handlers=1
obsess_over_services=0
obsess_over_hosts=0
check_service_freshness=1
check_host_freshness=0
enable_flap_detection=0
enable_failure_prediction=1
process_performance_data=0
global_host_event_handler=
global_service_event_handler=
total_external_command_buffer_slots=4096
used_external_command_buffer_slots=0
high_external_command_buffer_slots=0
total_check_result_buffer_slots=4096
used_check_result_buffer_slots=0
high_check_result_buffer_slots=2
}
host {
host_name=localhost
modified_attributes=0
check_command=check-host-alive
event_handler=
has_been_checked=1
should_be_scheduled=0
check_execution_time=0.019
check_latency=0.000
check_type=0
current_state=0
last_hard_state=0
plugin_output=PING OK - Packet loss = 0%, RTA = 3.57 ms
performance_data=
last_check=1233490883
next_check=0
current_attempt=1
max_attempts=10
state_type=1
last_state_change=1233489475
last_hard_state_change=1233489475
last_time_up=1233490883
last_time_down=0
last_time_unreachable=0
last_notification=0
next_notification=0
no_more_notifications=0
current_notification_number=0
notifications_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
flap_detection_enabled=1
failure_prediction_enabled=1
process_performance_data=1
obsess_over_host=1
last_update=1233491098
is_flapping=0
percent_state_change=0.00
scheduled_downtime_depth=0
}
service {
host_name=gateway
service_description=PING
modified_attributes=0
check_command=check_ping!100.0,20%!500.0,60%
event_handler=
has_been_checked=1
should_be_scheduled=1
check_execution_time=4.017
check_latency=0.210
check_type=0
current_state=0
last_hard_state=0
current_attempt=1
max_attempts=4
state_type=1
last_state_change=1233489432
last_hard_state_change=1233489432
last_time_ok=1233491078
last_time_warning=0
last_time_unknown=0
last_time_critical=0
plugin_output=PING OK - Packet loss = 0%, RTA = 2.98 ms
performance_data=
last_check=1233491078
next_check=1233491378
current_notification_number=0
last_notification=0
next_notification=0
no_more_notifications=0
notifications_enabled=1
active_checks_enabled=1
passive_checks_enabled=1
event_handler_enabled=1
problem_has_been_acknowledged=0
acknowledgement_type=0
flap_detection_enabled=1
failure_prediction_enabled=1
process_performance_data=1
obsess_over_service=1
last_update=1233491098
is_flapping=0
percent_state_change=0.00
scheduled_downtime_depth=0
}
It can have any number of hosts and a host can have any number of services.

Pfft, get yerself mk_livestatus. http://mathias-kettner.de/checkmk_livestatus.html

Nagiosity does exactly what you want:
http://code.google.com/p/nagiosity/

Having shamelessly stolen from the above examples,
Here's a version build for Python 2.4 that returns a dict containing arrays of nagios sections.
def parseConf(source):
conf = {}
patID=re.compile(r"(?:\s*define)?\s*(\w+)\s+{")
patAttr=re.compile(r"\s*(\w+)(?:=|\s+)(.*)")
patEndID=re.compile(r"\s*}")
for line in source.splitlines():
line=line.strip()
matchID = patID.match(line)
matchAttr = patAttr.match(line)
matchEndID = patEndID.match( line)
if len(line) == 0 or line[0]=='#':
pass
elif matchID:
identifier = matchID.group(1)
cur = [identifier, {}]
elif matchAttr:
attribute = matchAttr.group(1)
value = matchAttr.group(2).strip()
cur[1][attribute] = value
elif matchEndID and cur:
conf.setdefault(cur[0],[]).append(cur[1])
del cur
return conf
To get all Names your Host which have contactgroups beginning with 'devops':
nagcfg=parseConf(stringcontaingcompleteconfig)
hostlist=[host['host_name'] for host in nagcfg['host']
if host['contact_groups'].startswith('devops')]

Don't know nagios and its config file, but the structure seems pretty simple:
# comment
identifier {
attribute=
attribute=value
}
which can simply be translated to
<identifier>
<attribute name="attribute-name">attribute-value</attribute>
</identifier>
all contained inside a root-level <nagios> tag.
I don't see line breaks in the values. Does nagios have multi-line values?
You need to take care of equal signs within attribute values, so set your regex to non-greedy.

You can do something like this:
def parseConf(filename):
conf = []
with open(filename, 'r') as f:
for i in f.readlines():
if i[0] == '#': continue
matchID = re.search(r"([\w]+) {", i)
matchAttr = re.search(r"[ ]*([\w]+)=([\w\d]*)", i)
matchEndID = re.search(r"[ ]*}", i)
if matchID:
identifier = matchID.group(1)
cur = [identifier, {}]
elif matchAttr:
attribute = matchAttr.group(1)
value = matchAttr.group(2)
cur[1][attribute] = value
elif matchEndID:
conf.append(cur)
return conf
def conf2xml(filename):
conf = parseConf(filename)
xml = ''
for ID in conf:
xml += '<%s>\n' % ID[0]
for attr in ID[1]:
xml += '\t<attribute name="%s">%s</attribute>\n' % \
(attr, ID[1][attr])
xml += '</%s>\n' % ID[0]
return xml
Then try to do:
print conf2xml('conf.dat')

If you slightly tweak Andrea's solution you can use that code to parse both the status.dat as well as the objects.cache
def parseConf(source):
conf = []
for line in source.splitlines():
line=line.strip()
matchID = re.match(r"(?:\s*define)?\s*(\w+)\s+{", line)
matchAttr = re.match(r"\s*(\w+)(?:=|\s+)(.*)", line)
matchEndID = re.match(r"\s*}", line)
if len(line) == 0 or line[0]=='#':
pass
elif matchID:
identifier = matchID.group(1)
cur = [identifier, {}]
elif matchAttr:
attribute = matchAttr.group(1)
value = matchAttr.group(2).strip()
cur[1][attribute] = value
elif matchEndID and cur:
conf.append(cur)
del cur
return conf
It is a little puzzling why nagios chose to use two different formats for these files, but once you've parsed them both into some usable python objects you can do quite a bit of magic through the external command file.
If anybody has a solution for getting this into a a real xml dom that'd be awesome.

For the last several months I've written and released a tool that that parses the Nagios status.dat and objects.cache and builds a model that allows for some really useful manipulation of Nagios data. We use it to drive an internal operations dashboard that is a simplified 'mini' Nagios. Its under continual development and I've neglected testing and documentation but the code isn't too crazy and I feel fairly easy to follow.
Let me know what you think...
https://github.com/zebpalmer/NagParser