I'm trying to automate some boring monkey jobs for fortigate firewalls. I receive requests to create address objects for example for 100 different hosts.. many of which should belong to same address group.
My csv is
name,address,group
aix01,10.0.0.1,AIXGROUP1
aix02,10.0.0.2,AIXGROUP1
aix02,10.0.0.3,AIXGROUP2
aix245,10.0.0.4,AIXGROUP2
As you see above, there are 2x groups with 2x hosts in each.
I want to take the group names "AIXGROUP1" and "AIXGROUP2" as a dictionary keys and take list of each of the respective IPs as it's values.
So, it should be {AIXGROUP1:[10.0.0.1,10.0.0.2], AIXGROUP2:[10.0.0.3,10.0.0.4]}
I would then pass this dictionary to a function that prints the commands.
Here's the code
with open('hosts.csv', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter = ',')
Dictionary = {}
for each in reader:
if len(each[2])!=0:
Dictionary[each[2]] = []
at this point I just have a dictionary with correct keys and empty lists as values.. and here's where I'm stuck.. how do I populate these empty lists with ip addresses?
I feel like I'm inch away from the victory! :)
You are close to a solution. The only thing you need to add is a check if the key already exist in the dictionary, so you don't overwrite it and then add a row that adds the value to the list. And you can add the row next(reader, None) if you don't want the header row in the dictionary. Something like this should work:
with open('hosts.csv', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter = ',')
next(reader, None) # To skip the header row
Dictionary = {}
for each in reader:
if len(each[2])!=0 and each[2] not in Dictionary.keys(): # To not overwrite keys that already exist
Dictionary[each[2]] = []
Dictionary[each[2]].append(each[1]) #To add the values
Your loop needs to take different actions depending on whether the group name is already in the dictionary or not, like this (I'll use d instead of Dictionary for brevity):
group = each[2]
ip = each[1]
if group in d:
d[group].append(ip) # Append another IP to the list
else:
d[group] = [ip] # Initialize list with first element
I would do it like this:
result = {}
for row in reader:
if len(row[2])!=0:
try:
result[row[2]].append(row[1])
except:
result[row[2]] = [row[1]]
result
>>> {'AIXGROUP1': ['10.0.0.1', '10.0.0.2'], 'AIXGROUP2': ['10.0.0.3', '10.0.0.4']}
This approach creates the lists and fills them in the same loop. The try / except logic is faster than a conditional and works like this:
For each row of data, try to append the address to the right group.
If the exception is raised, that key doesn't yet exist in the dictionary, so we have to create it.
On creating the key, we define the key as a list with one element in it, which is the current address.
If the exception isn't raised, the dictionary already has one address for that key, so we just append the next one to the list referenced by that key.
Note that it's probably not best practice to name your output dict Dictionary. Also, if you are going to use csv.read(), don't forget to skip the header row!
import csv
# create the dictionary in which your data will be stored
ip_dict = {}
# open the wanted file
with open('hosts.csv', 'r') as csvfile:
# read the csv data
csv_reader = csv.reader(csvfile, delimiter=',')
# skip the header
next(csv_reader)
# parse your data
for _, ip, name in csv_reader:
# do whatever you need only if name is not empty
if name:
# if there is already an entry in the dictionary then append to it
if name in ip_dict:
ip_dict[name].append(ip)
# if there is not create a list with the current ip as the first entry
else:
ip_dict[name] = [ip]
I Think my final version will be this.
def parser():
Address_Groups = {}
Addresses = {}
with open ('Data.csv','r') as file:
reader = csv.reader(file, delimiter = ',')
next(reader)
for row in reader:
if row [0] not in Addresses:
Addresses[row[0]] = row[1]
if row[2] not in Address_Groups:
Address_Groups[row[2]] = []
if row[2] in row:
Address_Groups[f'{row[2]}'].append(f'{row[0]}')
return (Addresses,Address_Groups)
Like this I successfully map some server name to it's respective IP address and name to respective address group.
when I do return (Objects,Address_Groups)it returns a tuple of two dictionaries, right?
so if I do print(parser())
What I'm getting is a tuple(?) of two dictionaries.. is that right?
({'Host-01': '192.168.1.1', 'Host-02': '192.168.1.2', 'Host-03': '192.168.1.3', 'Host-04': '192.168.1.4'}, {'Test_Group-01': ['Host-01', 'Host-02'], 'Test_Group-02': ['Host-03'], 'Test_Group-012': ['Host-04']})
You can use defaultdict from the collections python library. It was created exactly for your case.
ar = [
'aix01,10.0.0.1,AIXGROUP1',
'aix02,10.0.0.2,AIXGROUP1',
'aix02,10.0.0.3,AIXGROUP2',
'aix245,10.0.0.4,AIXGROUP2'
]
from collections import defaultdict
d = defaultdict(list)
for row in ar:
elements = row.split(',')
d[elements[2]].append(elements[1])
print(dict(d))
This will print
{'AIXGROUP1': ['10.0.0.1', '10.0.0.2'], 'AIXGROUP2': ['10.0.0.3',
'10.0.0.4']}
So.. to make it more clear I will explain this from A to Z
I'm a network engineer who works primarily with fortigate firewalls.
Sometimes I receive the requests to add lots of address objects on the firewall.
Often times these objects should belong to address groups and later I use these address objects and address groups in firewall policies.
My objective was to create a short script that would write the firewall CLI commands in the file
so initially I structured the csv file in this way. Script should go through this and write firewall commands into a text file.
name,address,group
aix01,10.0.0.1,AIXGROUP1
aix02,10.0.0.2,AIXGROUP1
aix03,10.0.0.3,AIXGROUP2
aix04,10.0.0.4,AIXGROUP2
firewall cli command for adding object
config firewall address
edit <Address Object Name>
set subnet <IP> <Mask>
set comment <Comment>
next
end
and for adding hosts into address group
config firewall addrgrp
edit <Address Group Name>
set member "<Address Object>" "<Address Object>"
set comment <comment>
next
end
For My initial question I used very simple csv and question was how to build the dictionary from the csv file specifically for the address group mapping because how I see this working is to have group name as a KEY and list of group members as VALUES. so in the initial example desired result was
{
AIXGROUP1:[aix01,aix02],
AIXGROUP2:[aix03,aix04]
}
I was stuck at this part - I was getting correct keys but was unable to populate the value lists with correct IP addresses/hostnames.
I had all the command printer functions ready, waiting for this dictionary as argument but these argument was never coming because I was unable to build the dictionary.
So, thank to all the replied codes that is written in this thread, the initial code was fixed and I got the dictionary I wanted.
After solving the initial issue, I moved a bit further and modified the csv by adding extra column "action". So, whenever my client requests existing address object to be added to some address group, I will leave the "action" empty but if it's a new address object, it will have the action "add"..
Real life csv file looks very similar to this.. except it has much more data in it sometimes.
action,name,ip,group
,host01,192.168.0.1,Test01
,host02,192.168.0.2,Test01
add,host03,192.168.0.3,Test02
add,host04,192.168.0.4,Test02
add,host05,192.168.0.5,Test03
,host06,192.168.0.6,Test04
add,host07,192.168.0.7,Test04
add,host08,192.168.0.8,
add,host09,192.168.0.9,
Meaning:
1: host01 and host02 already exist on the firewall but needs to be added to Test01 group
2: host03 and host04 needs to be created AND added to Test02 group
3: host05 needs to be created AND added to Test03 group
4: host06 exists but needs to be added to Test04 group
5: host07 needs to be created AND added to Test04 group
6: host08 and host09 need to be just created.. without adding it to any group.
I updated the code accordingly so here is the final version that does exactly what I wanted (to get the dictionary and pass them to printer function)
Instead of one dictionary I now return tuple of two dictionaries and later on I pass that tuple indexes to command printer function.
first dictionary is for adding the address objects. In this dictionary keys are the address object names and values are it's respective IP addresses.. so,each key will have just one value - IP
Other dictionary is for groups where keys are the group names and values are the list of group members.
import csv
from ipaddress import IPv4Address,IPv4Network, ip_address, ip_network
from datetime import datetime
commands = open('commands.txt','w')
comment = f'"COMMENT-USERNAME-{datetime.now() :%b-%d-%Y}"'
'''
This function goes through the csv file and returns two dictionaries.
1: {Hostname:IP}
2: {Address_Groups:[list of hostnames]}
Later on we will use each of these dictionaries as Command_Printer function arguments.
'''
def Parser():
Address_Groups = {}
Addresses = {}
with open ('Data2.csv','r') as file:
reader = csv.reader(file, delimiter = ',')
next(reader)
for row in reader:
Action=row[0]
Hostname=row[1]
IP=row[2]
GROUP=row[3]
if 'add' in Action and Hostname not in Addresses:
Addresses[Hostname] = IP
if GROUP not in Address_Groups:
Address_Groups[GROUP] = []
if GROUP in row:
Address_Groups[f'{GROUP}'].append(f'"{Hostname}"')
return (Addresses,Address_Groups)
"""
Command printer
"""
def Command_Printer(addr,addrgrp):
for key,value in addr.items():
IPcheck = IPv4Network(value, strict = False).with_netmask
Address = IPcheck.split('/')
commands.write(f'config firewall address\n')
commands.write(f'edit {key}\n')
commands.write(f'set subnet {Address[0]} {Address[1]}\n')
commands.write(f'set comment {comment}\n')
commands.write(f'next\n')
commands.write(f'end\n \n')
for key, value in addrgrp.items():
if key != '':
commands.write(f'config firewall addrgrp\n')
commands.write(f'edit {key}\n')
commands.write(f'set member {" ".join(value)}\n')
commands.write(f'set comment {comment}\n')
commands.write(f'next\n')
commands.write(f'end\n \n')
"""Go babe! Go!"""
def Main():
Call = Parser()
addr = Call[0]
addrgrp=Call[1]
Command_Printer(addr,addrgrp)
Main()
commands.close
Result is this:
config firewall address
edit host03
set subnet 192.168.0.3 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host04
set subnet 192.168.0.4 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host05
set subnet 192.168.0.5 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host07
set subnet 192.168.0.7 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host08
set subnet 192.168.0.8 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host09
set subnet 192.168.0.9 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test01
set member "host01" "host02"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test02
set member "host03" "host04"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test03
set member "host05"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test04
set member "host06" "host07"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
Later on I'm planning to add the policies and maybe VPN tunnels too but for that I think I will need to come up with some class and use pandas instead of csv module. but that's a topic for another day.
If you find this useful, please feel free to use, update, etc the way you want. no limits whatsoever. There is probably million other ways to do this task.
p.s
Apart from all the "banana" and "apple" code examples I've gone through, this is my very first working script that actually has some real life use for me and it feels great and I'm happy I learned allot thank to code examples all of you provided.
Thanks for all the help and for your time to read all this <3
Related
I have an established connection with a notes database and I am able to loop through all the records in a view. What I am curious about if it is possible to open a document and get the data from it using python. (Like double clicking on a record from an HCL Notes Client).
Here is my code simplified:
import noteslib
db = noteslib.Database('my-domino-server','my-db.nsf', 'mypassword')
view = db.GetView('my_view')
doc = view.GetFirstDocument()
while doc:
print(doc.ColumnValues)
#here after printing the column values, I want to open the document and store it's values in a variable.
doc = view.GetNextDocument(doc)
I tried googling about LotusScript and I found the Open() method, but doc.Open() did not work.
Just use the LotusScript documentation to find examples for everything you need.
In your case you start with the NotesDatabase - class, then get an object of type NotesView and finally get a NotesDocument object.
This doc object does not need to be opened. You can directly access all items in that document either by their name or -if you don't know the name- by cycling through all items.
If you e.g. know the name of an item (can be found in the document properties box on the second tab, found with Alt + Enter) then you can read the value like this:
#Subject of a given mail
subject = doc.GetitemValue( "Subject" )[0]
#Start date of a calendar entry
startdate = doc.GetItemValue( "StartDate" )[0]
# cycle through all items
for item in doc.Items
print(item.Name)
value = item.Values
Take care: items are always arrays, even if they contain only a single value. To get the value of a single value item, always access the element at position 0.
I have a .csv file:
csv file
containing packet header data from a wireshark scan that I am iterating through line by line with a for loop. The list contains around 100,000 items, many of which are repeated. I am trying to find how many times each destination IP address is accessed using TCP protocol(6) on each port ranging from 1 to 1024. Essentially I am trying to create something that looks like this:
{ip address: {(protocol:port):count}}
Where I will know how many times a combination of protocol/port tried to use the IP address as a destination. So far I've tried this:
dst = defaultdict(list)
for pkt in csvfile:
if(pkt.tcpdport > 0 and pkt.tcpdport < 1025):
tup = (pkt.proto, pkt.tcpdport)
dst[pkt.ipdst].append(tup)
When I try to print this out I get a list of IP addresses with the protocol, port tuple listed multiple times per IP address. How can I get it so that I show the tuple followed by a count of how many times it occurs in each dictionary entry instead?
Currently, the line dst[pkt.ipdst].append(tup) is telling python, get the value associated with the IP address, and then append the tuple to it. In this case, that means you're appending the tuple to the dictionary associated with the IP address. This is why you're seeing multiple tuples listed per IP address.
To fix this, simply change your line to dst[pkt.ipdst][tup] += 1. This is telling python to get the dictionary associated with the IP address, get the count associated with the tuple in that dictionary, and then add 1. When printed, this should appear as intended.
Also, define dst as defaultdict(lambda:defaultdict(dict)) so that in case the protocol,port combination hasn't been tried, it won't throw a KeyError.
I want to use a dictionary to record data relating to IP addresses, essentially an IP address can have a number of groups associated with it, and I need to capture info about the groups relating to that IP address (these are controller groups on a wireless system so the data is all relating to configuration of access points). I want something like:
{<ip_addr>: [{group_name: my_aps, total_aps: 22, total_active_aps: 12},
{group_name: my-other_aps, total_aps: 15, total_active_aps:14},
{...}
]
}
My script is looping through a list of groups (there are 300+) and pulling the info off the wireless controller. With each loop I obtain the details of the new group. But I can't work out how to then add the group dictionary to the list. I am trying (where group_details is the group dictionary and lms_ip is the address that I want to list it against):
lms_groups[lms_ip].append(group_details)
But I get:
KeyError: 'xxx.xxx.xxx.xxx'
(IP address hidden fwiw)
The script seems to work up to that point, I think the dictionaries are being created ok.
Option 1
dict.setdefault
lms_groups.setdefault(lms_ip, []).append(group_details)
Option 2
collections.defaultdict
from collections import defaultdict
lms_groups = defaultdict(list)
...
lms_groups[lms_ip].append(group_details)
I'm not sure if this fix the error, but at least it's a better access to the dict.
ip_list = lms_groups.get(lms_ip, [])
ip_list.append(group_details)
lms_groups[lms_ip] = ip_list
I'm new to python and to stackoverflow itself, it's my first post here.
I'm working with a log file that looks like this:
Feb 1 00:00:02 bridge kernel: INBOUND TCP: IN=br0 PHYSIN=eth0 OUT=br0 PHYSOUT=eth1 SRC=XXX.XXX.XXX.XXX DST=XXX.XXX.XXX.XXX LEN=40 TOS=0x00 PREC=0x00 TTL=110 ID=12973 PROTO=TCP SPT=220 DPT=6129 WINDOW=16384 RES=0x00 SYN URGP=0
I need to search for everything between the colons. In this line the pattern matched would be INBOUND TCP, but there are other types of patterns.
I have to search that field, store all unique type and how many times they occured in the file.
I already know how to open the file and use re.compile to parse it and i managed to save the unique results in another text file.
Reading the documentation i imagine that i need to use a dictionary with some sort of loop in order to store the different patterns and their occurence number.
Can someone help me?
Thank you if read this far.
#!/usr/bin/python3
import sys
import os
import re
p= re.compile ('bridge kernel:.*:')
with open (sys.argv[1], "r") as f:
with open ('tipos.txt',"w" ) as f2:
for line in f:
if p.search(line):
f2.write(line.split(":")[3] + '\n')
os.system('sort tipos.txt|uniq > tipos2.txt')
dict={}
with open (sys.argv[1],"r") as log:
with open ('tipos2.txt','r') as f:
for l in f:
if f in log:
dict={"(f.line)", "(len(log))"}
print (dict)
First of all you shouldn't call your dictionary dict because it is already an existing keyword in python (The dict() constructor builds dictionaries directly from sequences of key-value pairs).
This line dict={"(f.line)", "(len(log))"} is incorrect, the curly brackets used like this mean you are actually defining a new set containing two strings, and not the variables you want - they are in quotes.
The declaration of the empty dictionary itself is fine.
To add values to an existing dictionary use dictName[key] = value. To declare a dictionary with value pairs use dictName = {key1 : value1, key2 : value2} etc.
I'm a novice in programming. Faced with such a problem. Are monitoring servers using Zabbix. It has its own API. The challenge is through a script in Python to connect to the monitor server and get information about printers and their counters and put in the file. The output file should have the format:
name printer \tab counter printer
Like that:
HP1212 124512
I connect and receive data, but cannot record it in two columns using a '\t'.
My code:
`
from pyzabbix import ZabbixAPI
zapi = ZabbixAPI("http://*****/zabbix")
zapi.login("******", "*******")
item_name='Print_counter'
hosts = zapi.host.get( #get printers name
groupids=8,
output=['name'])
items = zapi.item.get( #get printers counter
groupids=8,
output=['lastvalue'],
filter={'name':item_name})`
I understand that the problem is likely trivial, but how to solve I don't know.
I edited my question:
If im use:
for host in hosts:
a = host['name']
print a
.. I get:
tpr001
tpr002
...
tpr020
it my printers.
If i use:
for item in items:
b = host['value']
print b
I get:
12456
34645
...
56468
It counters off my printers.
I want to group the output of my query like this:
tpr001 12456
tpr002 34645
... ...
tpr020 56468
I think you need something like this :
for host in hosts:
a=host['name']
for item in items:
b=item['lastvalue']
print a,'\t',b`
I do not know what exactly is given by your zapi.host.get and zapi.item.get, but your loops dont work, as you expect.
In your first loop a gets a new value in each cycle so you find the last value in it, if the loop ends. And because of your print command after the loop you see exact that value.
Maybe you should put second loop into the first like
for host in hosts:
a=host['name']
for item in items:
b=item['lastvalue']
print a,'\t',b`
But in this case you would combine each row from hosts with each row from items.
Maybe your items.get- command needs the name es filter, something like
for host in hosts:
a=host['name']
items = zapi.item.get( #get printers counter
groupids=8,
output=['lastvalue'],
filter={'name':a})
Maybe you even do not need to ask for hosts, because all your information are inside items
for item in items:
b=item['lastvalue']
a=item['name']
print a,'\t',b
Hope, this helps, but I think you should learn about basics in programming, if you want to go further (and it is easier to understand, if you give speaking names instead of a and b, so not only we better understand, what you expect
It sounds strange to me, that you want to rely on Output order of two different lists. But if so, you could try
for i in range(len(Hosts)):
host = Hosts[i]
item = Items[i]
a=host['name']
b=item['lastvalue']
print a,'\t',b