Hi I am trying to get the IP of every interface on my machine using netifaces in python. I am still new to python and working my way through some concepts but it seems like I can query every NIC at once and then put the results into a list the problem is my end goal is to ask the user which network to work on and I want to exclude everything that returns 'No IP addr'.
I have already tried a few different methods including removing strings from the list and that didnt work or only adding IP addresses objects but im pretty sure im not doing it properly since it still errors out. Any help is appreciated.
import os
import socket
from netifaces import interfaces, ifaddresses, AF_INET
def get_self_IP():
for ifaceName in interfaces():
addresses = []
possibilities = [i['addr'] for i in ifaddresses(ifaceName).setdefault(AF_INET, [{'addr':'No IP addr'}] )]
print(' '.join(possibilities))
for i in possibilities:
if isinstance(i, ifaddress.ipaddress): # if i is an IP address
addresses.append(i)
print(addresses)
Also I have two lists now because ive changed it a few times to troubleshoot but if I can keep it as one list and only .append the IPs to it while its gathering IPs rather than have to do an entire separate list with a separate for or if loop it would be ideal but I do not know another way unfortunately.
Not sure if this is what you want, but:
I changed the function to:
def get_self_IP():
addresses = []
for ifaceName in interfaces():
ifaddresses_list = ifaddresses(ifaceName)
possibilities = [i['addr'] for i in ifaddresses_list.setdefault(AF_INET, [{'addr': 'No IP addr'}])]
for i in possibilities:
if ifaddresses_list[2][0]['addr'] != "No IP addr":
addresses.append(i)
print(addresses)
The ifaddresses(ifaceName) returns a dictionary which apparently the key 2 contains the IP address in the index 0 with the key 'addr', which in case the IP doesn't exist the value is No IP addr. In that the if checks if that is not the case and then adds the IP address to the addresses list.
If I didn't understand what you want clearly, please answer with more details :)
I'm trying to automate some boring monkey jobs for fortigate firewalls. I receive requests to create address objects for example for 100 different hosts.. many of which should belong to same address group.
My csv is
name,address,group
aix01,10.0.0.1,AIXGROUP1
aix02,10.0.0.2,AIXGROUP1
aix02,10.0.0.3,AIXGROUP2
aix245,10.0.0.4,AIXGROUP2
As you see above, there are 2x groups with 2x hosts in each.
I want to take the group names "AIXGROUP1" and "AIXGROUP2" as a dictionary keys and take list of each of the respective IPs as it's values.
So, it should be {AIXGROUP1:[10.0.0.1,10.0.0.2], AIXGROUP2:[10.0.0.3,10.0.0.4]}
I would then pass this dictionary to a function that prints the commands.
Here's the code
with open('hosts.csv', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter = ',')
Dictionary = {}
for each in reader:
if len(each[2])!=0:
Dictionary[each[2]] = []
at this point I just have a dictionary with correct keys and empty lists as values.. and here's where I'm stuck.. how do I populate these empty lists with ip addresses?
I feel like I'm inch away from the victory! :)
You are close to a solution. The only thing you need to add is a check if the key already exist in the dictionary, so you don't overwrite it and then add a row that adds the value to the list. And you can add the row next(reader, None) if you don't want the header row in the dictionary. Something like this should work:
with open('hosts.csv', 'r') as csvfile:
reader = csv.reader(csvfile, delimiter = ',')
next(reader, None) # To skip the header row
Dictionary = {}
for each in reader:
if len(each[2])!=0 and each[2] not in Dictionary.keys(): # To not overwrite keys that already exist
Dictionary[each[2]] = []
Dictionary[each[2]].append(each[1]) #To add the values
Your loop needs to take different actions depending on whether the group name is already in the dictionary or not, like this (I'll use d instead of Dictionary for brevity):
group = each[2]
ip = each[1]
if group in d:
d[group].append(ip) # Append another IP to the list
else:
d[group] = [ip] # Initialize list with first element
I would do it like this:
result = {}
for row in reader:
if len(row[2])!=0:
try:
result[row[2]].append(row[1])
except:
result[row[2]] = [row[1]]
result
>>> {'AIXGROUP1': ['10.0.0.1', '10.0.0.2'], 'AIXGROUP2': ['10.0.0.3', '10.0.0.4']}
This approach creates the lists and fills them in the same loop. The try / except logic is faster than a conditional and works like this:
For each row of data, try to append the address to the right group.
If the exception is raised, that key doesn't yet exist in the dictionary, so we have to create it.
On creating the key, we define the key as a list with one element in it, which is the current address.
If the exception isn't raised, the dictionary already has one address for that key, so we just append the next one to the list referenced by that key.
Note that it's probably not best practice to name your output dict Dictionary. Also, if you are going to use csv.read(), don't forget to skip the header row!
import csv
# create the dictionary in which your data will be stored
ip_dict = {}
# open the wanted file
with open('hosts.csv', 'r') as csvfile:
# read the csv data
csv_reader = csv.reader(csvfile, delimiter=',')
# skip the header
next(csv_reader)
# parse your data
for _, ip, name in csv_reader:
# do whatever you need only if name is not empty
if name:
# if there is already an entry in the dictionary then append to it
if name in ip_dict:
ip_dict[name].append(ip)
# if there is not create a list with the current ip as the first entry
else:
ip_dict[name] = [ip]
I Think my final version will be this.
def parser():
Address_Groups = {}
Addresses = {}
with open ('Data.csv','r') as file:
reader = csv.reader(file, delimiter = ',')
next(reader)
for row in reader:
if row [0] not in Addresses:
Addresses[row[0]] = row[1]
if row[2] not in Address_Groups:
Address_Groups[row[2]] = []
if row[2] in row:
Address_Groups[f'{row[2]}'].append(f'{row[0]}')
return (Addresses,Address_Groups)
Like this I successfully map some server name to it's respective IP address and name to respective address group.
when I do return (Objects,Address_Groups)it returns a tuple of two dictionaries, right?
so if I do print(parser())
What I'm getting is a tuple(?) of two dictionaries.. is that right?
({'Host-01': '192.168.1.1', 'Host-02': '192.168.1.2', 'Host-03': '192.168.1.3', 'Host-04': '192.168.1.4'}, {'Test_Group-01': ['Host-01', 'Host-02'], 'Test_Group-02': ['Host-03'], 'Test_Group-012': ['Host-04']})
You can use defaultdict from the collections python library. It was created exactly for your case.
ar = [
'aix01,10.0.0.1,AIXGROUP1',
'aix02,10.0.0.2,AIXGROUP1',
'aix02,10.0.0.3,AIXGROUP2',
'aix245,10.0.0.4,AIXGROUP2'
]
from collections import defaultdict
d = defaultdict(list)
for row in ar:
elements = row.split(',')
d[elements[2]].append(elements[1])
print(dict(d))
This will print
{'AIXGROUP1': ['10.0.0.1', '10.0.0.2'], 'AIXGROUP2': ['10.0.0.3',
'10.0.0.4']}
So.. to make it more clear I will explain this from A to Z
I'm a network engineer who works primarily with fortigate firewalls.
Sometimes I receive the requests to add lots of address objects on the firewall.
Often times these objects should belong to address groups and later I use these address objects and address groups in firewall policies.
My objective was to create a short script that would write the firewall CLI commands in the file
so initially I structured the csv file in this way. Script should go through this and write firewall commands into a text file.
name,address,group
aix01,10.0.0.1,AIXGROUP1
aix02,10.0.0.2,AIXGROUP1
aix03,10.0.0.3,AIXGROUP2
aix04,10.0.0.4,AIXGROUP2
firewall cli command for adding object
config firewall address
edit <Address Object Name>
set subnet <IP> <Mask>
set comment <Comment>
next
end
and for adding hosts into address group
config firewall addrgrp
edit <Address Group Name>
set member "<Address Object>" "<Address Object>"
set comment <comment>
next
end
For My initial question I used very simple csv and question was how to build the dictionary from the csv file specifically for the address group mapping because how I see this working is to have group name as a KEY and list of group members as VALUES. so in the initial example desired result was
{
AIXGROUP1:[aix01,aix02],
AIXGROUP2:[aix03,aix04]
}
I was stuck at this part - I was getting correct keys but was unable to populate the value lists with correct IP addresses/hostnames.
I had all the command printer functions ready, waiting for this dictionary as argument but these argument was never coming because I was unable to build the dictionary.
So, thank to all the replied codes that is written in this thread, the initial code was fixed and I got the dictionary I wanted.
After solving the initial issue, I moved a bit further and modified the csv by adding extra column "action". So, whenever my client requests existing address object to be added to some address group, I will leave the "action" empty but if it's a new address object, it will have the action "add"..
Real life csv file looks very similar to this.. except it has much more data in it sometimes.
action,name,ip,group
,host01,192.168.0.1,Test01
,host02,192.168.0.2,Test01
add,host03,192.168.0.3,Test02
add,host04,192.168.0.4,Test02
add,host05,192.168.0.5,Test03
,host06,192.168.0.6,Test04
add,host07,192.168.0.7,Test04
add,host08,192.168.0.8,
add,host09,192.168.0.9,
Meaning:
1: host01 and host02 already exist on the firewall but needs to be added to Test01 group
2: host03 and host04 needs to be created AND added to Test02 group
3: host05 needs to be created AND added to Test03 group
4: host06 exists but needs to be added to Test04 group
5: host07 needs to be created AND added to Test04 group
6: host08 and host09 need to be just created.. without adding it to any group.
I updated the code accordingly so here is the final version that does exactly what I wanted (to get the dictionary and pass them to printer function)
Instead of one dictionary I now return tuple of two dictionaries and later on I pass that tuple indexes to command printer function.
first dictionary is for adding the address objects. In this dictionary keys are the address object names and values are it's respective IP addresses.. so,each key will have just one value - IP
Other dictionary is for groups where keys are the group names and values are the list of group members.
import csv
from ipaddress import IPv4Address,IPv4Network, ip_address, ip_network
from datetime import datetime
commands = open('commands.txt','w')
comment = f'"COMMENT-USERNAME-{datetime.now() :%b-%d-%Y}"'
'''
This function goes through the csv file and returns two dictionaries.
1: {Hostname:IP}
2: {Address_Groups:[list of hostnames]}
Later on we will use each of these dictionaries as Command_Printer function arguments.
'''
def Parser():
Address_Groups = {}
Addresses = {}
with open ('Data2.csv','r') as file:
reader = csv.reader(file, delimiter = ',')
next(reader)
for row in reader:
Action=row[0]
Hostname=row[1]
IP=row[2]
GROUP=row[3]
if 'add' in Action and Hostname not in Addresses:
Addresses[Hostname] = IP
if GROUP not in Address_Groups:
Address_Groups[GROUP] = []
if GROUP in row:
Address_Groups[f'{GROUP}'].append(f'"{Hostname}"')
return (Addresses,Address_Groups)
"""
Command printer
"""
def Command_Printer(addr,addrgrp):
for key,value in addr.items():
IPcheck = IPv4Network(value, strict = False).with_netmask
Address = IPcheck.split('/')
commands.write(f'config firewall address\n')
commands.write(f'edit {key}\n')
commands.write(f'set subnet {Address[0]} {Address[1]}\n')
commands.write(f'set comment {comment}\n')
commands.write(f'next\n')
commands.write(f'end\n \n')
for key, value in addrgrp.items():
if key != '':
commands.write(f'config firewall addrgrp\n')
commands.write(f'edit {key}\n')
commands.write(f'set member {" ".join(value)}\n')
commands.write(f'set comment {comment}\n')
commands.write(f'next\n')
commands.write(f'end\n \n')
"""Go babe! Go!"""
def Main():
Call = Parser()
addr = Call[0]
addrgrp=Call[1]
Command_Printer(addr,addrgrp)
Main()
commands.close
Result is this:
config firewall address
edit host03
set subnet 192.168.0.3 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host04
set subnet 192.168.0.4 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host05
set subnet 192.168.0.5 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host07
set subnet 192.168.0.7 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host08
set subnet 192.168.0.8 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall address
edit host09
set subnet 192.168.0.9 255.255.255.255
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test01
set member "host01" "host02"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test02
set member "host03" "host04"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test03
set member "host05"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
config firewall addrgrp
edit Test04
set member "host06" "host07"
set comment "COMMENT-USERNAME-Nov-06-2021"
next
end
Later on I'm planning to add the policies and maybe VPN tunnels too but for that I think I will need to come up with some class and use pandas instead of csv module. but that's a topic for another day.
If you find this useful, please feel free to use, update, etc the way you want. no limits whatsoever. There is probably million other ways to do this task.
p.s
Apart from all the "banana" and "apple" code examples I've gone through, this is my very first working script that actually has some real life use for me and it feels great and I'm happy I learned allot thank to code examples all of you provided.
Thanks for all the help and for your time to read all this <3
I have a .csv file:
csv file
containing packet header data from a wireshark scan that I am iterating through line by line with a for loop. The list contains around 100,000 items, many of which are repeated. I am trying to find how many times each destination IP address is accessed using TCP protocol(6) on each port ranging from 1 to 1024. Essentially I am trying to create something that looks like this:
{ip address: {(protocol:port):count}}
Where I will know how many times a combination of protocol/port tried to use the IP address as a destination. So far I've tried this:
dst = defaultdict(list)
for pkt in csvfile:
if(pkt.tcpdport > 0 and pkt.tcpdport < 1025):
tup = (pkt.proto, pkt.tcpdport)
dst[pkt.ipdst].append(tup)
When I try to print this out I get a list of IP addresses with the protocol, port tuple listed multiple times per IP address. How can I get it so that I show the tuple followed by a count of how many times it occurs in each dictionary entry instead?
Currently, the line dst[pkt.ipdst].append(tup) is telling python, get the value associated with the IP address, and then append the tuple to it. In this case, that means you're appending the tuple to the dictionary associated with the IP address. This is why you're seeing multiple tuples listed per IP address.
To fix this, simply change your line to dst[pkt.ipdst][tup] += 1. This is telling python to get the dictionary associated with the IP address, get the count associated with the tuple in that dictionary, and then add 1. When printed, this should appear as intended.
Also, define dst as defaultdict(lambda:defaultdict(dict)) so that in case the protocol,port combination hasn't been tried, it won't throw a KeyError.
H = {'PC1': ['01:01:01:01:01:01', '192.168.0.1', '200', {}],
'PC2': ['02:02:02:02:02:02',
'192.168.0.2',
'200',
{'192.168.0.1': ('01:01:01:01:01:01', 1390710916.226)}],
'PC3': ['03:03:03:03:03:03', '192.168.0.3', '200', {}]}
In this dictionary I want the user to input four values host_id, IP, MAC and time.
So in first turn user inputs PC1 as host_id, 192.168.0.1 as IP, 01:01:01:01:01:01 as MAC, and 200 as time
Note inside the tuple are MAC and time (this is not the time as given above but called though time.time() function). Means key inside a nested dictionary is associated with tuple.
Now if in next turn a user inputs same IP but different MAC, e.g., 192.168.0.1 as IP and 04:04:04:04:04:04 as MAC, I need to check nested dictionary of all PCs so that if someone has old MAC and time (for the key) it should be replaced with latest information (new MAC and time), but if and only if (time already associated with the key (IP) + 200) is less than current time (time.time()). Otherwise I will delete the relevant key from nested dictionary of that host.
So after I entered new information for PC1 as host_id=PC1, IP=192.168.0.1, MAC=04:04:04:04:04:04, and time=200, PC2 would update because only it has relevant key in its nested dictionary.
Output will be:
H = {'PC1': ['01:01:01:01:01:01', '192.168.0.1', '200', {}],
'PC2': ['02:02:02:02:02:02',
'192.168.0.2',
'200',
{'192.168.0.1': ('04:04:04:04:04:04', 1390864671.669904)}],
'PC3': ['03:03:03:03:03:03', '192.168.0.3', '200', {}]}
Note: there can be many PCs, and user can input any host. So I want to check inside dictionaries of all other hosts.
Let's try to translate your English description into Python:
I need to check nested dictionary of all PCs
for key, (mac, ip, time, sub_dict) in H.items():
… so that if someone has old MAC and time (for the key)
I think here you want to search in each sub_dict, which maps ips to (mac, time) pairs, to see if any of them match the new entry's mac and time:
for sub_ip, (sub_mac, sub_time) in sub_dict.items():
if (sub_mac, sub_time) == (user_mac, user_time):
… if and only if (time already associated with the key (IP) + 200) is less than current time (time.time())
if sub_time + 200 < time.time():
… it should be replaced with latest information (new MAC and time)
sub_dict[sub_ip] = (user_mac, user_time)
… Otherwise I will delete the relevant key from nested dictionary of that host.
else:
del sub_dict[sub_ip]
However, this runs into a problem. Replacing the value for a key while iterating over a dictionary is fine, but adding or removing keys is not. You may get an exception, or incorrect results, or, worst of all, correct results most of the time but incorrect results every so often.
You can fix that by iterating over a copy of the dict, or by building a new replacement dict, e.g., using a comprehension.
Normally, I'd suggest a comprehension, but in this case, your condition is so complicated that might get hard to read, so let's use a copy instead.
for key, (mac, ip, time, sub_dict) in H.items():
for sub_ip, (sub_mac, sub_time) in dict(sub_dict).items():
if (sub_mac, sub_time) == (user_mac, user_time):
if sub_time + 200 < time.time():
sub_dict[sub_ip] = (user_mac, user_time)
else:
del sub_dict[sub_ip]
I'm not sure I understood your description correctly, but even if I didn't, this should serve as a template for you to write the code you need.
As I mentioned in a comment, this would all be a lot simpler—and faster—if you used a more sensible data structure. If you want to look things up, the thing you want to loop up should be a key in a dictionary, not a value. If there can be multiple values for each key, that's not a problem; just use a dict whose values are sets. If you need to look things up both ways, you can use one of the various "bidirectional dict" implementations on PyPI, or use "index maps", separate dictionaries that map each kind of "alternate key" to the appropriate "primary key". If I understood what you were trying to do and why, I could give a more specific suggestion, but hopefully this is enough to get you thinking about your data structure design.
Im looking for thoughts on solving what im trying to achieve.
Im looking to do a dns query in python say for instance;
import adns
c=adns.init()
c.synchronous("www.google.com", adns.rr.A)
Im looking to store the output;
(0, 'www.l.google.com', 1167604334, ('216.239.37.99', '216.239.37.104'))
in a list, but the list would need to be associated with teh domain name.
So www.google.com would need to be associated with the common name and ip addresses. So if I had a list of domains they are all printed with their relevant data.
Can you have a list inside another list? Abit like a database i guess?
yep, a dict will do
data = []
domain = "www.google.com"
data[domain] = c.synchronous(domain, adns.rr.A)
print data