I am working on building an embedded network appliance (linux based) and have come across the need to dynamically build daemon conf files. As such, I need to be able to do some network address calculations in the python code that will be building the conf files. I am not a programmer so I'm fearful I wrote a module that will fail to function as I'd hoped once the appliance starts shipping.
Below is what I have so far, its really pieced together with what I could find on this site and Google.
Is there a better method to find the network address and cidr for a network interface? Converting the netmask to a bin str and counting the 1's seems pretty inelegant.
import socket
import fcntl
import struct
SIOCGIFNETMASK = 0x891b
SIOCGIFADDR = 0x8915
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
def _GetIfaceMask(iface):
return struct.unpack('L', fcntl.ioctl(s, SIOCGIFNETMASK, struct.pack('256s', iface))[20:24])[0]
def _GetIfaceAddr(iface):
return struct.unpack('L', fcntl.ioctl(s, SIOCGIFADDR, struct.pack('256s', iface[:15]))[20:24])[0]
def GetIfaceNet(iface):
net_addr = _GetIfaceAddr(iface) & _GetIfaceMask(iface)
return socket.inet_ntoa(struct.pack('L', net_addr))
def GetIfaceCidr(iface):
bin_str = bin(_GetIfaceMask(iface))[2:]
cidr = 0
for c in bin_str:
if c == '1': cidr += 1
return cidr
Thanks for any input, I really am somewhat lost on this. If this isn't the place for this type of feedback, please let me know.
This can be solved using a Hamming weight algorithm. Stolen from How to count the number of set bits in a 32-bit integer? and translated into Python:
def number_of_set_bits(x):
x -= (x >> 1) & 0x55555555
x = ((x >> 2) & 0x33333333) + (x & 0x33333333)
x = ((x >> 4) + x) & 0x0f0f0f0f
x += x >> 8
x += x >> 16
return x & 0x0000003f
Another, more readable solution (but running in O(log x)):
def number_of_set_bits(x):
n = 0
while x:
n += x & 1
x = x >> 1
return n
You may check iptools python module http://code.google.com/p/python-iptools/ it can convert from long to dotted ip format and vise versa.
Related
I'm trying to get a quick implementation of the following problem, ideally such that it would work in a numba function. The problem is the following: I have two random integers a & b and consider their binary representation of length L, e.g.
L=4: a=10->1010, b=6->0110.
This is the information that is feed into the function. Then I cut both binary representations in two at the same random position and fuse one of the two results, e.g.
L=4: a=1|010, b=0|110 ---> c=1110 or 0010.
One of the two outcome is chosen with equal probability and that is the outcome of the function. The cut occurs between the first 1/0 and the last 0/1 of the binary representation.
This is currently my code:
def func(a,b,l):
bin_a = [int(i) for i in str(bin(a))[2:].zfill(l)]
bin_b = [int(i) for i in str(bin(b))[2:].zfill(l)]
randint = random.randint(1, l - 1)
print("randint", randint)
if random.random() < 0.5:
result = bin_a[0:randint]+bin_b[randint:l]
else:
result = bin_b[0:randint] + bin_a[randint:l]
return result
I have the feeling that there a possibly many shortcuts to this problem that I do not come up with. Also my code does not work in numba :/. Thanks for any help!
Edit: This is an update of my code, thanks to Prunes help! It also works as a numba function. If there is no further improvements to that, I would close the question.
def func2(a,b,l):
randint = random.randint(1, l - 1)
print("randint", randint)
bitlist_l = [1]*randint+[0]*(l-randint)
bitlist_r = [0]*randint+[1]*(l-randint)
print("bitlist_l", bitlist_l)
print("bitlist_r", bitlist_r)
l_mask = 0
r_mask = 0
for i in range(l):
l_mask = (l_mask << 1) | bitlist_l[i]
r_mask = (r_mask << 1) | bitlist_r[i]
print("l_mask", l_mask)
print("r_mask", r_mask)
if random.random() < 0.5:
c = (a & l_mask) | (b & r_mask)
else:
c = (b & l_mask) | (a & r_mask)
return c
You lose a lot of time converting between string and int. Try bit operations instead. Mask the items you want and construct the output without all the conversions. Try these steps:
size = [length of larger number in bits] There are many ways to get this.
Make a mask template, size 1-bits.
Pick your random position, pos randint is a poor anem, as it shadows the function you're using.
Make two masks: l_mask = mask << pos; r_mask = mask >> pos. This gives you two mutually exclusive and exhaustive bit-maps for your inputs.
Flip your random coin, the 50-50 chance. The < 0.5 result would be ...
(a & l_mask) | (b & rmask)
For the >= 0.5 result, switch a and b in that expression.
You can improve your code by realizing that you do not need a "human readable" binary representation to do binary operations.
For example, creating the mask:
m = (1<<randompos) - 1
The crossover can be done like so:
c = (a if coinflip else b) ^ ((a^b)&m)
And that's all.
Full example:
# create random sample
a,b = np.random.randint(1<<32,size=2)
randompos = np.random.randint(1,32)
coinflip = np.random.randint(2)
randompos
# 12
coinflip
# 0
# do the crossover
m = (1<<randompos) - 1
c = (a if coinflip else b) ^ ((a^b)&m)
# check
for i in (a,b,m,c):
print(f"{i:032b}")
# 11100011110111000001001111100011
# 11010110110000110010101001111011
# 00000000000000000000111111111111
# 11010110110000110010001111100011
I'm a newbie in this field and am trying to learn a bit about how to write cryptographic hash functions.
To get some hands-on, I tried updating the PySHA2 algorithm for Python 3.6 and up (the original version doesn't work on Python 2.5+ and the author says he won't fix this). I don't intend to use this algorithm for any work, just coding this for the sake of knowledge.
I've reached this far:
import copy
import struct
_initial_hashes = [0x6a09e667f3bcc908, 0xbb67ae8584caa73b, 0x3c6ef372fe94f82b, 0xa54ff53a5f1d36f1,
0x510e527fade682d1, 0x9b05688c2b3e6c1f, 0x1f83d9abfb41bd6b, 0x5be0cd19137e2179]
_round_constants = [0x428a2f98d728ae22, 0x7137449123ef65cd, 0xb5c0fbcfec4d3b2f, 0xe9b5dba58189dbbc,
0x3956c25bf348b538, 0x59f111f1b605d019, 0x923f82a4af194f9b, 0xab1c5ed5da6d8118,
0xd807aa98a3030242, 0x12835b0145706fbe, 0x243185be4ee4b28c, 0x550c7dc3d5ffb4e2,
0x72be5d74f27b896f, 0x80deb1fe3b1696b1, 0x9bdc06a725c71235, 0xc19bf174cf692694,
0xe49b69c19ef14ad2, 0xefbe4786384f25e3, 0x0fc19dc68b8cd5b5, 0x240ca1cc77ac9c65,
0x2de92c6f592b0275, 0x4a7484aa6ea6e483, 0x5cb0a9dcbd41fbd4, 0x76f988da831153b5,
0x983e5152ee66dfab, 0xa831c66d2db43210, 0xb00327c898fb213f, 0xbf597fc7beef0ee4,
0xc6e00bf33da88fc2, 0xd5a79147930aa725, 0x06ca6351e003826f, 0x142929670a0e6e70,
0x27b70a8546d22ffc, 0x2e1b21385c26c926, 0x4d2c6dfc5ac42aed, 0x53380d139d95b3df,
0x650a73548baf63de, 0x766a0abb3c77b2a8, 0x81c2c92e47edaee6, 0x92722c851482353b,
0xa2bfe8a14cf10364, 0xa81a664bbc423001, 0xc24b8b70d0f89791, 0xc76c51a30654be30,
0xd192e819d6ef5218, 0xd69906245565a910, 0xf40e35855771202a, 0x106aa07032bbd1b8,
0x19a4c116b8d2d0c8, 0x1e376c085141ab53, 0x2748774cdf8eeb99, 0x34b0bcb5e19b48a8,
0x391c0cb3c5c95a63, 0x4ed8aa4ae3418acb, 0x5b9cca4f7763e373, 0x682e6ff3d6b2b8a3,
0x748f82ee5defb2fc, 0x78a5636f43172f60, 0x84c87814a1f0ab72, 0x8cc702081a6439ec,
0x90befffa23631e28, 0xa4506cebde82bde9, 0xbef9a3f7b2c67915, 0xc67178f2e372532b,
0xca273eceea26619c, 0xd186b8c721c0c207, 0xeada7dd6cde0eb1e, 0xf57d4f7fee6ed178,
0x06f067aa72176fba, 0x0a637dc5a2c898a6, 0x113f9804bef90dae, 0x1b710b35131c471b,
0x28db77f523047d84, 0x32caab7b40c72493, 0x3c9ebe0a15c9bebc, 0x431d67c49c100d4c,
0x4cc5d4becb3e42b6, 0x597f299cfc657e2a, 0x5fcb6fab3ad6faec, 0x6c44198c4a475817]
def _rit_rot(on: int, by: int) -> int:
"""
helper function for right rotation as it isn't done by a simple bitwise operation (xor is done by '^')
:param on: value to be rotated
:param by: value by which to rotate
:return: right rotated 'on'
"""
return ((on >> by) | (on << (64 - by))) & 0xFFFFFFFFFFFFFFFF
def hash_main(chunk):
global _initial_hashes, _round_constants
# start the hashing process
# to begin, create a place to store the 80 words that we'll make
words = [0] * 80
# first 16 words will be saved without any changes
words[:16] = struct.unpack('!16Q', chunk)
# extend these 16 words into the remaining 64 words of 'message schedule array'
for i in range(16, 80):
part_1 = _rit_rot(words[i - 15], 1) ^ _rit_rot(words[i - 15], 8) ^ (words[i - 15] >> 7)
part_2 = _rit_rot(words[i - 2], 19) ^ _rit_rot(words[i - 2], 61) ^ (words[i - 2] >> 6)
words[i] = (words[i - 16] + part_1 + words[i - 7] + part_2) & 0xFFFFFFFFFFFFFFFF
# create the working variables
a, b, c, d, e, f, g, h = _initial_hashes
# start the compression function
for z in range(80):
var_1 = _rit_rot(a, 28) ^ _rit_rot(a, 34) ^ _rit_rot(a, 39)
var_2 = _rit_rot(e, 14) ^ _rit_rot(e, 18) ^ _rit_rot(e, 41)
var_3 = (a & b) ^ (a & c) ^ (b & c)
var_4 = (e & f) ^ ((~e) & g)
temp_1 = var_1 + var_3
temp_2 = h + var_2 + var_4 + _round_constants[z] + words[z]
# remix the hashes
h = g
g = f
f = e
e = (d + temp_2) & 0xFFFFFFFFFFFFFFFF
d = c
c = b
b = a
a = (temp_1 + temp_2) & 0xFFFFFFFFFFFFFFFF
# add this chunk to initial hashes
_initial_hashes = [(x + y) & 0xFFFFFFFFFFFFFFFF for x, y in zip(_initial_hashes,
[a, b, c, d, e, f, g, h])]
def _sha_backend_update(text_copy, _buffer, _counter):
"""
backend function that hashes given string
"""
global _initial_hashes, _round_constants
# create variables for cycling
_buffer += text_copy
_counter += len(text_copy)
# assert the variables are correct
if not text_copy:
return
if type(text_copy) is not str:
raise TypeError("Invalid Object! Please enter a valid string for hashing!")
# break the buffer into 128-bit chunks
while len(_buffer) >= 128:
chunk = _buffer[:128].encode()[1:]
hash_main(chunk)
_buffer = _buffer[128:]
def sha_backend_digest(text_to_hash: str, _buffer: str, _counter: int,
_output_size: int, hex_output: bool = False):
# initialize variables
variable_x = _counter & 0x7F
length = str(struct.pack('!Q', _counter << 3))
# set the thresholds
if variable_x < 112:
padding_len = 111 - variable_x
else:
padding_len = 239 - variable_x
# make a copy of the text_to_hash before starting hashing
text_copy = copy.deepcopy(text_to_hash)
m = '\x80' + ('\x00' * (padding_len + 8)) + length
# run the update function
_sha_backend_update(text_copy, _buffer, _counter)
# return the hash value
return_val = [hex(stuff) for stuff in _initial_hashes[:_output_size]]
if hex_output is True:
return_val = [int(stuff, base=16) for stuff in return_val]
return return_val
return ''.join(return_val)
def sha_512(text_to_hash: str, hex_digest: bool = False) -> str:
"""
frontend function for SHA512 hashing
:return: hashed string
"""
# before anything, check if the input is correct
if not text_to_hash:
return ""
if type(text_to_hash) is not str:
raise TypeError("Invalid content! Please provide content in correct format for hashing!")
# initialize default variables
_buffer = ''
_counter = 0
_output_size = 8
# start the backend function
return sha_backend_digest(text_to_hash, _buffer, _counter, _output_size, hex_output=hex_digest)
message = "This is a string to be hashed"
from hashlib import sha512
print("hashlib gives: ", sha512(message.encode()).hexdigest())
print("I give: ", sha_512(message))
As is obvious, I don't understand a lot of things in this algorithm and have literally copied many parts from the original code (also, I know it isn't good practice to write everything in a single function but I find it easier when trying to understand something).
But the biggest problem I have right now is it doesn't work! Whatever input message I provide to my function, it gives the same output:
0x6a09e667f3bcc9080xbb67ae8584caa73b0x3c6ef372fe94f82b0xa54ff53a5f1d36f1
0x510e527fade682d10x9b05688c2b3e6c1f0x1f83d9abfb41bd6b0x5be0cd19137e2179
I wrote a code at the bottom to compare it with python's hashlib module.
Where am I going wrong in this and how do I fix this?
EDIT: As mentioned in the comments, I tried to feed in a longer message string and the code seems to be working (it still gives longer output than hashlib though):
message = "This is a string to be hashed. I'll try to make this string as long as possible by adding" \
"as much information to it as I can, in the hopes that this string would somehow become longer than" \
"128 bits and my code can run properly. Hopefully, this is already longer than 128 bits, so lets see" \
"how it works..."
hash: 0x6fcc0f346f2577800x334bd9b6c1178a970x90964a3f45f7b5bb0xc14033d12f6607e60xb598bea0a8b0ac1e0x116b0e134691ab540x73d88e77e5b862ba0x89181da7462c5574
message = "This is a string to be hashed. I'll try to make this string as long as possible by adding" \
"as much information to it as I can, in the hopes that this string would somehow become longer than"
hash: 0x166e40ab03bc98750xe81fe34168b6994f0xe56b81bd5972b5560x8789265c3a56b30b0x2c810d652ea7b1550xa23ca2704602a8240x12ffb1ec8f3dd6d10x88c29f84cbef8988
You'll always have to pad the message. Padding and adding the length are always required as last step of the SHA-2 process. Currently you weren't performing that last step (to completion).
Here are my last two comments that pointed you in the right direction:
So generally you try and take one 128 byte block from the binary message, update the hash state using the information in that block, then move to the next one until you have a partial or 0 byte block. That block you need to pad & add size indication (in bits) and process. If you've not enough space for the padding / size indication then you need yet another block consisting entirely of padding and the size indication. If you read carefully, then you always process at least one block.
and
Hmm, it is already in sha_backend_digest (the 0x80 followed by zero bytes and the length which is input size * 8 (_counter << 3).
But of course you do need to perform that and not skip any step.
I have a python dictionary that I've created, this dictionary contains a list of subnets in the following format:
x.x.x.x/24
y.y.y,y/25
z.z.z.z/26
a.a.a.a/27
etc...
I would like to take the items in this dictionary, parse it, then spit out the results in the following format:
x.x.x.x 255.255.255.0
y.y.y.y 255.255.255.128
x.x.x.x 255.255.255.192
a.a.a.a 255.255.255.224
I don't have much on this as of right now because I can't find a lot on this topic on the web, not anything that can be in a quick and concise way that is. Thoughts?
Code:
import socket
import struct
def cidr_to_netmask(cidr):
network, net_bits = cidr.split('/')
host_bits = 32 - int(net_bits)
netmask = socket.inet_ntoa(struct.pack('!I', (1 << 32) - (1 << host_bits)))
return network, netmask
Usage:
>>> cidr_to_netmask('10.10.1.32/27')
('10.10.1.32', '255.255.255.224')
>>> cidr_to_netmask('208.128.0.0/11')
('208.128.0.0', '255.224.0.0')
>>> cidr_to_netmask('208.130.28.0/22')
('208.130.28.0', '255.255.252.0')
I thought I'd throw in my own solution, since I was going for a little bit more readability in it than the other answers shown.
def cidr_to_netmask(cidr):
cidr = int(cidr)
mask = (0xffffffff >> (32 - cidr)) << (32 - cidr)
return (str( (0xff000000 & mask) >> 24) + '.' +
str( (0x00ff0000 & mask) >> 16) + '.' +
str( (0x0000ff00 & mask) >> 8) + '.' +
str( (0x000000ff & mask)))
It's now easier to see what's going on, and that is:
Get the numeric mask by padding the front with 1s and having the cidr make up the rest
For each bit, apply the mask
Concatenate all the bits together, separated by periods
This is very procedural and does not use any libraries.
Try this solution:
Python3
from ipaddress import IPv4Network
networks = {'n1':'10.1.0.0/21','n2':'10.2.0.0/22','n3':'10.3.0.0/23','n4':'10.4.0.0/24'}
for x,y in enumerate(networks):
print(IPv4Network(networks[y]).network_address, IPv4Network(networks[y]).netmask)
Result:
10.1.0.0 255.255.248.0
10.2.0.0 255.255.252.0
10.3.0.0 255.255.254.0
10.4.0.0 255.255.255.0
Python2
from netaddr import IPNetwork
networks = {'n1':'10.1.0.0/21','n2':'10.2.0.0/22','n3':'10.3.0.0/23','n4':'10.4.0.0/24'}
for x,y in enumerate(networks):
print(str(IPNetwork(networks[y]).network), str(IPNetwork(networks[y]).netmask))
Result:
('10.1.0.0', '255.255.248.0')
('10.2.0.0', '255.255.252.0')
('10.3.0.0', '255.255.254.0')
('10.4.0.0', '255.255.255.0')
Try this
lsIP = []
ans = 0
CIDR = 32
IP = [1] * CIDR
for i in range(len(IP)):
iIdx = i % 8
if iIdx == 0:
if i >= 8:
lsIP.append(ans)
ans = 0
ans += pow(2, 7 - iIdx)
lsIP.append(ans)
[lsIP.append(0) for i in range(4 - len(lsIP))]
print lsIP
I have a Python script to collect hostnames of ip address with primes as byte entities.
E.g., 211.13.17.2 is a valid ip according to my problem set where every byte entity(decimal representation) is a prime.
Code:
from itertools import product
import socket
# prime or not
def prime(n):
if n > 1:
p = 0
for i in range(2, n-1):
if divmod(n, i)[1] == 0:
p = 1
break
if p == 0:
return True
def get_host_name(b1, b2, b3, b4):
addr = str(b1) + '.' + str(b2) + '.' + str(b3) + '.' + str(b4)
try:
return socket.gethostbyaddr(addr)
except socket.herror:
pass
# find host names whose ip addresses are all primes
byte = [b for b in range(0, 256) if prime(b)]
ips = list(product(byte, byte, byte, byte))
print 'Total ips = ', len(ips)
for ip in ips:
if get_host_name(*ip):
print get_host_name(*ip)
The problem is my script is too slow. I need expert help to optimize this code. Please pinpoint all mistakes and ways to make it behave faster.
for the prime numbers, you can use something like this,
import numpy as np
isprime = lambda x: np.all(np.mod(x, range(2, 1 + int(np.sqrt(x)))))
primes = np.array([ x for x in range(2, 255) if isprime(x) ])
and you can have a generator for ip addresses by
('{}.{}.{}.{}'.format(*x) for x in itertools.product(primes, repeat=4))
but most likely the code is slow in the socket part, and because of the number of combinations that it needs to check; for that you may try parallelism, by using a pool of worker processes; something like this:
from multiprocessing import Pool
from socket import gethostbyaddr
def gethost(addr):
try:
return gethostbyaddr(addr)
except:
pass
if __name__ == '__main__':
p = Pool(3)
print (p.map(gethost,['74.125.228.137',
'11.222.333.444',
'17.149.160.49',
'98.139.183.24']))
edit: for only prime numbers less than 50, (50K+ combinations) and 20 worker processes it takes almost 6 minutes on my machines and it finds 16K+ results. so, with this huge number of combinations parallelism cannot help much.
I've been looking for this info for awhile, and I have a number of ways to retrieve a list of local drives under Windows. Here are two examples:
print win32api.GetLogicalDriveStrings().split("\x00")
and
def getDriveLetters(self):
self.drvs = []
n_drives = win32api.GetLogicalDrives()
for i in range(0,25): #check all drive letters
j = 2**i # bitmask for each letter
if n_drives & j > 0:
self.drvs.append(chr(65+i)+":/")
print self.drvs
What I can't seem to find is a way to separate the floppies (A:), usb drives (G:), CD drives (E:), and network drives (P:) from the Local hard drives (C:, D:)
If they all were assigned the same letters it would be easy, but I'm writing this script to monitor local hard disk space across a network of computers with different configurations.
Any help would be appreciated! Thanks.
You can try the win32 GetDriveType function.
import win32file
>>> win32file.GetDriveType("C:/") == win32file.DRIVE_FIXED ##hardrive
True
>>> win32file.GetDriveType("Z:/") == win32file.DRIVE_FIXED ##network
False
>>> win32file.GetDriveType("D:/") == win32file.DRIVE_FIXED ##cd-rom
False
Thank you for your post - helped me with a ruby port. Method getDriveLetters returns hash (dict): drive letter string, drive type string.
require 'Win32API'
GetLogicalDrives = Win32API.new('kernel32', 'GetLogicalDrives', 'V', 'L')
GetDriveType = Win32API.new('kernel32', 'GetDriveType', 'P', 'I')
def GetDriveType(path)
GetDriveType.call(path)
end
def GetLogicalDrives()
GetLogicalDrives.call()
end
def getDriveLetters
drivetype = {
0 => 'DRIVE_UNKNOWN',
1 => 'DRIVE_NO_ROOT_DIR',
2 => 'DRIVE_REMOVABLE',
3 => 'DRIVE_FIXED',
4 => 'DRIVE_REMOTE',
5 => 'DRIVE_CDROM',
6 => 'DRIVE_RAMDISK'
}
drvs = []
n_drives = GetLogicalDrives()
for i in 0..25 do #check all drive letters
j = 2**i # bitmask for each letter
if n_drives & j > 0 then
drive = (65+i).chr + ":/"
drvs += [drive => drivetype[GetDriveType(drive)]]
end
end
return drvs
end
puts getDriveLetters