I try to read from a serial stream in python3 and convert a 2 bytes number in a stream into an integer. The stream is generated in C with
send_buffer[4] = (ostream.bytes_written >> 8) & 0xFF;
send_buffer[5] = (ostream.bytes_written >> 0) & 0xFF;
while True:
c = ser.read()
[ code to identify start sequence ]
if State == WAITING_FOR_SIZE_H:
data = c
print(c)
State = WAITING_FOR_SIZE_L
continue
if State == WAITING_FOR_SIZE_L:
data += c
print(c)
print(data)
data_size = struct.unpack('!h', data)[0]
print(data_size)
The serial data is 0 and 5. The result is:
H-Byte: b'\x00'
L-Byte: b'5'
data: b'\x005'
unpack: 53
So the 5 is not stored as hex value. When I try
data = b'\x00\x05'
It works:
data: b'\x00\x05'
unpack: 5
So how do I convert two bytes from the stream into an integer with struct?
Wrote test code to narrow it down to the from conversation when using ser.read() and concatenating the bytes value.
I have to implement a BCH error-correcting code. I have found some codes in Python BCH library Python and MATLAB BCH encoder in MATLAB. However, codes have different performance, BCH(127,70) in Python can correct up to 70 bitflips in a block size of 127. However, the MATLAB code can correct up to only 15 bits in 127 bits in BCH(127,15).
Why do these implementation perform differently?
Python Code
import bchlib
import hashlib
import os
import random
# create a bch object
BCH_POLYNOMIAL = 8219
BCH_BITS = 72
bch = bchlib.BCH(BCH_POLYNOMIAL, BCH_BITS)
# random data
data = bytearray(os.urandom(127))
# encode and make a "packet"
ecc = bch.encode(data)
packet = data + ecc
# print length of ecc, data, and packet
print('data size: %d' % (len(data)))
print('ecc size: %d' % (len(ecc)))
print('packet size: %d' % (len(packet)))
# print hash of packet
sha1_initial = hashlib.sha1(packet)
print('sha1: %s' % (sha1_initial.hexdigest(),))
def bitflip(packet):
byte_num = random.randint(0, len(packet) - 1)
bit_num = random.randint(0, 7)
packet[byte_num] ^= (1 << bit_num)
# make BCH_BITS errors
for _ in range(BCH_BITS):
bitflip(packet)
# print hash of packet
sha1_corrupt = hashlib.sha1(packet)
print('sha1: %s' % (sha1_corrupt.hexdigest(),))
# de-packetize
data, ecc = packet[:-bch.ecc_bytes], packet[-bch.ecc_bytes:]
# correct
bitflips = bch.decode_inplace(data, ecc)
print('bitflips: %d' % (bitflips))
# packetize
packet = data + ecc
# print hash of packet
sha1_corrected = hashlib.sha1(packet)
print('sha1: %s' % (sha1_corrected.hexdigest(),))
if sha1_initial.digest() == sha1_corrected.digest():
print('Corrected!')
else:
print('Failed')
This outputs
data size: 127
ecc size: 117
packet size: 244
sha1: 4ee71f947fc5d561b211a551c87fdef18a83404b
sha1: a072664312114fe59f5aa262bed853e35d70d349
bitflips: 72
sha1: 4ee71f947fc5d561b211a551c87fdef18a83404b
Corrected!
MATLAB code
%% bch params
M = 7;
n = 2^M-1; % Codeword length
k = 15; % Message length
nwords = 2; % Number of words to encode
% create a msg
msgTx = gf(randi([0 1],nwords,k));
%disp(msgTx)
%Find the error-correction capability.
t = bchnumerr(n,k)
% Encode the message.
enc = bchenc(msgTx,n,k);
%Corrupt up to t bits in each codeword.
noisycode = enc + randerr(nwords,n,1:t);
%Decode the noisy code.
msgRx = bchdec(noisycode,n,k);
% Validate that the message was properly decoded.
isequal(msgTx,msgRx)
which outputs:
t = 27
ans = logical 1
Increasing k>15 in MATLAB code gives following error:
Error using bchnumerr (line 72)
The values for N and K do not produce a valid narrow-sense BCH code.
Error in bchTest (line 10)
t = bchnumerr(n,k)
I discovered this question today (24 January 2021) as I searched for other information about BCH codes.
See Appendix A: Code Generators for BCH Codes (pdf) of Error-Correction Coding for Digital Communications by George C. Clark and J. Bibb Cain:
For n = 127 and k = 15, t = 27 is the number of errors that can be corrected.
For n = 127, the next option with larger k is k = 22 and t = 23.
Your use of the Python library is confusing. For standard usage of BCH codes, the length of a codeword is equal to 2m - 1 for some positive integer m. The codeword in your example is not of this form.
I have not used that Python library, so I cannot write with certainty. If ecc is of length 127, then I suspect that it is a codeword. Concatenating ecc and data yields a packet that has a copy of the original message data as well as a copy of the codeword. This is not how BCH codes are used. When you have the codeword, you don't need to send it and a separate copy of the original message.
If you do read the reference linked above, be aware of the notation used to describe the polynomials. For the n = 127 table, the polynomial g1(x) is denoted by 211, which is octal notation. The nonzero bits in the binary expressions indicate the nonzero coefficients of the polynomial.
octal: 211
binary: 010 001 001
polynomial: x7 + x3 + 1
The polynomial g2(x) is equal to g1(x) multiplied by another polynomial:
octal: 217
binary: 010 001 111
polynomial: x7 + x3 + x2 + x + 1
This means that
g2(x) = (x7 + x3 + 1)(x7 + x3 + x2 + x + 1)
Each gt+1(x) is equal to gt(x) multiplied by another polynomial.
I have been trying to get my head around CRC32 calculations without much success, the values that I seem to get do not match what I should get.
I am aware that Python has libraries that are capable of generating these checksums (namely zlib and binascii) but I do not have the luxury of being able to use them as the CRC functionality do not exist on the micropython.
So far I have the following code:
import binascii
import zlib
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0x000000ffL] ^ (value >> 8)
return value
teststring = "test"
print "binascii calc: 0x%08x" % (binascii.crc32(teststring) & 0xffffffff)
print "zlib calc: 0x%08x" % (zlib.crc32(teststring) & 0xffffffff)
print "my calc: 0x%08x" % (crc32(teststring))
Then I get the following output:
binascii calc: 0xd87f7e0c
zlib calc: 0xd87f7e0c
my calc: 0x2780810c
The binascii and zlib calculations agree where as my one doesn't. I believe the calculated table of bytes is correct as I have compared it to examples available on the net. So the issue must be the routine where each byte is calculated, could anyone point me in the correct direction?
Thanks in advance!
I haven't looked closely at your code, so I can't pinpoint the exact source of the error, but you can easily tweak it to get the desired output:
import binascii
from array import array
poly = 0xEDB88320
table = array('L')
for byte in range(256):
crc = 0
for bit in range(8):
if (byte ^ crc) & 1:
crc = (crc >> 1) ^ poly
else:
crc >>= 1
byte >>= 1
table.append(crc)
def crc32(string):
value = 0xffffffffL
for ch in string:
value = table[(ord(ch) ^ value) & 0xff] ^ (value >> 8)
return -1 - value
# test
data = (
'',
'test',
'hello world',
'1234',
'A long string to test CRC32 functions',
)
for s in data:
print repr(s)
a = binascii.crc32(s)
print '%08x' % (a & 0xffffffffL)
b = crc32(s)
print '%08x' % (b & 0xffffffffL)
print
output
''
00000000
00000000
'test'
d87f7e0c
d87f7e0c
'hello world'
0d4a1185
0d4a1185
'1234'
9be3e0a3
9be3e0a3
'A long string to test CRC32 functions'
d2d10e28
d2d10e28
Here are a couple more tests that verify that the tweaked crc32 gives the same result as binascii.crc32.
from random import seed, randrange
print 'Single byte tests...',
for i in range(256):
s = chr(i)
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
seed(42)
print 'Multi-byte tests...'
for width in range(2, 20):
print 'Width', width
r = range(width)
for n in range(1000):
s = ''.join([chr(randrange(256)) for i in r])
a = binascii.crc32(s) & 0xffffffffL
b = crc32(s) & 0xffffffffL
assert a == b, (repr(s), a, b)
print('ok')
output
Single byte tests... ok
Multi-byte tests...
Width 2
Width 3
Width 4
Width 5
Width 6
Width 7
Width 8
Width 9
Width 10
Width 11
Width 12
Width 13
Width 14
Width 15
Width 16
Width 17
Width 18
Width 19
ok
As discussed in the comments, the source of the error in the original code is that this CRC-32 algorithm inverts the initial crc buffer, and then inverts the final buffer contents. So value is initialised to 0xffffffff instead of zero, and we need to return value ^ 0xffffffff, which can also be written as ~value & 0xffffffff, i.e. invert value and then select the low-order 32 bits of the result.
If using binary data where the crc is chained over multiple buffers I used the following (using the OPs table):
def crc32(data, crc=0xffffffff):
for b in data:
crc = table[(b ^ crc) & 0xff] ^ (crc >> 8)
return crc
One can XOR the final result with -1 to agree with the online calculators.
crc = crc32(b'test')
print('0x{:08x}'.format(crc))
crc = crc32(b'te')
crc = crc32(b'st', crc)
print('0x{:08x}'.format(crc))
print('xor: 0x{:08x}'.format(crc ^ 0xffffffff))
output
0x278081f3
0x278081f3
xor: 0xd87f7e0c
I have a problem with my code.
I need to receive a MSB and LSB 10 bits data from a PICAXE KIT, but i dont know how i can convert the MSB ans LSB on integer that can i use.
I try that, but dont works
for x in range(1,400,10):
MSB = ser.read(2)
LSB = ser.read(2)
datos = (MSB*256)+LSB
print datos
degree = float(int(datos,2))
linea = (432,200)
linea_len = 100
x = linea[0] + math.cos(math.radians(degree)) * linea_len
y = linea[1] + math.sin(math.radians(degree)) * linea_len
How can i receive the MSB and LSB from picaxe by serial??
Thanks you
How i can create a frame icmp
I mark malformet packet on python
import socket, struct, uuid, subprocess, fcntl, time,random
List item
from binascii import hexlify, unhexlify from datetime import datetime
tarRed = raw_input('Ingrese nombre de Tarjeta de Red Utilizada: ')
subprocess.call(['ifconfig', tarRed, 'promisc']) pt = 0x0800 s =
socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(pt))
s.bind((tarRed, pt))
#GETCheck def getchecksum(ip_header,size): cksum = 0 pointer = 0 while size > 1: cksum += int((str("%02x" % (ip_header[pointer],))
+ str("%02x" % (ip_header[pointer+1],))), 16) size -= 2 pointer += 2 #if size: #This accounts for a situation where the header is odd
#cksum += int(ip_header[pointer]) cksum = (cksum >> 16) +
(cksum & 0xffff) cksum += (cksum >>16) return (~cksum) &
0xFFFF def _checksum(data): #calculate the header sum
ip_header_sum = sum(struct.unpack_from("6H", data)) #add the carry
ip_header_sum = (ip_header_sum & 0xFFFF) + (ip_header_sum >> 8 &
0xFFFF) ip_header_sum = ~ip_header_sum & 0xFFFF return
ip_header_sum def i_checksum(checksum_packet): total = 0 num_words
= len(checksum_packet) / 2 for chunk in struct.unpack("!%sH" % num_words, checksum_packet[0:num_words*2]): total += chunk if
len(checksum_packet) % 2: total += ord(checksum_packet[-1]) << 8
total += total >> 16 return (~total + 0xffff & 0xffff)
def checksum(source_string): sum = 0 countTo =
(len(source_string)/2)*2 count = 0 while count<countTo:
thisVal = ord(source_string[count + 1])*256 + ord(source_string[count]) sum = sum + thisVal sum = sum &
0xffffffff
count = count + 2 if countTo<len(source_string): sum = sum + ord(source_string[len(source_string) - 1]) sum = sum &
0xffffffff sum = (sum >> 16) + (sum & 0xffff) sum = sum + (sum
>> 16) answer = ~sum answer = answer & 0xffff #answer = answer >> 8 | (answer << 8 & 0xff00) return answer def
checksuma(source_string): sum = 0 countTo =
(len(source_string)/2)*2 count = 0 while count<countTo:
thisVal = ord(source_string[count + 1])*256 + ord(source_string[count]) sum = sum + thisVal sum = sum &
0xffffffff
count = count + 2 if countTo<len(source_string): sum = sum + ord(source_string[len(source_string) - 1]) sum = sum &
0xffffffff sum = (sum >> 16) + (sum & 0xffff) sum = sum + (sum
>> 16) answer = ~sum answer = answer & 0xffff answer = answer >> 8 | (answer << 8 & 0xff00) return answer def ultimo(str):
csum = 0
countTo = (len(str) / 2) * 2
count = 0
while count < countTo:
thisVal = ord(str[count+1]) * 256 + ord(str[count])
csum = csum + thisVal
csum = csum & 0xffffffff
count = count + 2
if countTo < len(str):
csum = csum + ord(str[len(str) - 1])
csum = csum & 0xffffffff
csum = (csum >> 16) + (csum & 0xffff)
csum = csum + (csum >> 16)
answer = ~csum
answer = answer & 0xffff
answer = answer >> 8 | (answer << 8 & 0xff00)
return answer
#GETCheck
#MACs adst='ffffffffffff' mac_destino = unhexlify(adst) ma = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) info =
fcntl.ioctl(ma.fileno(), 0x8927, struct.pack('256s', tarRed[:15]))
asrc = ''.join(['%02x:' % ord(char) for char in info[18:24]])[:-1]
sd = asrc.split(':') asrc = sd[0]+sd[1]+sd[2]+sd[3]+sd[4]+sd[5]
mac_origen = unhexlify(asrc)
#MACs cabEther = struct.pack('!6s6sh',mac_destino,mac_origen,pt)
#datosIP
#version = '4'
#IHL = '5' tipoServicio = unhexlify('00') longitudT = struct.pack('!BB',00,24) identificador = struct.pack('!BB',00,01)
flag_Pos = 0x4000 tiempoVida = 05 ptIP = unhexlify('01') SCC = 0
#IPs ip = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) ipO= str( socket.inet_ntoa(fcntl.ioctl(ip.fileno(),0x8915,struct.pack('256s',
tarRed[:15]))[20:24])) ipO1 = ipO.split('.') ipOr =
struct.pack("!BBBB",int(ipO1[0]),int(ipO1[1]),int(ipO1[2]),int(ipO1[3]))
ipD = raw_input('Ingrese la direccion IP destino: ') ipD1 =
ipD.split('.') ipDest =
struct.pack("!BBBB",int(ipD1[0]),int(ipD1[1]),int(ipD1[2]),int(ipD1[3]))
#IPs
#datosIP cabIP = struct.pack('!1s1s2s2shB1sH4s4s',unhexlify('45'),tipoServicio,longitudT,identificador,flag_Pos,tiempoVida,ptIP,SCC,ipOr,ipDest)
SCC = checksum(cabIP) cabIP =
struct.pack('!1s1s2s2shB1sH4s4s',unhexlify('45'),tipoServicio,longitudT,identificador,flag_Pos,tiempoVida,ptIP,socket.htons(SCC),ipOr,ipDest)
#datosIcmp tipo = 8 codigo = 0x00 check = 0x0000 identificador = int((id(1) * random.random()) % 65535) secuencia = 0x0000
#datos = 'qwertyasdfghzxcvbn0102030405060708091011121314151617181920' datos =
1 * 'Q'
#datosIcmp cabIcmp = struct.pack('!bbHHh',tipo,0,0,identificador,1) my_checksum = ultimo(cabIcmp + datos) cabIcmp =
struct.pack('!bbHHh',tipo,0,socket.htons(my_checksum),identificador,1)
cabIcmp = cabIcmp + datos tiempo = datetime.now() print tiempo for i
in range(15): s.send(cabEther + cabIP + cabIcmp,0)
Your question is barely understandable. It appears you're asking how to create an ICMP ECHO packet since the packet you are creating in the listed code is reported as malformed by your system. I'll try to answer why your code won't work.
First, the socket type you're using is incorrect. You need type 1, not 8. This can be done easily enough with the third parameter:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
(Note you'll need root or administrator privileges to make this call.)
Second, after creating the socket, you'll need to create the ICMP header. An ICMP header consists of the following structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Type | Code | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | REST OF HEADER |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Notice the second four bytes "REST OF HEADER". This second set of four bytes depends on the TYPE/CODE in the first two bytes. For example, if TYPE is 0 and CODE is 0, it means you're dealing with an echo reply, and as such, the second four bytes will become 2-byte Identifier and 2-byte Sequence Number fields, starting at offset 4 and ending at offset 7 as the picture shows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Type | Code | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | Identifier | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
8 | Data ...
+-+-+-+-+-
To create this header, I'm going to use type 8, the echo request structure, as such (note that bytes 4-5 and 6-7 in this creation is just for illustration, typically this would be different values other than zero):
icmp = struct.pack(">BBHHH", 8, 0, 0, 0, 0)
On this initial creation of the header, the checksum is zero. What we now need is to actually calculate the checksum (with this zero value), and THEN recreate the header with this new checksum value:
icmp = struct.pack(">BBHHH", 8, 0, cksum(icmp), 0, 0)
But this is where your code is incorrect again. The implementation of i_checksum() is wrong. To fix, I'll leave it to you to figure it out. The checksum is called 1's complement, and there are numerous online articles about it, as well as C code on Linux systems that implement it.
Once you have the header and its correct checksum (important because your system won't send the packet if incorrect), you simply then send it somewhere:
s.sendto(icmp, (SOME_REMOTE_HOST, 0))
And then receive replies if any:
s.recvfrom(1500)
(Note I chose 1500 bytes as that is the MTU of an IP frame, and it is thus unlikely a reply will be bigger than this, though it is certainly possible.)
AND AT THIS POINT, YOUR CODE TOTALLY FAILS -- simply because you do not have code which does the last two steps, let alone additional code to handle what to do with replies and struct-parsing of the header fields.
As for the other two functions you've defined, they're odd. One of them seems to imply it is dealing with link layer fields -- which is never the case on sockets as sockets only deal in IP layer and above -- while the other seems to be an attempt at calculating a checksum...albeit incorrectly (again).
In short, as abarnert implied, please clarify (and add some more) code AND your question.