We Keep Coding

Verifying CRC32 of UDP given .jpg file of payload

I'm running a server that receives UDP packets that contain a 2 byte CRC32 polynomial and a variable number of XOR'd DWORDs corresponding to a .jpg file. The packets also contain the index of the corresponding DWORD in the .jpg file for each DWORD in the packet. I am also given the actual .jpg file.
For example, the packet could contain 10 DWORDs and specify the starting index as 3, so we can expect the received DWORDs to correspond with the 4th through 11th DWORDs making up the .jpg.
I want to verify the integrity of each of the DWORDs by comparing their CRC32 values against the CRC32 values of the corresponding DWORDs in the .jpg.
I thought that the proper way to do this would be to divide each DWORD in the packet and its corresponding DWORD in the .jpg by the provided CRC polynomial and analyze the remainder. If the remainders are the same after doing these divisions, then there is no problem with the packet. However, even with packets that are guaranteed to be correct, these remainders are never equal.
Here is how I'm reading the bytes of the actual .jpg and splitting them up into DWORDs:
def split(data):
# Split the .jpg data into DWORDs
chunks = []
for i in range(0, len(data), 4):
chunks.append(data[i: i + 4])
return chunks
def get_image_bytes():
with open("dog.jpg", "rb") as image:
f = image.read()
jpg_bytes = split(f)
return jpg_bytes
Now I have verified my split() function works and to my knowledge, get_image_bytes() reads the .jpg correctly by calling image.read().
After receiving a packet, I convert each DWORD to binary and perform the mod 2 division like so:
jpg_bytes = get_image_bytes()
crc_key_bin = '1000110111100' # binary representation of the received CRC32 polynomial
d_words = [b'\xc3\xd4)v', ... , b'a4\x96\xbb']
iteration = 0 # For simplicity, assume the packet specified that the starting index is 0
for d in d_words:
d_bin = format(int(d.hex(), 16), "b") # binary representation of the DWORD from the packet
jpg_dword = format(int(jpg_bytes[iteration].hex(), 16), "b") # binary representation of the corresponding DWORD in dog.jpg
remainder1 = mod2div(d_bin, crc_key_bin) # <--- These remainders should be
remainder2 = mod2div(jpg_dword, crc_key_bin) # <--- equal, but they're not!
iteration += 1
I have tested the mod2div() function, and it returns the expected remainder after performing mod 2 division.
Where am I going wrong? I'm expecting the 2 remainders to be equal, but they never are. I'm not sure if the way I'm reading the bytes from the .jpg file is incorrect, if I'm performing the mod 2 division with the wrong values, or if I'm completely misunderstanding how to verify the CRC32 values. I'd appreciate any help.

First off, there's no such thing as a "2 byte CRC32 polynomial". A 32-bit CRC needs 32-bits to specify the polynomial.
Second, a CRC polynomial is something that is fixed for a given protocol. Why is a CRC polynomial being transmitted, as opposed to simply specified? Are you sure it's the polynomial? Where is this all documented?
What does "XOR'd DWORDs" means? Exclusive-or'd with what?
And, yes, I think you are completely misunderstanding how to verify CRC values. All you need to do is calculate the check values on the message the same way it was done at the other end, and compare that to the check values that were transmitted. (That is true for any check value, not just CRCs.) However I cannot tell from your description what was calculated on what, or how.

python- reading certain number of bytes from a variable

I'm trying to implement the AES algorithm, for which the message is to be divided into b-blocks each of 1byte(AES-128 would require 1 byte per state cell). So, if the message is: "This is saturday, and it is time to tell tale.", I'd have to read 1 byte out of this and store it in a state cell.
So, my first problem is, Is it possible to read(or extract) a certain number of bytes from a variable?
And the problem that follows immediately is, "if it is possible to get a certain number of bytes from a variable, then, how do we obtain the bits in that byte?"

had to do that just recently. this is an option:
from itertools import islice
byteorder = 'big'
plain = b"This is saturday, and it is time to tell tale."
def chunked(iterable, n):
it = iter(iterable)
values = bytes(islice(it, n))
while values:
yield values
values = bytes(islice(it, n))
for block_bytes in chunked(plain, n=8):
block_int = int.from_bytes(block_bytes, byteorder)
print(block_bytes, bin(block_int))
which outputs
b'This is ' 0b101010001101000011010010111001100100000011010010111001100100000
b'saturday' 0b111001101100001011101000111010101110010011001000110000101111001
b', and it' 0b10110000100000011000010110111001100100001000000110100101110100
b' is time' 0b10000001101001011100110010000001110100011010010110110101100101
b' to tell' 0b10000001110100011011110010000001110100011001010110110001101100
b' tale.' 0b1000000111010001100001011011000110010100101110
note that the byteorder can be 'little' as well.
from block_int it is easy to obtain the individual bits: e.g. the least-significant bit is block_int & 1; for bits in other positions you can shift: (block_int >> 5) & 1 etc. or you get the desired byte from block_bytes (which is an array of ints) and select the bit you want; e.g. (block_bytes[4] >> 7) & 1.
maybe this answer ia also helpful.

Converting bytes to signed numbers in Python

I am faced with a problem in Python and I think I don't understand how signed numbers are handled in Python. My logic works in Java where everything is signed so need some help in Python.
I have some bytes that are coded in HEX and I need to decode them and interpret them to numbers. The protocol are defined.
Say the input may look like:
raw = '016402570389FFCF008F1205DB2206CA'
And I decode like this:
bin_bytes = binascii.a2b_hex(raw)
lsb = bin_bytes[5] & 0xff
msb = bin_bytes[6] << 8
aNumber = int(lsb | msb)
print(" X: " + str(aNumber / 4000.0))
After dividing by 4000.0, X can be in a range of -0.000025 to +0.25.
This logic works when X is in positive range. When X is expected
to be negative, I am getting back a positive number.
I think I am not handling "msb" correctly when it is a signed number.
How should I handlehandle negative signed number in
Python?
Any tips much appreciated.

You can use Python's struct module to convert the byte string to integers. It takes care of endianness and sign extension for you. I guess you are trying to interpret this 16-byte string as 8 2-byte signed integers, in big-endian byte order. The format string for this is '>8h. The > character tells Python to interpret the string as big endian, 8 means 8 of the following data type, and h means signed short integers.
import struct
nums = struct.unpack('>8h', bin_bytes)
Now nums is a tuple of integers that you can process further.
I'm not quite sure if your data is little or big endian. If it is little-endian, you can use < to indicate that in the struct.unpack format string.

Can i send a 2 byte size variable as 1 byte size variable? [closed]

Closed. This question needs details or clarity. It is not currently accepting answers.
Want to improve this question? Add details and clarify the problem by editing this post.
Closed 7 years ago.
Improve this question
I am working on some ADC(Analog-to-Digital) value conversions and need to send those values over the UART but due to the speed and other size limitations I want to convert those to one byte size
The ADC is 12 bit for a range 0.0 to 5.0.
Currently I am splitting the 2 byte variable to two separate bytes and joining them back in the receiver section
Sending (C)
// i am trying to send 0xFF8 (4088) which is read from that register
send[0] = (VADC_G5RES4.B.RESULT) >> 8 ;
send[1] = (VADC_G5RES4.B.RESULT) & 0x0FF;
.
send[30] = .....
Receiving (Python)
adc1 = data[3]
adc1 = adc1 << 8 | data[4]
adc1 = (float(adc1 * 5) / 4096)
# gives me 4.990234 as the value
Is there any way that i can send the value in just one byte (maybe by performing the calculation on the sending side)
Only 2 digits after the decimal (4.99) would be fine

If there were some magic way to stuff two-byte values into one-byte values, we would do it to everything, and then we would do it again, and we would keep doing it and keep halving the size of everything until we were shoving all the world's information into a single byte. Such magic is not possible.
If you instead throw away half your data, you can reduce the size of things, but you say you want at least 2 digits after the decimal point. A byte can only encode 256 different values, and encoding everything from 0.00 to 9.99 takes 1000 different values, so you still can't do it.
UPDATE: With the additional information that these are only 12-bit values, rather than 16-bit, and that the maximum value is 5.0, we can do a lot better, though still not quite 3 significant figures in 8 bits.
First, instead of wasting 4 bits per value, we can pack 2 values into 3 bytes. That'd look something like
def pack(val1, val2):
byte1 = val1 >> 4
byte2 = ((val1 & 0xf) << 4) | (val2 >> 8)
byte3 = val2 & 0xff
return byte1, byte2, byte3
You might need to deal with a trailing half-byte.
If 1.5 bytes per value is still too much, there are lossy encoding options. For example, we can just throw away the 4 least-significant bits of every value:
encoded = val >> 4
and decode appropriately:
decoded = encoded * 5.0 / 256
This keeps a precision of 5/256, or about 0.02. Other encodings might perform better, depending on what you expect the signal to look like. For example, chux's solution encodes values exactly as long as the values don't change too fast, but only has a precision of about 0.04 when the values are changing quickly.

Use temporal compression.
Trade time for accuracy. When the ADC is changing fast, sent a 7-bit approximation. When it changes slow, send the delta.
The following is a coarse implementation idea.
Assume 2-byte ADC value in the ranges 0-3FFF.
sample = ADC();
encode_value = sample >> 7;
send(encode_value);
diff = sample - (encode_value << 7);
loop:
previous_sample = sample
sample = ADC();
diff = previous_sample - sample;
if (diff >= -64 && diff <= 63) {
send(diff | 0x80);
diff = 0;
} else {
encode_value = sample >> 7;
send(encode_value);
diff = sample - (encode_value << 7);
}
// Receive
valid = 0;
loop:
sample = receive()
if (sample & 0x80) {
delta = sample & 0x7F
if (delta & 0x40) delta -= 128;
ADC += delta;
} else {
ADC = sample << 9;
valid = 1;
}
Likely should reserve another bit to detect a missed sequence when sending a delta. Other considerations apply, yet here it is to give the OP another point of view.
Refinements include having the delta give up 2 bits of mantissa for a 2-bit exponent.
[Edit]
12 bits of random data do not fit in 8 bits. Some ideas:
Give up precision. Simple divide the 12-bit value by 16, send those 8 bits and multiply by 16 on the receiving end.
Send the first A/D sample in 2 parts by cutting the 12-bit value into 2 6-bits halves. Use the MSB to distinguish if the upper or lower is sent. The receiving end needs 2 samples to put things back together. The 2nd A/D sample on the transmits end is thrown away.
Like #2, but send 2 out of every 3 samples. 24-bits of data in 3 8-bit messages.
As answered in the beginning of this answer. Sometimes sending the course value, other times, the delta.
As commented below #Clifford always send a signed 8-bit delta. Might need a minor adjustment to insure any bias in the receiving sum eventually works its way out.
Take a couple 1000 (million) samples and write it as packed 12-bit data into a file. Zip (compress) the file. What every the ration of compression is found, is an indicator of the best compression scheme we could derive. If it is not at least 33%, then you best to accept that data is too dynamic to be completely transmitted with the requirements given.

It cannot be done without loosing bits and therefore precision. Essentially you would need treat your ADC as if it were 8 bit. That may not be as bad as it sounds, for example, if your ADC is 12 bit (not clear in your question), you loose just 4 bits.
uint8_t send = (adc_high_byte << 6) | (adc_low_byte >> 2) ;
then in the receiver:
adc1 = (data * 5.0) / 256 ;
In your example 4088 is sent as 4088/16 = 255 and at the receiver it is transformed:
(255 * 5.0) / 256 = 4.98
but note that
(254 * 5.0) / 256= 4.96,
so you end up with a precision of approximately 0.02 (precisely speaking, if your maximum value represented by 255 is 5.0, then the precision is 5.0/256 = 0.01953125). This compares to the original precision of 5/4096 = 0.00122
A further refinement is possible using companding. In some applications, such as audio, it is more important to retain detail a small signal levels than high, so you can transform the 12 bit data to 8 using a non-linear encoding so that the difference between say 0 and 1 is much smaller than between 254 and 255. The applicability and success of this will be application dependent - it works well for audio. In other applications a different non-linear encoding may be appropriate, for example for battery monitoring you might use high resolution at the knee of the discharge curve, where rate of voltage drop increases rapidly (in some battery chemistries).
Finally as suggested by #chux, you can transmit delta (changes in level) in fewer bits and retain the full resolution (with some loss of absolute accuracy), but that was his suggestion so I won't elaborate, except to say that the technique can be used with companding.

If I correctly deciphered your problem, it seems that:
you have values in the range 0x000 - 0xfff
you will make some math with those values ((val * 5) / 4096 ) and you will not care about what comes after the first two decimals after the comma of the resulting number
If this is your question, you can first of all verify how much information you lose if you throw away the least significant hex digit from your values after the math:
>>> (0x010 * 5) / 4096
0.01953125
This means that if you spare it, you will have an error of ~0.02.
If you are ok with that, you can then encode your value in a single byte.
You just need to right-shift by 4 your value before sending:
0 f f 8 >> 4 = [ 0 0 ] f f
0000 1111 1111 1000 [ 0000 0000]1111 1111
That is possible because you already had a vacant leading half-byte, and
are now freeing another half at the end.
On the receiving side, you will left-shift it again and then do the calculations:
>>> b = 0xff << 4
>>> (b * 5) / 4096
4.98046875

You can convert those in the sending side and multiply with 2 and join a '.' at the 2nd position.. This method works( but not very accurate)
(4088*5)/4096 = 4.990234
use the first 3 digits and remove the decimal point
499
divide the value by 2 so that it can be sent using one signed byte (0-255)
499/2 = 249
send the value using your regular data transfer method
send[0] = 249
On receiving section multiply the value with 2 and add a decimal point
adc1 = 249*2
newvalue= adc1[:1]+'.'+adc1[1:]
# you get newvalue = 4.98 ( But you loose the accuracy of the converted adc values )

How do I convert 32b (four characters) from an int value to an ASCII string in python

Hi I have a 32b value that I need to easily truncate in to it's four bytes, convert each byte to ASCII and combine them to a four letter string. And I also need the reverse process. I have been able to do this in one direction in the following ugly way:
## the variable "binword" is a 32 bit value read directly from an MCU, where each byte is an
## ASCII character
char0 = (binword & 0xFF000000) >> 24
char1 = (binword & 0xFF0000) >> 16
char2 = (binword & 0xFF00) >> 8
char3 = (binword & 0xFF)
fourLetterWord = str(unichr(char0))+str(unichr(char1))+str(unichr(char2))+str(unichr(char3))
Now, I find this method really un-elegant and time consuming, so the question is how do I do this better? And, I guess the more important question, how do I convert the other way?

You should use the struct module's pack and unpack calls for these convertions
number = 32424234
import struct
result = struct.pack("I", number)
and back:
number = struct.unpack("I", result)[0]
Please, refer to the official docs on the struct module for the struct-string syntax,
and markers to ensure endiannes, and number size.
https://docs.python.org/2/library/struct.html
On a side note - this is by no way "ASCII" - it is a bytestring.
ASCII refers to a particular text encoding with codes on the 32-127 numeric range.
The point is that you should not think on bytestrings as text, if you need a stream of bytes - and much less think of "ASCII" as an alias for text strings - as it can represent less than 1% of textual characters existing in the World.