We Keep Coding

Separate Data From Gyroscope Serial Port, Python

I have a Bluetooth gyroscope that I only want accelerometer data from. When I open the port the data will come in as a single, jumbled stream, right? How do I grab the data that I want? I want to simulate a keypress if acceleration is over a certain value, if that helps.

Since the data packet is 11 bytes, read 11 bytes at a time from the port, then parse the message. Note, you may want to read 1 byte at a time until you get a start-message byte (0x55) then read the following 10.
# data is a byte array, len = 11
def parse_packet(data):
# Verify message
if data[0] == 0x55 and data[1] == 0x51 and len(data) == 11:
# Verify checksum
if ((sum(data) - data[10]) & 0xFF) == data[10]:
g = 9.8 # Gravity
parsed = {}
parsed["Ax"] = ((data[3] << 8) | data[2]) / 32768.0 * 16 * g
parsed["Ay"] = ((data[5] << 8) | data[4]) / 32768.0 * 16 * g
parsed["Az"] = ((data[7] << 8) | data[6]) / 32768.0 * 16 * g
# Temp in in degrees celsius
parsed["Temp"] = ((data[9] << 8) | data[8]) / 340.0 + 36.53
return parsed
return None
The one thing you need to verify is the checksum calculation. I couldn't find it in the manual. The other calculations came from the manual I found here: https://www.manualslib.com/manual/1303256/Elecmaster-Jy-61-Series.html?page=9#manual

How to find Run length encoding in python

I have an array ar = [2,2,2,1,1,2,2,3,3,3,3].
For this array, I want to find the lengths of consecutive same numbers like:
values: 2, 1, 2, 3
lengths: 3, 2, 2, 4
In R, this is obtained by using rle() function. Is there any existing function in python which provides required output?

You can do this with groupby
from itertools import groupby
ar = [2,2,2,1,1,2,2,3,3,3,3]
print([(k, sum(1 for i in g)) for k,g in groupby(ar)])
# [(2, 3), (1, 2), (2, 2), (3, 4)]

Here is an answer for pure numpy:
import numpy as np
def find_runs(x):
"""Find runs of consecutive items in an array."""
# ensure array
x = np.asanyarray(x)
if x.ndim != 1:
raise ValueError('only 1D array supported')
n = x.shape[0]
# handle empty array
if n == 0:
return np.array([]), np.array([]), np.array([])
else:
# find run starts
loc_run_start = np.empty(n, dtype=bool)
loc_run_start[0] = True
np.not_equal(x[:-1], x[1:], out=loc_run_start[1:])
run_starts = np.nonzero(loc_run_start)[0]
# find run values
run_values = x[loc_run_start]
# find run lengths
run_lengths = np.diff(np.append(run_starts, n))
return run_values, run_starts, run_lengths
Credit goes to https://github.com/alimanfoo

Here is an answer using the high-performance pyrle library for run length arithmetic:
# pip install pyrle
# (pyrle >= 0.0.25)
from pyrle import Rle
v = [2,2,2,1,1,2,2,3,3,3,3]
r = Rle(v)
print(r)
# +--------+-----+-----+-----+-----+
# | Runs | 3 | 2 | 2 | 4 |
# |--------+-----+-----+-----+-----|
# | Values | 2 | 1 | 2 | 3 |
# +--------+-----+-----+-----+-----+
# Rle of length 11 containing 4 elements
print(r[4])
# 1.0
print(r[4:7])
# +--------+-----+-----+
# | Runs | 1 | 2 |
# |--------+-----+-----|
# | Values | 1.0 | 2.0 |
# +--------+-----+-----+
# Rle of length 3 containing 2 elements
r + r + 0.5
# +--------+-----+-----+-----+-----+
# | Runs | 3 | 2 | 2 | 4 |
# |--------+-----+-----+-----+-----|
# | Values | 4.5 | 2.5 | 4.5 | 6.5 |
# +--------+-----+-----+-----+-----+
# Rle of length 11 containing 4 elements

Here is an optimized answer using numpy arrays which runs quickly if the run lengths are long.
In this case I want to encode an array of uint16 that can be much
larger than 2**16 using 16 bit unsigned integer run length encoding.
To allow this the array is "chunked" so the indices never exceed
2**16:
import numpy as np
def run_length_encode(array, chunksize=((1 << 16) - 1), dtype=np.int16):
"Chunked run length encoding for very large arrays containing smallish values."
shape = array.shape
ravelled = array.ravel()
length = len(ravelled)
chunk_cursor = 0
runlength_chunks = []
while chunk_cursor < length:
chunk_end = chunk_cursor + chunksize
chunk = ravelled[chunk_cursor : chunk_end]
chunk_length = len(chunk)
change = (chunk[:-1] != chunk[1:])
change_indices = np.nonzero(change)[0]
nchanges = len(change_indices)
cursor = 0
runlengths = np.zeros((nchanges + 1, 2), dtype=dtype)
for (count, index) in enumerate(change_indices):
next_cursor = index + 1
runlengths[count, 0] = chunk[cursor] # value
runlengths[count, 1] = next_cursor - cursor # run length
cursor = next_cursor
# last run
runlengths[nchanges, 0] = chunk[cursor]
runlengths[nchanges, 1] = chunk_length - cursor
runlength_chunks.append(runlengths)
chunk_cursor = chunk_end
all_runlengths = np.vstack(runlength_chunks).astype(dtype)
description = dict(
shape=shape,
runlengths=all_runlengths,
dtype=dtype,
)
return description
def run_length_decode(description):
dtype = description["dtype"]
runlengths = description["runlengths"]
shape = description["shape"]
array = np.zeros(shape, dtype=dtype)
ravelled = array.ravel()
cursor = 0
for (value, size) in runlengths:
run_end = cursor + size
ravelled[cursor : run_end] = value
cursor = run_end
array = ravelled.reshape(shape) # redundant?
return array
def testing():
A = np.zeros((50,), dtype=np.uint16)
A[20:30] = 10
A[30:35] = 6
A[40:] = 3
test = run_length_encode(A, chunksize=17)
B = run_length_decode(test)
assert np.alltrue(A == B)
print ("ok!")
if __name__=="__main__":
testing()
I built this for a project having to do with classifying
microscopy images of mouse embryos.
https://github.com/flatironinstitute/mouse_embryo_labeller
Note: I edited the entry after I found I had to caste the type
in this line to it to work for large arrays:
all_runlengths = np.vstack(runlength_chunks).astype(dtype)

Python - reading 10 bit integers from a binary file

I have a binary file containing a stream of 10-bit integers. I want to read it and store the values in a list.
It is working with the following code, which reads my_file and fills pixels with integer values:
file = open("my_file", "rb")
pixels = []
new10bitsByte = ""
try:
byte = file.read(1)
while byte:
bits = bin(ord(byte))[2:].rjust(8, '0')
for bit in reversed(bits):
new10bitsByte += bit
if len(new10bitsByte) == 10:
pixels.append(int(new10bitsByte[::-1], 2))
new10bitsByte = ""
byte = file.read(1)
finally:
file.close()
It doesn't seem very elegant to read the bytes into bits, and read it back into "10-bit" bytes. Is there a better way to do it?
With 8 or 16 bit integers I could just use file.read(size) and convert the result to an int directly. But here, as each value is stored in 1.25 bytes, I would need something like file.read(1.25)...

Here's a generator that does the bit operations without using text string conversions. Hopefully, it's a little more efficient. :)
To test it, I write all the numbers in range(1024) to a BytesIO stream, which behaves like a binary file.
from io import BytesIO
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if len(b) == 0:
break
n = int.from_bytes(b, 'big')
#Split n into 4 10 bit integers
t = []
for i in range(4):
t.append(n & 0x3ff)
n >>= 10
yield from reversed(t)
# Make some test data: all the integers in range(1024),
# and save it to a byte stream
buff = BytesIO()
maxi = 1024
n = 0
for i in range(maxi):
n = (n << 10) | i
#Convert the 40 bit integer to 5 bytes & write them
if i % 4 == 3:
buff.write(n.to_bytes(5, 'big'))
n = 0
# Rewind the stream so we can read from it
buff.seek(0)
# Read the data in 10 bit chunks
a = list(tenbitread(buff))
# Check it
print(a == list(range(maxi)))
output
True
Doing list(tenbitread(buff)) is the simplest way to turn the generator output into a list, but you can easily iterate over the values instead, eg
for v in tenbitread(buff):
or
for i, v in enumerate(tenbitread(buff)):
if you want indices as well as the data values.
Here's a little-endian version of the generator which gives the same results as your code.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
for i in range(4):
yield n & 0x3ff
n >>= 10
We can improve this version slightly by "un-rolling" that for loop, which lets us get rid of the final masking and shifting operations.
def tenbitread(f):
''' Generate 10 bit (unsigned) integers from a binary file '''
while True:
b = f.read(5)
if not len(b):
break
n = int.from_bytes(b, 'little')
#Split n into 4 10 bit integers
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n & 0x3ff
n >>= 10
yield n
This should give a little more speed...

As there is no direct way to read a file x-bit by x-bit in Python, we have to read it byte by byte. Following MisterMiyagi and PM 2Ring's suggestions I modified my code to read the file by 5 byte chunks (i.e. 40 bits) and then split the resulting string into 4 10-bit numbers, instead of looping over the bits individually. It turned out to be twice as fast as my previous code.
file = open("my_file", "rb")
pixels = []
exit_loop = False
try:
while not exit_loop:
# Read 5 consecutive bytes into fiveBytesString
fiveBytesString = ""
for i in range(5):
byte = file.read(1)
if not byte:
exit_loop = True
break
byteString = format(ord(byte), '08b')
fiveBytesString += byteString[::-1]
# Split fiveBytesString into 4 10-bit numbers, and add them to pixels
pixels.extend([int(fiveBytesString[i:i+10][::-1], 2) for i in range(0, 40, 10) if len(fiveBytesString[i:i+10]) > 0])
finally:
file.close()

Adding a Numpy based solution suitable for unpacking large 10-bit packed byte buffers like the ones you might receive from AVT and FLIR cameras.
This is a 10-bit version of #cyrilgaudefroy's answer to a similar question; there you can also find a Numba alternative capable of yielding an additional speed increase.
import numpy as np
def read_uint10(byte_buf):
data = np.frombuffer(byte_buf, dtype=np.uint8)
# 5 bytes contain 4 10-bit pixels (5x8 == 4x10)
b1, b2, b3, b4, b5 = np.reshape(data, (data.shape[0]//5, 5)).astype(np.uint16).T
o1 = (b1 << 2) + (b2 >> 6)
o2 = ((b2 % 64) << 4) + (b3 >> 4)
o3 = ((b3 % 16) << 6) + (b4 >> 2)
o4 = ((b4 % 4) << 8) + b5
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), 4*o1.shape[0])
return unpacked
Reshape can be omitted if returning a buffer instead of a Numpy array:
unpacked = np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1).tobytes()
Or if image dimensions are known it can be reshaped directly, e.g.:
unpacked = np.reshape(np.concatenate((o1[:, None], o2[:, None], o3[:, None], o4[:, None]), axis=1), (1024, 1024))
If the use of the modulus operator appears confusing, try playing around with:
np.unpackbits(np.array([255%64], dtype=np.uint8))
Edit: It turns out that the Allied Vision Mako-U cameras employ a different ordering than the one I originally suggested above:
o1 = ((b2 % 4) << 8) + b1
o2 = ((b3 % 16) << 6) + (b2 >> 2)
o3 = ((b4 % 64) << 4) + (b3 >> 4)
o4 = (b5 << 2) + (b4 >> 6)
So you might have to test different orders if images come out looking wonky initially for your specific setup.

Extract bitfields from an int in Python

I have a number like 0x5423 where I want to extract 4 values:
a = 0x5 # 15 downto 12
b = 0x42 # 11 downto 3
c = 0x3 # 3 downto 2
d = 0x00 # 1 downto 0
I discovered the module bitstrings that looks great. Unfortunately, for an unknown reason, the bits are numbered from the right.
This is bad because if a add some upper bits like 0xA5423 my extraction won't work anymore:
field = bitstrings.BitArray('0x5423')
a = field[0:4].uint
b = field[4:12].uint
c = field[12:14].uint
d = field[14:16].uint
How can I properly extract my bitfields without complex arithmetic manipulations such as:
b = (a >> 4) & 0xFF
Ideally I would have:
b = field.range(11, 4)

Convert the string to 0x#### format before pass to bitstring.BitArray:
>>> n = '0xA5423'
>>> n = '0x{:04x}'.format(int(n, 16) & 0xffff) # => '0x5423'
>>> field = bitstring.BitArray(n)
>>> field[0:4].uint
5
>>> field[4:12].uint # 0x42 == 66
66
>>> field[12:14].uint
0
>>> field[14:16].uint
3
UPDATE another solution that does not depend on bitstring, and count from left(according to OP):
Convert the number into binary format:
>>> n = '0xA5423'
>>> n = format(int(n, 16), '016b')[::-1] # reversed
>>> n
'11000100001010100101'
>>> int(n[0:2][::-1], 2) # need to reverse again to get proper value
3
>>> int(n[2:4][::-1], 2)
0
>>> int(n[4:12][::-1], 2)
66
>>> int(n[12:16][::-1], 2)
5

Malformed Packet on ICMP python

How i can create a frame icmp
I mark malformet packet on python
import socket, struct, uuid, subprocess, fcntl, time,random
List item
from binascii import hexlify, unhexlify from datetime import datetime
tarRed = raw_input('Ingrese nombre de Tarjeta de Red Utilizada: ')
subprocess.call(['ifconfig', tarRed, 'promisc']) pt = 0x0800 s =
socket.socket(socket.AF_PACKET, socket.SOCK_RAW, socket.htons(pt))
s.bind((tarRed, pt))
#GETCheck def getchecksum(ip_header,size): cksum = 0 pointer = 0 while size > 1: cksum += int((str("%02x" % (ip_header[pointer],))
+ str("%02x" % (ip_header[pointer+1],))), 16) size -= 2 pointer += 2 #if size: #This accounts for a situation where the header is odd
#cksum += int(ip_header[pointer]) cksum = (cksum >> 16) +
(cksum & 0xffff) cksum += (cksum >>16) return (~cksum) &
0xFFFF def _checksum(data): #calculate the header sum
ip_header_sum = sum(struct.unpack_from("6H", data)) #add the carry
ip_header_sum = (ip_header_sum & 0xFFFF) + (ip_header_sum >> 8 &
0xFFFF) ip_header_sum = ~ip_header_sum & 0xFFFF return
ip_header_sum def i_checksum(checksum_packet): total = 0 num_words
= len(checksum_packet) / 2 for chunk in struct.unpack("!%sH" % num_words, checksum_packet[0:num_words*2]): total += chunk if
len(checksum_packet) % 2: total += ord(checksum_packet[-1]) << 8
total += total >> 16 return (~total + 0xffff & 0xffff)
def checksum(source_string): sum = 0 countTo =
(len(source_string)/2)*2 count = 0 while count<countTo:
thisVal = ord(source_string[count + 1])*256 + ord(source_string[count]) sum = sum + thisVal sum = sum &
0xffffffff
count = count + 2 if countTo<len(source_string): sum = sum + ord(source_string[len(source_string) - 1]) sum = sum &
0xffffffff sum = (sum >> 16) + (sum & 0xffff) sum = sum + (sum
>> 16) answer = ~sum answer = answer & 0xffff #answer = answer >> 8 | (answer << 8 & 0xff00) return answer def
checksuma(source_string): sum = 0 countTo =
(len(source_string)/2)*2 count = 0 while count<countTo:
thisVal = ord(source_string[count + 1])*256 + ord(source_string[count]) sum = sum + thisVal sum = sum &
0xffffffff
count = count + 2 if countTo<len(source_string): sum = sum + ord(source_string[len(source_string) - 1]) sum = sum &
0xffffffff sum = (sum >> 16) + (sum & 0xffff) sum = sum + (sum
>> 16) answer = ~sum answer = answer & 0xffff answer = answer >> 8 | (answer << 8 & 0xff00) return answer def ultimo(str):
csum = 0
countTo = (len(str) / 2) * 2
count = 0
while count < countTo:
thisVal = ord(str[count+1]) * 256 + ord(str[count])
csum = csum + thisVal
csum = csum & 0xffffffff
count = count + 2
if countTo < len(str):
csum = csum + ord(str[len(str) - 1])
csum = csum & 0xffffffff
csum = (csum >> 16) + (csum & 0xffff)
csum = csum + (csum >> 16)
answer = ~csum
answer = answer & 0xffff
answer = answer >> 8 | (answer << 8 & 0xff00)
return answer
#GETCheck
#MACs adst='ffffffffffff' mac_destino = unhexlify(adst) ma = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) info =
fcntl.ioctl(ma.fileno(), 0x8927, struct.pack('256s', tarRed[:15]))
asrc = ''.join(['%02x:' % ord(char) for char in info[18:24]])[:-1]
sd = asrc.split(':') asrc = sd[0]+sd[1]+sd[2]+sd[3]+sd[4]+sd[5]
mac_origen = unhexlify(asrc)
#MACs cabEther = struct.pack('!6s6sh',mac_destino,mac_origen,pt)
#datosIP
#version = '4'
#IHL = '5' tipoServicio = unhexlify('00') longitudT = struct.pack('!BB',00,24) identificador = struct.pack('!BB',00,01)
flag_Pos = 0x4000 tiempoVida = 05 ptIP = unhexlify('01') SCC = 0
#IPs ip = socket.socket(socket.AF_INET, socket.SOCK_DGRAM) ipO= str( socket.inet_ntoa(fcntl.ioctl(ip.fileno(),0x8915,struct.pack('256s',
tarRed[:15]))[20:24])) ipO1 = ipO.split('.') ipOr =
struct.pack("!BBBB",int(ipO1[0]),int(ipO1[1]),int(ipO1[2]),int(ipO1[3]))
ipD = raw_input('Ingrese la direccion IP destino: ') ipD1 =
ipD.split('.') ipDest =
struct.pack("!BBBB",int(ipD1[0]),int(ipD1[1]),int(ipD1[2]),int(ipD1[3]))
#IPs
#datosIP cabIP = struct.pack('!1s1s2s2shB1sH4s4s',unhexlify('45'),tipoServicio,longitudT,identificador,flag_Pos,tiempoVida,ptIP,SCC,ipOr,ipDest)
SCC = checksum(cabIP) cabIP =
struct.pack('!1s1s2s2shB1sH4s4s',unhexlify('45'),tipoServicio,longitudT,identificador,flag_Pos,tiempoVida,ptIP,socket.htons(SCC),ipOr,ipDest)
#datosIcmp tipo = 8 codigo = 0x00 check = 0x0000 identificador = int((id(1) * random.random()) % 65535) secuencia = 0x0000
#datos = 'qwertyasdfghzxcvbn0102030405060708091011121314151617181920' datos =
1 * 'Q'
#datosIcmp cabIcmp = struct.pack('!bbHHh',tipo,0,0,identificador,1) my_checksum = ultimo(cabIcmp + datos) cabIcmp =
struct.pack('!bbHHh',tipo,0,socket.htons(my_checksum),identificador,1)
cabIcmp = cabIcmp + datos tiempo = datetime.now() print tiempo for i
in range(15): s.send(cabEther + cabIP + cabIcmp,0)

Your question is barely understandable. It appears you're asking how to create an ICMP ECHO packet since the packet you are creating in the listed code is reported as malformed by your system. I'll try to answer why your code won't work.
First, the socket type you're using is incorrect. You need type 1, not 8. This can be done easily enough with the third parameter:
s = socket.socket(socket.AF_INET, socket.SOCK_RAW, socket.IPPROTO_ICMP)
(Note you'll need root or administrator privileges to make this call.)
Second, after creating the socket, you'll need to create the ICMP header. An ICMP header consists of the following structure:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Type | Code | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | REST OF HEADER |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
Notice the second four bytes "REST OF HEADER". This second set of four bytes depends on the TYPE/CODE in the first two bytes. For example, if TYPE is 0 and CODE is 0, it means you're dealing with an echo reply, and as such, the second four bytes will become 2-byte Identifier and 2-byte Sequence Number fields, starting at offset 4 and ending at offset 7 as the picture shows:
0 1 2 3
0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
0 | Type | Code | Checksum |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
4 | Identifier | Sequence Number |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
8 | Data ...
+-+-+-+-+-
To create this header, I'm going to use type 8, the echo request structure, as such (note that bytes 4-5 and 6-7 in this creation is just for illustration, typically this would be different values other than zero):
icmp = struct.pack(">BBHHH", 8, 0, 0, 0, 0)
On this initial creation of the header, the checksum is zero. What we now need is to actually calculate the checksum (with this zero value), and THEN recreate the header with this new checksum value:
icmp = struct.pack(">BBHHH", 8, 0, cksum(icmp), 0, 0)
But this is where your code is incorrect again. The implementation of i_checksum() is wrong. To fix, I'll leave it to you to figure it out. The checksum is called 1's complement, and there are numerous online articles about it, as well as C code on Linux systems that implement it.
Once you have the header and its correct checksum (important because your system won't send the packet if incorrect), you simply then send it somewhere:
s.sendto(icmp, (SOME_REMOTE_HOST, 0))
And then receive replies if any:
s.recvfrom(1500)
(Note I chose 1500 bytes as that is the MTU of an IP frame, and it is thus unlikely a reply will be bigger than this, though it is certainly possible.)
AND AT THIS POINT, YOUR CODE TOTALLY FAILS -- simply because you do not have code which does the last two steps, let alone additional code to handle what to do with replies and struct-parsing of the header fields.
As for the other two functions you've defined, they're odd. One of them seems to imply it is dealing with link layer fields -- which is never the case on sockets as sockets only deal in IP layer and above -- while the other seems to be an attempt at calculating a checksum...albeit incorrectly (again).
In short, as abarnert implied, please clarify (and add some more) code AND your question.