serial.write() method in pyserial seems to only send string data. I have arrays like [0xc0,0x04,0x00] and want to be able to send/receive them via the serial port? Are there any separate methods for raw I/O?
I think I might need to change the arrays to ['\xc0','\x04','\x00'], still, null character might pose a problem.
An alternative method, without using the array module:
def a2s(arr):
""" Array of integer byte values --> binary string
"""
return ''.join(chr(b) for b in arr)
You need to convert your data to a string
"\xc0\x04\x00"
Null characters are not a problem in Python -- strings are not null-terminated the zero byte behaves just like another byte "\x00".
One way to do this:
>>> import array
>>> array.array('B', [0xc0, 0x04, 0x00]).tostring()
'\xc0\x04\x00'
I faced a similar (but arguably worse) issue, having to send control bits through a UART from a python script to test an embedded device. My data definition was "field1: 8 bits , field2: 3 bits, field3 7 bits", etc. It turns out you can build a robust and clean interface for this using the BitArray library. Here's a snippet (minus the serial set-up)
from bitstring import BitArray
cmdbuf = BitArray(length = 50) # 50 byte BitArray
cmdbuf.overwrite('0xAA', 0) # Init the marker byte at the head
Here's where it gets flexible. The command below replaces the 4 bits at
bit position 23 with the 4 bits passed. Note that it took a binary
bit value, given in string form. I can set/clear any bits at any location
in the buffer this way, without having to worry about stepping on
values in adjacent bytes or bits.
cmdbuf.overwrite('0b0110', 23)
# To send on the (previously opened) serial port
ser.write( cmdbuf )
Related
I'm asking for your help because I've been stuck with the same problem for 3 days.
If I have :
Value1 = 0, Value2 = 3.10 and IPv6 = '2001::1'
I would like to pack all 3 values with this command: package = struct.pack(*format*, value1, value2, IPv6)
My problem is: I don't know what format characters in C type I can use to pack the IPv6 and keep its 16 bytes.
I know that I can use format = 'i f ?' with i for integer / f for float but I need to find with what to replace the '?' which is the format characters in C type for an IPv6 address to pack the three values.
Please, someone can help me?
It is not that straightforward to use struct for this, as you'd have to know the proper byte values of the IPv6 address before adding then to a struct.
'2001::1' is a textual representation that is nowhere close to giving you those values: you'd have to split the string on :, replace missing values with "0", then you 'd have a 4 16 bit number list to pack in the struct. and them, certainly there are corner cases and special syntax in the IPv6 string representation you'd have to account for.
Fortunately Python already handles that for you in the. ipaddress module of the stdlib.
Just import ipaddress, format the struct for the first part of your package and concatenate it with the "packed" attribute in the IPv6Address Python automatically genreates for you:
import struct
import ipaddress
Value1 = 0
Value2 = 3.10
IPv6 = '2001::1'
payload = struct.pack("if", (Value1, Value2)) + ipaddress.ip_address(IPv6).packed
However, I wonder if it will be productive to simply pack an int and a float along with an IP address in this way - whatever code will be reading this will be super coupled with the code you are writing to that.
If you are simply storing it to a file to be read back by a Python program under your control just use pickle instead. If you intend to send these values to a non Python program over a network, a schemaless textual way of conveying them, like JSON, might be much simpler.
If you really want to store these, and only these, in a compact way in order to save space, and there are tens of thousands of them, and they will be read back by the same program: try numpy arrays. They will take care of the compact binary representation for each object type and can be read and written to binary files, and numpy will take care of record offset for you.
The only use case I could see for that is if you have a program not under your control in a low level protocol that would be expecting exactly this record format. Since you are speculating about how to create the payload, and trying to convey "3.10" as a floating point value, this does not seem to be the case. Talking about that, "3.10" or other numbers might not round trip well as a nicely formed 2-decimal digit value with structs like this, due to how floating points are represented internally. I suggest you review your goals and needs there, and not overcomplicate things.
To unpack back, the easier thing is to use struct to recover just the numeric values, and pass the remaining 16 bytes back to the ip_address factory function - it automatically creates an IPv6 object, which string representation is the human friendly "2001::1".
I type the "roundtrip" in an interactive prompt:
In [30]: import struct, ipaddress
In [31]: x = ipaddress.ip_address('2001::1')
In [32]: v1 = 2;v2 = 3.10
In [33]: payload = struct.pack(">if",v1, v2) + x.packed
In [34]: print(payload)
b'\x00\x00\x00\x02#Fff \x01\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x01'
In [35]: v3, v4, nx = struct.unpack(">if", aa[:-16]) + (ipaddress.ip_address(aa[-16:]),)
In [36]: print(v3, v4, nx)
2 3.0999999046325684 2001::1
Two things to be noted: the added > prefix on the struct formatting string, to ensure byte order when encoding its contents,
and that the "3.10" value, as described above, does not roundtrip as a "nice two decimal places" value. Use round(number, 2) when representing the number on the server side, if needed.
I have an issue with the pymodbus decoder with strings. For example, when I try to read 'abcdefg' pymodbus gives me 'badcfehg'. The byteorder and the wordorder don't change the result.
Here is my code:
result=client.read_holding_registers(25000,4)
decoder = BinaryPayloadDecoder.fromRegisters(result.registers,byteorder=Endian.Little,wordorder=Endian.Big)
decoder.decode_string(8)
Can someone explain why the order does not change the result? I try with the builder and it's the same problem. However, I don't have this problem with 32 bits floats for example.
I also tried with an older version of pymodbus and it works:
decoder = BinaryPayloadDecoder.fromRegisters(registers,endian=Endian.Little)
Note: I already read the following topic: pymodbus: Issue reading String & multiple type of data from Modbus device but I don't have any access to the modbus server.
The problem is that Modbus specs does not define in what order the two bytes for char strings are sent or even in what order 16-bit words are sent for 32-bit types.
Then some Modbus devices send bytes or words in an order and others do the opposite.
If you are writing a Modbus client then you should add the option in the configuration to be able to invert the order of both bytes and 16-bit words in 32-bit data types.
https://en.wikipedia.org/wiki/Endianness
I've encountered this same issue with byteorder not working (wordorder seems to be fine). The solution I came up with is to use Struct:
import struct
Then:
count = 4 #Read 4 16bit registers
result = client.read_holding_registers(25000,count)
for i in range(count):
result.registers[i] = struct.unpack("<H", struct.pack(">H", result.registers[i]))[0]
decoder = BinaryPayloadDecoder.fromRegisters(result.registers)
print(decoder.decode_string(7)) #Since string is 7 characters long
This uses Struct to unpack and pack as an unsigned short integer. The endianness does not matter since all you're doing is swapping bytes. The result overwrites the registers so you can then use the BinaryPayloadDecoder as you normally would.
I would have preferred to iterate through the responses instead of using range(count), but couldn't find a way to do it and wanted to post this workaround. If I figure it out, I will let you know.
I need to get an int through the network. Is this the proper way to convert to bytes in big-endian?
pack("I",socket.htonl(integer_value))
I unpack it as:
socket.ntohl(unpack("I",data)[0])
I noticed that pack-unpack also have the <> to use for endian conversion so I am not sure if I could just directly use that instead or if htonl is safer.
You should use only the struct module for communicating with another system. By using the htonl first, you'll end up with an indeterminate order being transmitted.
Since you need to convert the integer into a string of bytes in order to send it to another system, you'll need to use struct.pack (because htonl just returns a different integer than the one passed as argument and you cannot directly send an integer). And in using struct.pack you must choose an endianness for that string of bytes (if you don't specify one, you'll get a default ordering which may not be the same on the receiving side so you really need to choose one).
Converting an integer to a sequence of bytes in a definite order is exactly what struct.pack("!I", integer_value) does and a sequence of bytes in a definite order is exactly what you need on the receiving end.
On the other hand, if you use struct.pack("!I", socket.htonl(integer_value)), what does that do? Well, first it puts the integer into big-endian order (network byte order), then it takes your already big-endian integer and converts it to bytes in "big-endian order". But, on a little endian machine, that will actually reverse the ordering again, and you will end up transmitting the integer in little-endian byte order if you do both those two operations.
But on a big-endian machine htonl is a no-op, and then you're converting the result into bytes in big-endian order.
So using ntohl actually defeats the purpose and a receiving machine would have to know the byte-order used on the sending machine in order to properly decode it. Observe...
Little-endian box:
>>> print(socket.htonl(27))
452984832
>>> print(struct.pack("!I", 27))
b'\x00\x00\x00\x1b'
>>> print(struct.pack("!I", socket.htonl(27)))
b'\x1b\x00\x00\x00'
Big-endian box:
>>> print(socket.htonl(27))
27
>>> print(struct.pack("!I", 27))
b'\x00\x00\x00\x1b'
>>> print(struct.pack("!I", socket.htonl(27)))
b'\x00\x00\x00\x1b'
struct.unpack() uses '!' in the format specifiers for network byte order. But its the same as '>'...
I'm using Python to communicate data over a TCP socket between a server and client application. I need to send a 4 bytes which represent a data sample. The initial sample is an 32-bit unsigned integer. How can I send those 4 bytes of raw data through the socket?
I want to send the data: 0x12345678 and 0xFEDCBA98
The raw data sent over the socket should be exactly that if I read it on wireshark/tcpdump/etc. I don't want each value in the 8 hex numbers to be represented as an ascii character, I want the raw data to remain intact.
Thank you
The main method to send binary data in Python is using the struct module.
For example, packing 3 4-byte unsigned integers is done like this
In [3]: struct.pack("III", 3, 4, 5)
Out[3]: '\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00'
Note to keep the endianess correct, using "<", ">", and so on.
you could use bytes(). Thats what I use to send strings over a network but you can use it for ints as well. Its usage is bytes([3])
Edit: bytes converts an int (3) to bytes. a string of bytes is represented as b'\x03'. so like your byte string: 0x12345678 would be b'\x12345678'
check the docs for more info:
https://docs.python.org/3.1/library/functions.html#bytes
I usually perform things like this in C++, but I'm using python to write a quick script and I've run into a wall.
If I have a binary list (or whatever python stores the result of an "fread" in). I can access the individual bytes in it with: buffer[0], buffer[1], etc.
I need to change the bytes [8-11] to hold a new 32-bit file-size (read: there's already a filesize there, I need to update it). In C++ I would just get a pointer to the location and cast it to store the integer, but with python I suddenly realized I have no idea how to do something like this.
How can I update 4 bytes in my buffer at a specific location to hold the value of an integer in python?
EDIT
I'm going to add more because I can't seem to figure it out from the solutions (though I can see they're on the right track).
First of all, I'm on python 2.4 (and can't upgrade, big corporation servers) - so that apparently limits my options. Sorry for not mentioning that earlier, I wasn't aware it had so many less features.
Secondly, let's make this ultra-simple.
Lets say I have a binary file named 'myfile.binary' with the five-byte contents '4C53535353' in hex - this equates to the ascii representations for letters "L and 4xS" being alone in the file.
If I do:
f = open('myfile.binary', 'rb')
contents = f.read(5)
contents should (from Sven Marnach's answer) hold a five-byte immutable string.
Using Python 2.4 facilities only, how could I change the 4 S's held in 'contents' to an arbitrary integer value? I.e. give me a line of code that can make byte indices contents [1-4] contain the 32-bit integer 'myint' with value 12345678910.
What you need is this function:
struct.pack_into(fmt, buffer, offset, v1, v2, ...)
It's documented at http://docs.python.org/library/struct.html near the top.
Example code:
import struct
import ctypes
data=ctypes.create_string_buffer(10)
struct.pack_into(">i", data, 5, 0x12345678)
print list(data)
Similar posting: Python: How to pack different types of data into a string buffer using struct.pack_into
EDIT: Added a Python 2.4 compatible example:
import struct
f=open('myfile.binary', 'rb')
contents=f.read(5)
data=list(contents)
data[0:4]=struct.pack(">i", 0x12345678)
print data
Have a look at Struct module. You need pack function.
EDIT:
The code:
import struct
s = "LSSSS" # your string
s = s[0] + struct.pack('<I', 1234567891) # note "shorter" constant than in your example
print s
Output:
L╙☻ЦI
struct.pack should be available in Python2.4.
Your number "12345678910" cannot be packed into 4 bytes, I shortened it a bit.
The result of file.read() is a string in Python, and it is immutable. Depending on the context of the task you are trying to accomplish, there are different solutions to the problem.
One is using the array module: Read the file directly as an array of 32-bit integers. You can modify this array and write it back to the file.
with open("filename") as f:
f.seek(0, 2)
size = f.tell()
f.seek(0)
data = array.array("i")
assert data.itemsize == 4
data.fromfile(f, size // 4)
data[2] = new_value
# use data.tofile(g) to write the data back to a new file g
You could install the numpy module, which is often used for scientific computing.
read_data = numpy.fromfile(file=id, dtype=numpy.uint32)
Then access the data at the desired location and make your changes.
The following is just a demonstration for you to understand what really happens when the four bytes are converted into an integer.
Suppose you have a number: 15213
Decimal: 15213
Binary: 0011 1011 0110 1101
Hex: 3 B 6 D
On little-endian systems (i.e x86 machines), this number can be represented using a length-4 bytearray as: b"\x6d\x3b\x00\x00" or b"m;\x00\x00" when you print it on the screen, to convert the four bytes into an integer, we simply do a bit of base conversion, which in this case, is:
sum(n*(256**i) for i,n in enumerate(b"\x6d\x3b\x00\x00"))
This gives you the result: 15213