I have an issue with the pymodbus decoder with strings. For example, when I try to read 'abcdefg' pymodbus gives me 'badcfehg'. The byteorder and the wordorder don't change the result.
Here is my code:
result=client.read_holding_registers(25000,4)
decoder = BinaryPayloadDecoder.fromRegisters(result.registers,byteorder=Endian.Little,wordorder=Endian.Big)
decoder.decode_string(8)
Can someone explain why the order does not change the result? I try with the builder and it's the same problem. However, I don't have this problem with 32 bits floats for example.
I also tried with an older version of pymodbus and it works:
decoder = BinaryPayloadDecoder.fromRegisters(registers,endian=Endian.Little)
Note: I already read the following topic: pymodbus: Issue reading String & multiple type of data from Modbus device but I don't have any access to the modbus server.
The problem is that Modbus specs does not define in what order the two bytes for char strings are sent or even in what order 16-bit words are sent for 32-bit types.
Then some Modbus devices send bytes or words in an order and others do the opposite.
If you are writing a Modbus client then you should add the option in the configuration to be able to invert the order of both bytes and 16-bit words in 32-bit data types.
https://en.wikipedia.org/wiki/Endianness
I've encountered this same issue with byteorder not working (wordorder seems to be fine). The solution I came up with is to use Struct:
import struct
Then:
count = 4 #Read 4 16bit registers
result = client.read_holding_registers(25000,count)
for i in range(count):
result.registers[i] = struct.unpack("<H", struct.pack(">H", result.registers[i]))[0]
decoder = BinaryPayloadDecoder.fromRegisters(result.registers)
print(decoder.decode_string(7)) #Since string is 7 characters long
This uses Struct to unpack and pack as an unsigned short integer. The endianness does not matter since all you're doing is swapping bytes. The result overwrites the registers so you can then use the BinaryPayloadDecoder as you normally would.
I would have preferred to iterate through the responses instead of using range(count), but couldn't find a way to do it and wanted to post this workaround. If I figure it out, I will let you know.
Related
I've got an I2S microphone connected to a microcontroller and have managed to dump 16-bit audio WAV audio to a python bytearray object that looks like this (using a micropython library):
raw = bytearray(b"\xac\xffW\x00\xfc\xfe\xac\xffs\xfe\xfc\xfe+\xfes\xfe7\xfe+\xfe\x8c\xfe7\xfe\x1f\xff\x8c\xfe\xcf\xff\x1f\xfft\x00\xcf\xff\xfb\x00t\x00?\x01\xfb...")
I have successfully written these bytearray dumps to a file I created like this:
wav = open('16bitaudio.wav','wb')
#....some code to write the wav header
wav.write(raw)
wav.close()
When I open this on my PC it plays the samples I've recorded faithfully, sounds great.
My issue comes - I want to translate this data to an integer which represents the average intensity of sound in my samples. I first attempted to do this:
intensity = sum(raw)/count(raw)
However, this tends to result in a number ~128 almost all the time - suggesting to me it's being read as random bytes. Upon further investigation, these array functions seem to assume that we've only got an 8 bit byte (reading the value b'\xffW' which I believe is a little endian 22527):
>>> int(raw[1])
255
which appears to be just the b'\xff' part.
I can get my expected value by parsing just the byte manyally into int.from_bytes:
>>> int.from_bytes(b'\xffW','little')
22527
However I can't seem to iterate through the bytearray without it truncating to 8-bit.
Finally, I have read the struct.unpack methods - which look OK, but I'm not sure bytearray get packed with bytes of consistent length.... e.g.:
>>> len(bytearray(b'\xfdo\xfe\x7f\xfd\xd3\xf1d'))
8
Even though I see only 6 bytes represented. The ultimate problem I have with unpacking I'm not sure if each byte is 8 or 16 bit ahead of time, so I don't know what combination of letters to use in the second argument...
So, given the b-string representation, it seems that python DOES have knowledge of the way the bytes are encoded, however it seems like the normal array functions I've got on hand are getting this info from the bytearray. I'm sure there is a pythonic way to parse this bytearray to integers, but I just can't find it...
Any help extremely appreciated
Thanks #juanpa.arrivillaga for the answer I was looking for. I used the array library which seemed to solve all of my problems:
import array
result = array.array('h', raw)
Graphing the values output here is the same as the oscilloscope for my audio file.... Cheers!
integers
oscilloscope of my working .wav
I'm making a script to get Valve's server information (players online, map, etc)
the packet I get when I request for information is this:
'\xff\xff\xff\xffI\x11Stargate Central CAP SBEP\x00sb_wuwgalaxy_fix\x00garrysmod\x00Spacebuild\x00\xa0\x0f\n\x0c\x00dw\x00\x0114.09.08\x00\xb1\x87i\x06\xb4g\x17.\x15#\x01gm:spacebuild3\x00\xa0\x0f\x00\x00\x00\x00\x00\x00'
This may help you to see what I'm trying to do https://developer.valvesoftware.com/wiki/Server_queries#A2S_INFO
The problem is, I don't know how to decode this properly, it's easy to get the string but I have no idea how to get other types like byte and short
for example '\xa0\x0f'
For now I'm doing multiple split but do you know if there is any better way of doing this?
Python has functions for encoding/decoding different data types into bytes. Take a look at the struct package, the functions struct.pack() and struct.unpack() are your friends there.
taken from https://docs.python.org/2/library/struct.html
>>> from struct import *
>>> pack('hhl', 1, 2, 3)
'\x00\x01\x00\x02\x00\x00\x00\x03'
>>> unpack('hhl', '\x00\x01\x00\x02\x00\x00\x00\x03')
(1, 2, 3)
The first argument of the unpack function defines the format of the data stored in the second argument. Now you need to translate the description given by valve to a format string. If you wanted to unpack 2 bytes and a short from a data string (that would have a length of 4 bytes, of course), you could do something like this:
(first_byte, second_byte, the_short) = unpack("cc!h", data)
You'll have to take care yourself, to get the correct part of the data string (and I don't know if those numbers are signed or not, be sure to take care of that, too).
The strings you'll have to do differently (they are null-terminated here, so start were you know a string starts and read to the first "\0" byte).
pack() work's the other way around and stores data in a byte string. Take a look at the examples on the python doc and play around with it a bit to get a feel for it (when a tuple is returned/needed, e.g.).
struct supports you in getting the right byte order, which most of the time is network byte order and different from your system. That is of course only necessary for multi byte integers (like short) - so a format string of `"!h" should unpack a short correctly.
I have a C pipe client (pulled directly from the CallNamedPipe example found here) that, if given the string "first", sends the following message to my Python pipeserver:
b'f\x00i\x00r\x00s\x00t\x00\x00\x00'
The struct documentation gives examples where I did both the packing and unpacking in Python. That means I know the format because I explicitly specified it when I called struct.pack.
Is there some way for me to either a) infer the format from the above output or b) set the format in C the same way I do in Python?
Here's the relevant client code:
LPTSTR lpszPipename = TEXT("\\\\.\\pipe\\testpipe");
LPTSTR lpszWrite = TEXT("first");
fSuccess = CallNamedPipe(
lpszPipename, // pipe name
lpszWrite, // message to server
...
>>> b'f\x00i\x00r\x00s\x00t\x00\x00\x00'.decode('utf-16le')
u'first\x00'
"The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!)"
Your C code is not writing a struct to the pipe, it is writing a null-terminated string encoded as little-endian UTF-16 text, which is produced by the TEXT() macro when you compile your Windows program in Unicode mode for an Intel CPU. Python knows how to decode these strings without using the struct module. Try this:
null_terminated_unicode_string = data.decode('utf-16le')
unicode_string = null_terminated_unicode_string[:-1]
You can use decode('utf-16') if your python code is running on the same CPU architecture as the C program that writes the data. You might want to read up on python's unicode codecs.
EDIT: You can infer the type of that data by knowing how UTF-16 and Windows string macros work, but python cannot infer it. You could set the string encoding in C the same way you would in python if you wanted to write some code to do so, but it's probably not worth your time.
I have a web application developed in Adobe Flex 3 and Python 2.5 (deployed on Google App Engine). A RESTful web service has been created in Python and its results are currently in an XML format which is being read by Flex using the HttpService object.
Now the main objective is to compress the XML so that there is as less a time between the HttpService send() method and result events. I looked up Python docs and managed to use zlib.compress() to compress the XML result.
Then I set the HttpService result type from "xml" to "text" and tried using ByteArrays to uncompress the string back to XML. Here's where I failed. I am doing something like this:
var byteArray:ByteArray = new ByteArray();
byteArray.writeUTF( event.result.toString() );
byteArray.uncompress();
var xmlResult:XML = byteArray.readUTF();
Its throwing an exception at byteArray.uncompress() and says unable to uncompress the byteArray. Also when I trace the length of the byteArray it gets 0.
Unable to figure out what I'm doing wrong. All help is appreciated.
-- Edit --
The code:
# compressing the xml result in Python
print zlib.compress(xmlResult)
# decompresisng it in AS3
var byteArray:ByteArray = new ByteArray();
byteArray.writeUTF( event.result.toString() );
byteArray.uncompress()
Event is of type ResultEvent.
The error:
Error: Error #2058: There was an error decompressing the data.
The error could be because the value of byteArray.bytesAvailable = 0 which means the raw bytes python generated hasn't been written into byteArray properly..
-- Sri
What is byteArray.writeUTF( event.result.toString() ); supposed to do? The result of zlib.compress() is neither unicode nor "UTF" (meaningless without a number after it!?); it is binary aka raw bytes; you should neither decode it nor encode it nor apply any other transformation to it. The receiver should decompress immediately the raw bytes that it receives, in order to recover the data that was passed to zlib.compress().
Update What documentation do you have to support the notion that byteArray.uncompress() is expecting a true zlib stream and not a deflate stream (i.e. a zlib stream after you've snipped the first 2 bytes and the last 4)?
The Flex 3 documentation of ByteArray gives this example:
bytes.uncompress(CompressionAlgorithm.DEFLATE);
but unhelpfully doesn't say what the default (if any) is. If there is a default, it's not documented anywhere obvious, so it would be a very good idea for you to use
bytes.uncompress(CompressionAlgorithm.ZLIB);
to make it obvious what you intend.
AND the docs talk about a writeUTFBytes method, not a writeUTF method. Are you sure that you copy/pasted the exact receiver code in your question?
Update 2
Thanks for the URL. Looks like I got hold of the "help", not the real docs :=(. A couple of points:
(1) Yes, there is an explicit inflate() method. However uncompress DOES have an algorithm arg; it can be either CompressionAlgorithm.ZLIB (the default) or CompressionAlgorithm.DEFLATE ... interestingly the latter is however only available in Adobe Air, not in Flash Player. At least we know the uncompress() call appears OK, and we can get back to the problem of getting the raw bytes onto the wire and off again into a ByteArray instance.
(2) More importantly, there are both writeUTF (Writes a UTF-8 string to the byte stream. The length of the UTF-8 string in bytes is written first, as a 16-bit integer, followed by the bytes representing the characters of the string) and writeUTFBytes (Writes a UTF-8 string to the byte stream. Similar to the writeUTF() method, but writeUTFBytes() does not prefix the string with a 16-bit length word).
Whatever the merits of supplying UTF8-encoded bytes (nil, IMHO), you don't want a 2-byte length prefix there; using writeUTF() is guaranteed to cause uncompress() to bork.
Getting it on to the wire: using Python print on binary data doesn't seem like a good idea (unless sys.stdout has been nobbled to run in raw mode, which you didn't show in your code).
Likewise doing event.result.toString() getting a string (similar to a Python unicode object, yes/no?) -- with what and then encoding it in UTF-8 seem rather unlikely to work.
Given I didn't know that flex existed until today, I really can't help you effectively. Here are some further suggestions towards self-sufficiency in case nobody who knows more flex comes along soon:
(1) Do some debugging. Start off with a minimal XML document. Show repr(xml_doc). Show repr(zlib_compress_output). In (a cut-down version of) your flex script, use the closest function/method to repr() that you can find to show: event.result, event.result.toString() and the result of writeUTF*(). Make sure you understand the effects of everything that can happen after zlib.compress(). Reading the docs carefully may help.
(2) Look at how you can get raw bytes out of event.result.
HTH,
John
serial.write() method in pyserial seems to only send string data. I have arrays like [0xc0,0x04,0x00] and want to be able to send/receive them via the serial port? Are there any separate methods for raw I/O?
I think I might need to change the arrays to ['\xc0','\x04','\x00'], still, null character might pose a problem.
An alternative method, without using the array module:
def a2s(arr):
""" Array of integer byte values --> binary string
"""
return ''.join(chr(b) for b in arr)
You need to convert your data to a string
"\xc0\x04\x00"
Null characters are not a problem in Python -- strings are not null-terminated the zero byte behaves just like another byte "\x00".
One way to do this:
>>> import array
>>> array.array('B', [0xc0, 0x04, 0x00]).tostring()
'\xc0\x04\x00'
I faced a similar (but arguably worse) issue, having to send control bits through a UART from a python script to test an embedded device. My data definition was "field1: 8 bits , field2: 3 bits, field3 7 bits", etc. It turns out you can build a robust and clean interface for this using the BitArray library. Here's a snippet (minus the serial set-up)
from bitstring import BitArray
cmdbuf = BitArray(length = 50) # 50 byte BitArray
cmdbuf.overwrite('0xAA', 0) # Init the marker byte at the head
Here's where it gets flexible. The command below replaces the 4 bits at
bit position 23 with the 4 bits passed. Note that it took a binary
bit value, given in string form. I can set/clear any bits at any location
in the buffer this way, without having to worry about stepping on
values in adjacent bytes or bits.
cmdbuf.overwrite('0b0110', 23)
# To send on the (previously opened) serial port
ser.write( cmdbuf )