We Keep Coding

Modbus home assistant and python struct (little endian, big endian, negative values from uint16)

I have a question regarding to modbus settings. I have read the documentation carefully, tried to search some topics, but unfortunately I did not find an answer to my problem.
I have a heat pump, which is able to communicate through modbus. In the past without HA I had my own application on ESP8266, reading the data, uploading them, etc. Now I would like to move it into HA. I found the modbus protocol is implemented in HA, which is great.
Now, in my custom app I had to read the registry and modify the respond for each value as the device has MSB implementation, let me provide an example:
From the device documentation, the only things I know (and it seems to be enough as I was able to implement the app) are: The heat pump communicates on ..... IP address, on .... port. It uses slave and the slave ID is .... All values are represented in MSB (most significant byte). Now about the values, for example outside temperature is on address 0, type is read only and the scale is 100 (for modbus in configuration file it should be 0.01 - lets ignore it for now), unit is °C.
So, my configuration look like:
# Modbus configuration
modbus:
- name: ...
type: tcp
host: ...
port: ...
delay: 5
timeout: 5
sensors:
- name: Heat pump outside temperature
address: 0
slave: 1
input_type: holding
device_class: temperature
state_class: measurement
data_type: uint16
unique_id: "ac_heating_outside_temp"
This results to value of 65436 on this entity, which is wrong obviously. The real value at this moment is -1. The biggest value for uint16 is 65535, 65436-65535 = -99, multiplied by 0.01 (or divided by 100) is -0.99, which is (if we deduct zero) -1.00 degree... This is the value which I need. Well, in my C app, I have been doing this recalculation on my own (in bytes). Unfortunately I have no idea how to do that in "our" modbus yaml description.
I have been looking to SWAP, DATA_TYPE as well as STRUCTURE in the documentation: DOCUMENTATION (documentation link) unfortunately nothing is working for me. I know I have to set custom data_type if I would like to provide structure, but defining the custom type and ">I" in the structure requires 2 registries to read, but the address of the entity is 0, which is 1 registry. Even like that I tried that, but I am not able to get the proper value. Having the data_type to uint16 with the swap byte or even swap word does not seems to work. I tried to play (out of necessity) with uint8, 2 registries and swap together, but no combination leads to the proper result. Python struct documentation: https://docs.python.org/3.8/library/struct.html
Can anybody help me with this one?

You can consider Modbus-to-MQTT bridge pattern, using some professional Modbus tool to poll device data and publish to MQTT, and then home assistant can easily integrate such data via MQTT integration.
There are at least 3 advantages by following the above solution,
A Modbus tool (e.g. modpoll) gives you much more options (Big/Little Word/Byte Endian, various data types) to handle some non-standard implementations from various vendors
MQTT broker decouples Modbus workload from HA, which is good for modular design principle
MQTT broker makes debugging/troubleshooting much easier, you can subscribe to specific topic anytime without HA downtime.

i am sending commands through serial port in python but they are sent multiple times instead of one

i am sending some commands having particular response serially using com port..the commands are kept in a file..i am reading each command through the file line by line and sending it serially over the com port..but when i am seeing it from the receiver end using Magic Terminal(Software)..i found that each command is going multiple times..which i am sending only one time..i have made a code in pycharm..and in the console i am seeing that command is going only once but from the uart receiving end the story is something else..i am stuck with this problem..i have maintain the same baudrate and everything but not able to diagnose the issue..
github link for the code is: https://github.com/AkshatPant06/Akshat-Pant/blob/master/cmd%20list
def recvResponse():
ser.write(serial.to_bytes(intCmd))
time.sleep(1)
data_recv=ser.read(2)
return data_recv
this i have used to receive the 2 byte response..

There seems to be nothing wrong with your code. At least to the extent I could reproduce, it only sends the command once (I tried your function after setting up my serial port in loopback).
I cannot say for sure but it might be that the terminal you're using has two windows, one for input and another one for output and somehow you're getting confused with what is in and out of your port.
One easy way to deal with this kind of issue is to use a sniffer on your port. You can do that combining com0com and Termite on Windows, as I recently explained here.
As you can see there is only one window on this terminal, and after setting up the forwarding you'll everything that comes in and out of your port. That should make it easier to see what your code is writing and reading.
To give you a conventional scenario to apply the sniffer trick you can refer to the following screenshot:
In this case, we have two real serial ports on a computer. On the first (COM9) we are running a Modbus server (you can imagine it as a bunch of memory addresses, each of one storing a 16-bit number). On COM10 we have a client that is sending queries asking for the contents of the first 10 addresses (called registers using the Modbus terminology). In a general use case, we have those ports linked with a cable, so we know (theoretically) that the client on COM10 is sending a data frame asking for those ten registers and the server on COM9 is answering with the numbers stored on those registers. But we are only able to see the contents on the server (left side of the picture) and what the client is receiving (right). What we don't see is what is traveling on the bus (yeah, we know what it is, but we don't know exactly how the Modbus protocol looks like on the inside).
If we want to tap on the bus to see what is being sent and received on each side we can create a couple of virtual ports with com0com and a port forwarding connection with Termite, something like the following screenshot:
Now we have moved our Modbus server to one of the virtual serial ports (COM4 in this case). After installing com0com we got (by default, but you can change names or add more port pairs, of course) a pair of forwarded ports (COM4<-->COM5). Now, if we want to see what is circulating through the ports we open Termite (bottom-right side of the picture) and set up another port forwarding scheme, in this case from virtual port COM5 to the real port COM9.
Finally (and exactly the same as before we were sniffing), we have COM9 connected together with COM10 with a cable. But now we are able to see all data going to and fro on the bus (all those HEX values you see on Termite displayed with the green/blue font).
As you can see, this will offer something similar to what you can do with more professional tools.

Control hardware (a Raman spectrometer) using Python

I have an old Raman spectrometer (one of these - http://www.camo.com/downloads/partners/deltanu/Inspector_Raman_Datasheet.pdf) and I'd like to write code, ideally in Python, that can provide it with input parameters, operate it and receive data from it.
The spectrometer connects to a PC via a USB, although it is assigned to a virtual COM port. I currently control it using an .exe file provided by the company that used to sell it, that I believe was produced using LabVIEW.
Is it possible to write my own code to control this sort of hardware? How can I pass parameters and commands to hardware? What information would I need to know to do this?
Although I'm a fairly proficient Python coder, this is a brand new area for me, so any advice on where to start would be super appreciated. I'm open to coding in another language if that would be more appropriate. And let me know if I need to provide any more info.
Cheers, Liam

A google search for the device model name and "programming manual" is usually where I start with something like this. Those keywords hopefully turn up something from the manufacturer that tells you how to do exactly what you're trying to do, and a lot of them include code samples. Unfortunately, with the little information I have on your device, I couldn't find anything. That's going to make it much, much harder.
Everything beyond this point is a guess based on what I've seen before. Typically, if a LabVIEW program interacts with a device over a virtual COM port, the program sends an ASCII command to the device using the protocol defined in the manual, and then receives ASCII data in return. You can try sniffing that data with the NI I/O Trace tool (http://www.ni.com/download/ni-io-trace-14.0.1/4914/en/) while running the manufacturer's application, and then trying to make sense of the flood of data that you see on that port.
It could also be a Modbus device, which may help you figure out the structure of the communication.
In short, this'll be tough without a programming manual, but there is some hope. Good luck!

How to identify a device broadcasting data over a specific port using Python?

Summary
I have set up a udp packet listener in python, and I would like to be able to identify the device that is broadcasting the data it receives.
The Goal
I have a PHP web page that is reading the data from a database, which is being populated by the listener inserting the data when it receives it. My goal is to have a toggle switch that allows the user to select which device to hear data from. So currently, data is only being broadcast by either an MT4000 telemetry device, or using the terminal to manually send data across port 30000.
I don't want to identify it from a specific serial port, as described in: Identifying serial/usb device python
But rather wherever it is connected (any serial ports).
My Method
My idea at the moment is to somehow, send a message back to the same device from the listener, acting as both an acknowledger, and as a scan, to ask what the device is. Is that a feasible way?
Problems
Increases the amount of data being transmitted massively with more back and forth packets.
It may not work for every device connected, methods of extracting identity may be different for each device.
Once the python has identified the device, I will insert into the database, and when the user selects a device, a modified query will be sent, ie
("SELECT * FROM table WHERE device = MT4000");
I feel that this is not a clean method to use, and would be very open for different suggestions.
The solution
Unless it helps get across an answer, I'm not looking for specific code, but rather the theory of the task.

You may want to look into the way that nmap performs service detection. It is my understanding that it uses several different approaches and then takes the best match available. Those different approaches include:
What port the service is running on
What welcome banner the service provides for an initial connection
What OS the server runs (and thus what services could possibly run on that server)
You can read more about this in the service and application detection chapter.
Since you are also receiving data from these devices you can look at that data to determine what type it is. The file command on linux is a tool that performs a similar function, and that can determine the type based on:
File extension (obviously inapplicable here)
Magic numbers that appear at or near the start of the file
The composition of the data (mostly binary, or mostly ascii/unicode/etc, byte endiness and so on)
The underlying functionality of the file command is available as libmagic, a C library. It would be worth trying to use that directly, rather than duplicating it's functionality.
It's worth pointing out that a lot of these techniques provide statistical probabilities rather than certain answers. This may mean that you have to deal with a degree of uncertainty in your results, leading to misclassifications. To mitigate this you can collect data until you are sure enough that the device providing the data has been correctly identified.

Identifying serial/usb device python

The solution to this problem is probably pretty simple, but I am new to interfacing with a device dynamically. What I'm doing is I am making a python executable code, so the user doesn't have to have Idle on their computer or any kind of python interpreter, which means I don't know which USB port the device will be plugged in to. The program needs to be able to open a connection to a device that is connected through a serial to usb converter. How can I determine which connected device is the correct device to open a port to? I am using pySerial to interact with the device. Any help would be greatly appreciated.

No matter how you configure your device, at some point you're probably going to have to ask the user where the port is, or poll all serial devices for a known response. (Polling has it's pitfalls though, so read on!). Unlike USB devices, there is no vendor/device ID that is made known to the OS when you attach a plain-old serial device.
First you need to find the serial ports. Here's a question that might help: What is the cross-platform method of enumerating serial ports in Python (including virtual ports)?.
Once you have a list of serial ports, you could ask the user whether they know which one to use. If they do, problem solved!
If they don't, you could offer to poll ALL serial devices with some data that you know will yield a certain response from your device. Keep in mind though that if the user has other serial devices attached, your string of "hello" bytes might actually be the self-destruct sequence for some other device! Hence, you should warn the user that polling may interfere with other devices, and always prompt them before you do so.
Without knowing more about your code (eg. what comms framework, if any, you're using; are you doing this in the console or are you using a GUI toolkit, etc), it's impossible to say what the best way to code this might be. In the simplest case, you could just loop over all of your serial devices, send the greeting and check for a response. (You could also do this in parallel: loop once for the greeting, and loop again to check what's in the buffer. If you're going to get more fancy than that, use a proper library.)
Side note: You might be able to get around this if you have a built-in converter that you can set the vendor/device ID for, but the converter will still be automatically detected by any modern OS and enumerated as a serial port; you won't get to talk to it directly as a USB device. It could be possible to figure out which port goes with which ID, but I've never tried to do that. But this approach is useless if you're not the one who gets to pick the converter (eg. if it's a user-supplied cable).

This is the way that I used for identify Serial-USB conveter device like Arduino.
It checks 'USB' String of device description.
import serial as ser
import serial.tools.list_ports as prtlst
global COMs
COMs=[]
def getCOMs():
global COMs
pts= prtlst.comports()
for pt in pts:
if 'USB' in pt[1]: #check 'USB' string in device description
COMs.append(pt[0])

The only way I can think to get around the problem with probing unknown devices is to have the device send unsolicited "hello" responses continually. That was you can just connect to all serial devices and listen for the "hellos". Connecting and listening to a serial device shouldn't ever mess it up.
The downside is you have these messages cluttering up your serial stream. You could have a "I'm here now, stfu" command but then you can only connect to it once.
FTDI chips have a method of identification but you have to use their library to access the data.