I am working on location-based services project where I have several sensors that need to send asynchronous readings to a server, which will correlate the readings and generate a result. There will be some level of sensor to sensor communication as well, and I am interested in using XMPP as a transport due to its efficient messaging, real-time nature and NAT traversal.
I am hoping to find an example of (python, or any other langauge) XMPP machine to machine (M2M) services, hopefully using a PubSub model for asynchronous communication rather than a polling-based RPC. I have not been able to find any examples online or in XMPP books that I have seen, as they seem to be mostly focused on XMPP for human interaction such as chat, video, etc.
The general requirements that I have to work with are:
1. Multiple sensors sharing data with each other over XMPP
2. Asynchronous (PubSub) communication, subscribing to messages of interest
3. Hopefully written in Python, but any language would be a good starting point
4. Server correlates data from all the sensors and generates results, which can be made available to subscribers
5. Easy configuration / setup through discovery
Any ideas about where to look, or a good starting point would be much appreciated.
Thanks!
XMPP for M2M sounds like a nice idea.
About clients and servers, see http://xmpp.org/about-xmpp/technology-overview/pubsub/
In pubsub server does basically all the hard work, and you have to implement very little intelligence to clients. But this depends on what you want to do with published information. I haven't tested any clients which actually do something with the published information.
This fits the pubsub model of XMPP pretty well.
All your machines would be both publishers and subscribers.
Your processing server in this case would also be another subscriber that will do its data processing as it receives published items.
Any example you find dealing with pubsub is easily applicable. In XMPP, whether the JID (Jabber ID) represents a user of a machine is irrelevant, and pubsub is not actually oriented toward human interaction, unlike say, Multi User Chat.
There are many XMPP servers that support pubsub. I have used Smack and OpenFire for a similar purpose myself. The server is of less importance to you, since any off the shelf product that supports PubSub will do the job. More importantly is a client library that has pubsub support. I know Smack has this, but it is a Java library not python.
I do not know anything with all those requisites but you can use SleekXMPP to build your own. It is pure python and well documented XMPP library. XMPP has been used to do computer-to-computer communication which is quite nice because you can just test it from your own chat client. Look for example, http://www.python.org/about/success/projectpipe/
Good luck
Related
Our use case involves one class that has to remotely initialize several instances of another class (each on a different IoT device) and has to get certain results from each of these instances. At most, we would need to receive 30 messages a second from each remote client, with each message being relatively small. What type of architecture would you all recommend to solve this?
We were thinking that each class that is located on the IoT device will serve as a server and the class that receives the results would be the client, so should we create a server, each with its own channel, for each IoT device? Or is it possible to have each IoT device use the same service on the same server (meaning there would be multiple instances of the same service on the same server but on different devices)?
The question would benefit from additional detail to help guide an answer.
gRPC (and its use of HTTP/2) are 'heavier' protocols than e.g. MQTT. MQTT is more commonly used with IoT devices as it has a smaller footprint. REST/HTTP (even though heavier than MQTT) may also have benefits for you over gRPC/HTTP2.
If you're committed to gRPC, I wonder whether it would not be better to invert your proposed architecture and have the IoT device be the client? This seems to provide additional security in that the clients initiate communications with your servers rather than expose services. Either way (and if you decide to use MQTT), hopefully you'll be using mTLS. I assume (!?) client implementations are smaller than server implementations.
Regardless of the orientation, clients and servers can (independently) stream messages. The IoT devices (client or server) could stream the 30 messages/second. The servers could stream management|control messages.
I've no experience managing fleets of IoT devices but, remote management|monitoring and over-the-air upgrades|patching are, I assume, important requirements for you. gRPC does not limit any of these capabilities but debugging can be more challenging. With e.g. REST/HTTP, it is trivial to curl endpoints but with gRPC (even with the excellent grpcurl) you'll be constrained to the services implemented. Yes, you can't call a non-existent REST API either but I find remote-debugging gRPC services more challenging than REST.
I am working on MQTT and using python paho-mqtt https://pypi.python.org/pypi/paho-mqtt
I am unable to understand how can I publish msg to a specific client or list of clients?
I'll appreciate your help.
This isn't directly possible with strict MQTT, although some brokers may offer that functionality, or you can construct your application so that the topic design works to do what you need.
Although I do agree that in some cases it would be useful to send a message to a particular client (or list of clients) that's simply not how the publish/subscribe messaging paradigm works. Read more on the publish-subscribe pattern on Wikipedia. If all your system needs to do is send messages to unique clients, then I would perhaps suggest thinking of a different architecture for the system you are designing. That being said, you can leverage off pub/sub to achieve what you want using a clever topic design architecture.
For example, let's assume all clients are part of a group (list), you could think of the following topic design:
Unique per client: P2P/< client-name >
List/Group subscription: LIST/< list-name >
For example, P2P/user12345 and LIST/QA where only user12345 subscribes to P2P/user12345 but all users of the QA group subscribe to LIST/QA.
It would be the client's responsibility to ensure that it is subscribed to its own topic(s) (or if your broker allows it, you could also add the topics administratively to non-clean clients).
With this design, a publisher would be able to send a message to a specific user or all members of a defined group (list).
I am implementing a small distributed system (in Python) with nodes behind firewalls. What is the easiest way to pass messages between the nodes under the following restrictions:
I don't want to open any ports or punch holes in the firewall
Also, I don't want to export/forward any internal ports outside my network
Time delay less than, say 5 minutes, is acceptable, but closer to real time would be nice, if possible.
1+2 → I need to use a third party, accessible by all my nodes. From this follows, that I probably also want to use encryption
Solutions considered:
Email - by setting up separate or a shared free email accounts (e.g. Gmail) which each client connects to using IMAP/SMTP
Google docs - using a shared online spreadsheet (e.g. Google docs) and some python library for accessing/changing cells using a polling mechanism
XMPP using connections to a third party server
IRC
Renting a cheap 5$ VPS and setting up a Zero-MQ publish-subscribe node (or any other protocol) forwarded over SSH and having all nodes connect to it
Are there any other public (free) accessible message queues available (or platforms that can be misused as a message queue)?
I am aware of the solution of setting up my own message broker (RabbitMQ, Mosquito) etc and make it accessible to my nodes somehow (ssh-forwardning to a third host etc). But my questions is primarily about any solution that doesn't require me to do that, i.e. any solutions that utilizes already available/accessible third party infrastructure. (i.e. are there any public message brokers I can use?)
How about Mosquitto: message broker that implements the MQ Telemetry Transport protocol versions 3.1 and 3.1.1. MQTT provides a lightweight method of carrying out messaging using a publish/subscribe model. This makes it suitable for "machine to machine" messaging. It supports encryption. Time to setup: approximatively 15 mins you should be up and running. Since it is a message broker, you can write your own code to ensure you can communicate with 3rd party solutions. Also, it achieves soft real-time, but depending on your setup you can achieve hard real-time. After you look into Mosquitto have a look at Paho, which is a port of Mosquito to Eclipse Foundation.
Paho also provides a Python Client, which offers support for both MQTT v3.1 and v3.1.1 on Python 2.7 or 3.x. It also provides some helper functions to make publishing one off messages to an MQTT server very straightforward. Plenty of documentation and examples to get you up and running.
I would recommend RabbitMQ or Redis (RabbitMQ preferred because it is a very mature technology and insanely reliable). ZMQ is an option if you want a single hop messaging system instead of a brokered messaging system such as RabbitMQ but ZMQ is harder to use than RabbitMQ. Depending on how you want to utilize the message passing (is it a task dispatch in which case you can use Celery or if you need a slightly more low-level access in which case use Kombu with librabbitmq transport )
Found https://www.cloudamqp.com/ which offers a free plan with a cloud based installation of RabbitMQ. I will try that and see if it fulfill my needs.
Introduction
I am working on a robotic sampling application. Each robot has a cabled interface with power, TCP/IP and a gas-sensor tubing. The robots are on an ARM platform, and I intend to do most of the programming in Python. The robots move slowly and there is nothing computationally intensive running on them.
Each robot should perform these "services":
Move left/right (for manual control)
Move up/down (for manual control)
Go to next sector
Each robot report these sensor readings or events:
Temperature
End-switch right
Docked at sector with ID: ###
Encoder count [lateral,longitudinal]
Error event
Client - Server Architecture
I view each robot as a client, and the sensor hub computer as the server.
The server will have a known ip and listening port, and allow the robots to connect.
The server will do the measurement scheduling, and command the robots to move from sector to sector.
The server may maintain and update a model of each robot with a state-vector containing:
[ position, switches, sensor-readings, status]
Questions
From debugging serial communication, I have experienced the benefits of having a human readable communication interface with a strict poll-response structure. I am however not sure how we should go forth designing this interface.
Are there any best practices in designing communication interfaces for devices like these?
Should I think about packet loss and corruption, or is this fully handled by TCP?
Should I design everything as services polled by the server, or should the robots broadcast it's sensor readings and events?
Should I implement Acknowledgment of commands, e.g. go-to-next-section
I apologize for the broad and vague problem formulation, this may be more a philosophy question than a software problem. However I will greatly appreciate your thoughts, experiences and advice.
TLDR
What are the guiding principles of designing TCP communication protocols for client-server architectures?
Overall Id suggest using python Twisted to build your server and client (robot side) applications (https://twistedmatrix.com/trac/). Anyways to answer your question:
"Are there any best practices in designing communication interfaces for devices like these?"
See answers to your other questions bellow.
"Should I think about packet loss and corruption, or is this fully handled by TCP?"
TCP guarantees the integrity of the data you are getting. The primary things to worry about is if the client/servers are connected or not. You can use a ReconnectingClientProtocol to make your connections a little more robust when server is restarted (see Twisted specs). Also be aware that TCP is a streaming protocol (you may not get the whole message at once), so make sure you got the whole message before acting on it. If you are sending messages quickly you may also have more than one message in your TCP buffer for that client.
"Should I design everything as services polled by the server, or should the robots broadcast it's sensor readings and events?"
Avoid polling. When the robots start up they should establish a persistent TCP connection with the server. Messages should be sent and received (handled) asynchronously.
"Should I implement Acknowledgment of commands, e.g. go-to-next-section"
Wouldn't hurt. Would be good for flow control within your application as well as recovering from situations where the server or robots are restarted and you can't be sure whether a message was processed or not.
"What are the guiding principles of designing TCP communication protocols for client-server architectures?"
Probably the thing to do for your app is to design a simple command response protocol. Start by designing simple message sets, on going from client to server, the other from server to client. You could use simple human readable XML message set as follows:
Server to Client
<SCMessage type="TurnRight"></SCMessage>
<SCMessage type="TurnLeft"></SCMessage>
<SCMessage type="NextSector"><param key="sectorName" value="B"/></SCMessage>
<SCMessage type="GetStatus"></SCMessage>
<SCMessage type="Ack"></SCMessage>
Client to Server
<SCMessage type="SensorUpdate"><param key="data" value="123"/></SCMessage>
<SCMessage type="StatusChanged"><param key="status" value="Good"/></SCMessage>
....
<SCMessage type="Ack"></SCMessage>
So when parsing these messages you can teas them apart by looking for the SCMessage start stop tags. Once you have a message you could then use an XML parser to parse the messages contents. Alternatively you could use JSON which would actually probably be a lot easier (basically you'd be sending little dictionaries back and forth).
You've got a lot of reading to-do ;) Id start by reading up on python Twisted a bit and make little toy programs to get comfortable with things.
The question: How do I create a python application that can connect and send packets over the internet to another computer running the same application? Is there any existing code/library I could use?
The background: I am pretty new to programming (HS senior). I've created a lot of simple things in python but I've recently decided to start on a bigger project. I'm considering creating a Magic: the Gathering booster draft simulator, but I'm not sure if it is feasible given my skill set so I'm asking around before I get started. The application would need to send data between computers about which cards are being picked/passed.
Thanks!
Twisted is a python event-driven networking engine licensed under MIT. Means that a single machine can communicate with one or more other machines, while doing other things between data being received and sent, all asynchronously, and running a in a single thread/process.
It supports many protocols out of the box, so you can just as well using an existing one. That's better because you get support for the protocol from 3rd party software (i.e. using HTTP for communication means middleware software that uses HTTP will be compatible: proxies etc.)
It also makes easy to create your own communication protocol, if that's what you want.
The documentation is filled with examples.
The standard library includes SocketServer (documented here), which might do what you want.
However I wonder if a better solution might be to use a message queue. Lots of good implementations already exist, including Python interfaces. I've used RabbitMQ before. The idea is that the computers both subscribe to the queue, and can either post or listen for events.
A great place to start looking is the Python Standard Library. In particular there are a few sections that will be relevant:
18. Interprocess Communication and Networking
19. Internet Data Handling
21. Internet Protocols and Support
Since you mentioned that you have no experience with this, I'd suggest starting with a HTTP based implementation. The HTTP protocol is fairly simple and easy to work with. In addition, there are nice frameworks to support this operation, such as webpy for the server and HTTPLib from the standard library for the client.
If you really want to dive into networking, then a socket based implementation might be educational. This will teach you the underlying concepts that are used in lots of networking code while resulting in an interface that is similar to a file stream.
Also, check out Pyro (Python remoting objects). Se this answer for more details.
Pyro basically allows you to publish Python object instances as services that can be called remotely. Pyro is probably the easiest way to implement Python-to-python process communication.
It's also worth looking at Kamaelia for this sort of thing - it's original usecase was network systems, and makes building such things relatively intuitive.
Some links: Overview of basic TCP system, Simple Chat server, Building a layered protocol, walk-through of how to evolve new components. Other extreme: P2P radio system: source, peer.
If it makes any difference, we've tested whether the system is accessible to novices through involvement in google summer of code 3 years in a row, actively taking on both experienced and inexperienced developers. All of them managed to build useful systems.
Essentially, if you've ever played with unix pipelines the ideas should be familiar.
Caveat: I wrote major chunks of Kamaelia :-)
If you want to learn how to do these things though, playing with a few different approaches makes sense, and you should definitely check out Twisted (the standard answer to this question), Pyro & the standard library tools. Each has a different approach, and learning them will definitely benefit you!
However, like nosklo, I would recommend against using the socket library directly and use a library instead - simply because it is much much harder to get sockets programming correct than people tend to realise.
Communication will take place with sockets, one way or another. Just a question of whether you use existing higher-level libraries, or roll your own.
If you're doing this as a learning experience, probably want to start as low-level as you can, to see the real nuts and bolts. Which means you probably want to start with a SocketServer, using a TCP connection (TCP is basically guaranteed delivery of data; UDP is not).
Google for some simple example code. Setting one up is very easy. But you will have to define all the details of your communications protocol: which end sends when and what, which end listens and when, what exactly the listener will expect, does it reply to confirm receipt, etc.