We Keep Coding

How to connect to a bluetooth profile using dbus APIs

I have a python3 script that successfully opens a RFCOMM socket to a server using old-style bluetooth. I'm trying to accomplish the same thing using dbus, which is the way I'm reading you're supposed to use bluetooth on Linux these days. (This is a proof-of-concept for significant changes to be made to a Linux app written in C.)
When I run the script below I see this:
connecting...
ex from ConnectProfile(): g-io-error-quark: GDBus.Error:org.bluez.Error.NotAvailable: Operation currently not available (36)
onPropertiesChanged( org.bluez.Device1 {'Connected': True} [] )
onPropertiesChanged( org.bluez.Device1 {'ServicesResolved': True} [] )
onPropertiesChanged( org.bluez.Device1 {'ServicesResolved': False, 'Connected': False} [] )
Note that the property changes happen after the call to ConnectProfile fails. I've seen suggestions that I should be opening an RFCOMM socket from inside the property-changed callback, taking advantage of the moment when the connection is open. But server-side (I'm using the excellent bluez-rfcomm-example on github) dbus/bluez takes care of creating the socket: you just get passed a file descriptor. I'm expecting ConnectProfile to work similarly, but can't find any examples.
How should I modify my new_style() function so that it gives me a working socket?
Thanks,
--Eric
#!/usr/bin/env python3
# for new_style()
from pydbus import SystemBus
from gi.repository import GLib
# for old_style()
import bluetooth
PROFILE = 'b079b640-35fe-11e5-a432-0002a5d5c51b'
ADDR = 'AA:BB:CC:DD:EE:FF'
# Works fine. But you're supposed to use dbus these days
def old_style():
service_matches = bluetooth.find_service(uuid=PROFILE, address=ADDR)
if len(service_matches):
first_match = service_matches[0]
port = first_match['port']
host = first_match['host']
sock = bluetooth.BluetoothSocket(bluetooth.RFCOMM)
sock.connect((host, port))
while True:
data = input()
if not data:
break
sock.send(data)
sock.close()
# Does not work. First an exception fires:
# g-io-error-quark: GDBus.Error:org.bluez.Error.NotAvailable: Operation currently not available (36)
# then onPropertiesChanged lists stuff -- after the failure, not during the connection attempt.
def new_style():
nucky = SystemBus().get('org.bluez', '/org/bluez/hci0/dev_' + ADDR.replace(':', '_'))
# Callback: (s, a{sv}, as)
nucky.onPropertiesChanged = lambda p1, p2, p3: print('onPropertiesChanged(', p1, p2, p3, ')')
def try_connect():
print('connecting...')
try:
nucky.ConnectProfile(PROFILE)
except Exception as ex:
print('ex from ConnectProfile():', ex)
GLib.timeout_add( 250, try_connect )
GLib.MainLoop().run()
if False:
old_style()
else:
new_style()
(Added later)
Let me clarify my question. On a Linux box I'm running a bluez-rfcomm-example server that I modified to use a custom Service UUID. It probably creates a service record, but on the client (Android) side these three lines of Java are enough to get a connected socket to it (assuming the server has bluetooth mac AA:BB:CC:DD:EE:FF and the two are paired):
BluetoothDevice remote = BluetoothAdapter.getDefaultAdapter().getRemoteDevice( "AA:BB:CC:DD:EE:FF" );
BluetoothSocket socket = remote.createRfcommSocketToServiceRecord( MY_SERVICE_UUID );
socket.connect();
Is there a way to do this on Linux using dbus/bluez that is remotely close to this simple? I'm assuming Device1/ConnectProfile(UUID) is what I want -- that it's the same thing as createRfcommSocketToServiceRecord() -- but that assumption might be totally wrong! Should this even be possible from Linux using blues/dbus? Or should I stick with the older methods?
Thanks, and sorry for the vague initial question.
--Eric

The code below works to get and use a rfcomm socket for a connection to a remote service specified by a UUID. The answer I accepted, by ukBaz, included all I needed, but I didn't understand enough background to make sense of it immediately. I was right that calling ConnectProfile() was the way to start, but missed two things:
Providing a Profile on the calling side was necessary for two reasons. First, it provides a callback by which you get hold of the socket. But without it -- without the NewConnection method specifically -- the connection fails (ConnectProfile() returns an error.)
I needed to make the ConnectProfile() call on a background thread. The callback will come in on the glib loop's main thread, so ConnectProfile(), which doesn't return until the connection succeeds or fails, mustn't block that thread!
It's possible that different Bluetooth connection types require subtly different machinations, but for RFCOMM socket connections anyway this does the trick.
#!/usr/bin/env python3
import socket, threading
from pydbus import SystemBus
from gi.repository import GLib
import dbus, dbus.service, dbus.mainloop.glib
CUSTOM_UUID = 'b079b640-35fe-11e5-a432-0002a5d5c51b'
ADDR = 'AA:BB:CC:DD:EE:FF'
PATH = '/org/neednt/match/remote'
class Profile(dbus.service.Object):
#dbus.service.method("org.bluez.Profile1",
in_signature="oha{sv}", out_signature="")
def NewConnection(self, path, fd, properties):
None
print('NewConnection: fd:', fd);
try:
self.socket = socket.socket(fileno=fd.take())
print('got socket:', self.socket)
self.socket.send(b"You there?")
except Exception as ex:
print('ex:', ex)
def connect_thread_main():
print('connect_thread_main()...')
SystemBus().get('org.bluez', '/org/bluez/hci0/dev_' + ADDR.replace(':', '_')) \
.ConnectProfile(CUSTOM_UUID)
dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
bus = dbus.SystemBus()
Profile(bus, PATH) # added by side-effect apparently
dbus.Interface(bus.get_object("org.bluez","/org/bluez"),
"org.bluez.ProfileManager1") \
.RegisterProfile(PATH, CUSTOM_UUID, {})
threading.Thread(target=connect_thread_main).start()
GLib.MainLoop().run()

There is a good (if slightly old now) blog comparing pybluez and using Python 3 sockets:
https://blog.kevindoran.co/bluetooth-programming-with-python-3/
If you want to do it with the BlueZ D-Bus API then the key documentations is:
https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/doc/profile-api.txt
And the BlueZ example is at:
https://git.kernel.org/pub/scm/bluetooth/bluez.git/tree/test/test-profile
Creating this with pydbus has some issues as documented at: https://github.com/LEW21/pydbus/issues/54

Why does pyzmq subscriber behave differently with asyncio?

I have an XPUB/XSUB device and a number of mock publishers running in one process. In a separate process, I want to connect a subscriber and print received message to the terminal. Below I will show two variants of a simple function to do just that. I have these functions wrapped as command-line utilities.
My problem is that the asyncio variant never receives messages.
On the other hand, the non-async variant works just fine. I have tested all cases for ipc and tcp transports. The publishing process never changes in my tests, except when I restart it to change transport. The messages are short strings and published roughly once per second, so we're not looking at performance problem.
The subscriber program sits indefinitely at the line msg = await sock.receive_multipart(). In the XPUB/XSUB device I have instrumentation that shows the forwarding of the sock.setsockopt(zmq.SUBSCRIBE, channel.encode()) message, same as when the non-async variant connects.
The asyncio variant (not working, as described)
def subs(url, channel):
import asyncio
import zmq
import zmq.asyncio
ctx = zmq.asyncio.Context.instance()
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
async def task():
while True:
msg = await sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
try:
asyncio.run(task())
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
The regular blocking variant (works fine)
def subs(url, channel):
import zmq
ctx = zmq.Context.instance()
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
def task():
while True:
msg = sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
try:
task()
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
For this particular tool there is no need to use asyncio. However, I am experiencing this problem elsewhere in my code too, where an asynchronous recv never receives. So I'm hoping that by clearing it up in this simple case I'll understand what's going wrong in general.
My versions are
import zmq
zmq.zmq_version() # '4.3.2'
zmq.__version__ # '19.0.2'
I'm on MacOS 10.13.6.
I'm fully out of ideas. Internet, please help!

A working async variant is
def subs(url, channel):
import asyncio
import zmq
import zmq.asyncio
ctx = zmq.asyncio.Context.instance()
async def task():
sock = ctx.socket(zmq.SUB)
sock.connect(url)
sock.setsockopt(zmq.SUBSCRIBE, channel.encode())
try:
while True:
msg = await sock.recv_multipart()
print(' | '.join(m.decode() for m in msg))
finally:
sock.setsockopt(zmq.LINGER, 0)
sock.close()
asyncio.run(task())
I conclude that, when using asyncio zmq, sockets must be created with a call running on the event loop from which the sockets will be awaited. Even though the original form did not do anything fancy with event loops, it appears that the socket has an event loop different from that used by asyncio.run. I'm not sure why, and I didn't open an issue with pyzmq because their docs show usage as in this answer, without comment.
Edit in response to a comment:
asyncio.run always creates a new event loop, so the loop presumably created for the sockets instantiated outside of the co-routine passed to asyncio.run (as in the asyncio variant in the original question) is obviously different.

Python multithreaded ZeroMQ REQ-REP

I am looking to implement a REQ-REP pattern with Python and ZeroMQ using multithreading.
With Python, I can create a new thread when a new client connects to the server. This thread will handle all communications with that particular client, until the socket is closed:
# Thread that will handle client's requests
class ClientThread(threading.Thread):
# Implementation...
def __init__(self, socket):
threading.Thread.__init__(self)
self.socket = socket
def run(self):
while keep_alive:
# Thread can receive from client
data = self.socket.recv(1024)
# Processing...
# And send back a reply
self.socket.send(reply)
while True:
# The server accepts an incoming connection
conn, addr = sock.accept()
# And creates a new thread to handle the client's requests
newthread = ClientThread(conn)
# Starting the thread
newthread.start()
Is it possible to do the same[*] using ZeroMQ? I have seen some examples of multithreading with ZeroMQ and Python, but in all of them a pool of threads is created with a fixed number of threads at the beginning and it seems to be more oriented to load balancing.
[*] Notice what I want is to keep the connection between a client and its thread alive, as the thread is expecting multiple REQ messages from the client and it will store information that must be kept between messages (i.e.: a variable counter that increments its value on a new REQ message; so each thread has its own variable and no other client should ever be able to access that thread). New client = new thread.

Yes, ZeroMQ is a powerful can-do toolbox
However, the major surprise will be, that ZeroMQ <socket>-s are by far more structured than their plain counterparts, you use in the sample.
{ aZmqContext -> aZmqSocket -> aBehavioralPrimitive }
ZeroMQ builds a remarkable, abstraction-rich framework, under a hood of a "singleton" ZMQ-Context, which is (and shall remain) the only thing used as "shared".
Threads shall not "share" any other "derived" objects, the less any their state, as there is a strong distributed-responsibility framework architecture implemented, both in the sake of clean-design and a high performance & low-latency.
For all ZMQ-Socket-s one shall rather imagine a much smarter, layered sub-structure, where one receives off-loaded worries about I/O-activities ( managed inside ZMQ-Context responsibility -- thus keep-alive issues, timing issues and fair-queue buffering / select-polling issues simply cease to be visible for you ... ), with one sort of a formal communication pattern behaviour ( given by a chosen ZMQ-Socket-type archetype ).
Finally
ZeroMQ and similarly nanomsg libraries are rather LEGO-like projects, that empower you, as an architect & designer, more than one typically realises at the very beginning.
One thus can focus on distributed-system behaviour, as opposed to lose time and energy on solving just-another-socket-messaging-[nightmare].
( Definitely worth a look into both books from Pieter Hintjens, co-father of the ZeroMQ. There you find plenty Aha!-moments on this great subject. )
... and as a cherry on a cake -- you get all of this as a Transport-agnostic, universal environment, whether passing some messages on inproc://, other over ipc:// and also in parallel listening / speaking over tcp:// layers.
EDIT#12014-08-19 17:00 [UTC+0000]
Kindly check comments below and further review your -- both elementary and advanced -- design-options for a <trivial-failure-prone>-spin-off processing, for a <load-balanced>-REP-worker queueing, for a <scale-able>-distributed processing and a <fault-resilient_mode>-REP-worker binary-start shaded processing.
No heap of mock-up SLOC(s), no single code-sample will do a One-Size-Fits-All.
This is exponentially valid in designing distributed messaging systems.
"""REQ/REP modified with QUEUE/ROUTER/DEALER add-on ---------------------------
Multithreaded Hello World server
Author: Guillaume Aubert (gaubert) <guillaume(dot)aubert(at)gmail(dot)com>
"""
import time
import threading
import zmq
print "ZeroMQ version sanity-check: ", zmq.__version__
def aWorker_asRoutine( aWorker_URL, aContext = None ):
"""Worker routine"""
#Context to get inherited or create a new one trick------------------------------
aContext = aContext or zmq.Context.instance()
# Socket to talk to dispatcher --------------------------------------------------
socket = aContext.socket( zmq.REP )
socket.connect( aWorker_URL )
while True:
string = socket.recv()
print( "Received request: [ %s ]" % ( string ) )
# do some 'work' -----------------------------------------------------------
time.sleep(1)
#send reply back to client, who asked --------------------------------------
socket.send( b"World" )
def main():
"""Server routine"""
url_worker = "inproc://workers"
url_client = "tcp://*:5555"
# Prepare our context and sockets ------------------------------------------------
aLocalhostCentralContext = zmq.Context.instance()
# Socket to talk to clients ------------------------------------------------------
clients = aLocalhostCentralContext.socket( zmq.ROUTER )
clients.bind( url_client )
# Socket to talk to workers ------------------------------------------------------
workers = aLocalhostCentralContext.socket( zmq.DEALER )
workers.bind( url_worker )
# --------------------------------------------------------------------||||||||||||--
# Launch pool of worker threads --------------< or spin-off by one in OnDemandMODE >
for i in range(5):
thread = threading.Thread( target = aWorker_asRoutine, args = ( url_worker, ) )
thread.start()
zmq.device( zmq.QUEUE, clients, workers )
# ----------------------|||||||||||||||------------------------< a fair practice >--
# We never get here but clean up anyhow
clients.close()
workers.close()
aLocalhostCentralContext.term()
if __name__ == "__main__":
main()

ZeroMQ: HWM on PUSH does not work

I am trying to write a server/client script with a server that vents the tasks, and multiple workers that execute it.
The problem is that my ventilator has so many tasks that it would fill up the memory in a heartbeat.
I tried to set the HWM before it binds, but with no success. It just keeps on sending messages as soon as a worker connects, completely disregarding the HWM that was set. I also have a sink that keeps record of the tasks that were done.
server.py
import zmq
def ventilate():
context = zmq.Context()
# Socket to send messages on
sender = context.socket(zmq.PUSH)
sender.setsockopt(zmq.SNDHWM, 30) #Big messages, so I don't want to keep too many in queue
sender.bind("tcp://*:5557")
# Socket with direct access to the sink: used to syncronize start of batch
sink = context.socket(zmq.PUSH)
sink.connect("tcp://localhost:5558")
print "Sending tasks to workers…"
# The first message is "0" and signals start of batch
sink.send('0')
print "Sent starting signal"
while True:
sender.send("Message")
if __name__=="__main__":
ventilate()
worker.py
import zmq
from multiprocessing import Process
def work():
context = zmq.Context()
# Socket to receive messages on
receiver = context.socket(zmq.PULL)
receiver.connect("tcp://localhost:5557")
# Socket to send messages to
sender = context.socket(zmq.PUSH)
sender.connect("tcp://localhost:5558")
# Process t asks forever
while True:
msg = receiver.recv_msg()
print "Doing sth with msg %s"%(msg)
sender.send("Message %s done"%(msg))
if __name__ == "__main__":
for worker in range(10):
Process(target=work).start()
sink.py
import zmq
def sink():
context = zmq.Context()
# Socket to receive messages on
receiver = context.socket(zmq.PULL)
receiver.bind("tcp://*:5558")
# Wait for start of batch
s = receiver.recv()
print "Received start signal"
while True:
msg = receiver.recv_msg()
print msg
if __name__=="__main__":
sink()

Ok, I had a play around, I don't think the issue is with the PUSH HWM, but rather that you can't set a HWM for PULL. If you look at this documentation, you can see there it says N/A for action on HWM.
The PULL sockets seem to be taking hundreds of messages each (and I did try setting a HWM just in case it did anything on the PULL socket. It didn't.). I evidenced this by changing the ventilator to send messages with an incrementing integer, and changing each worker in the pool to wait 2 seconds between calls to recv(). The workers print out that they are processing messages with vastly different integers. For instance, one worker will be working on message 10, while the next is working on message 400. As time goes on, you see the worker who was processing message 10, is now processing message 11, 12, 13, etc. while the other is processing 401, 402, etc.
This indicates to me that the ZMQ_PULL socket is buffering the messages somewhere. So while the ZMQ_PUSH socket does have a HWM, the PULL socket is requesting messages quickly, despite them not actually being accessed by a call to recv(). So that results in the PUSH HWM effectively being ignored if a PULL socket is connected. As far as I can see, you can't control the length of the buffer of the PULL socket (I would expect the RCVHWM socket option to control this but it doesn't appear to).
This behaviour of course begs the question what is the point of the ZMQ_PULL HWM option, which only makes sense to have if you can also control the receiving sockets HWM.
At this point, I'd start asking the 0MQ people whether you are missing something obvious, or if this is considered a bug.
Sorry I couldn't be more help!

ZeroMQ has buffers on both sending and receiving ends of a socket, hence you need to set high water marks on both the PUSH and the PULL socket in your code (and indeed before a bind() or connect()).
In the Python bindings this is now conveniently done via socket.hwm = 1 which will set both ZMQ_SNDHWM and ZMQ_RCVHWM in one go.

Interprocess communication in Python

What is a good way to communicate between two separate Python runtimes? Thing's I've tried:
reading/writing on named pipes e.g. os.mkfifo (feels hacky)
dbus services (worked on desktop, but too heavyweight for headless)
sockets (seems too low-level; surely there's a higher level module to use?)
My basic requirement is to be able to run python listen.py like a daemon, able to receive messages from python client.py. The client should just send a message to the existing process and terminate, with return code 0 for success and nonzero for failure (i.e. a two-way communication will be required.)

The multiprocessing library provides listeners and clients that wrap sockets and allow you to pass arbitrary python objects.
Your server could listen to receive python objects:
from multiprocessing.connection import Listener
address = ('localhost', 6000) # family is deduced to be 'AF_INET'
listener = Listener(address, authkey=b'secret password')
conn = listener.accept()
print 'connection accepted from', listener.last_accepted
while True:
msg = conn.recv()
# do something with msg
if msg == 'close':
conn.close()
break
listener.close()
Your client could send commands as objects:
from multiprocessing.connection import Client
address = ('localhost', 6000)
conn = Client(address, authkey=b'secret password')
conn.send('close')
# can also send arbitrary objects:
# conn.send(['a', 2.5, None, int, sum])
conn.close()

Nah, zeromq is the way to go. Delicious, isn't it?
import argparse
import zmq
parser = argparse.ArgumentParser(description='zeromq server/client')
parser.add_argument('--bar')
args = parser.parse_args()
if args.bar:
# client
context = zmq.Context()
socket = context.socket(zmq.REQ)
socket.connect('tcp://127.0.0.1:5555')
socket.send(args.bar)
msg = socket.recv()
print msg
else:
# server
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind('tcp://127.0.0.1:5555')
while True:
msg = socket.recv()
if msg == 'zeromq':
socket.send('ah ha!')
else:
socket.send('...nah')

Based on #vsekhar's answer, here is a Python 3 version with more details and multiple connections:
Server
from multiprocessing.connection import Listener
listener = Listener(('localhost', 6000), authkey=b'secret password')
running = True
while running:
conn = listener.accept()
print('connection accepted from', listener.last_accepted)
while True:
msg = conn.recv()
print(msg)
if msg == 'close connection':
conn.close()
break
if msg == 'close server':
conn.close()
running = False
break
listener.close()
Client
from multiprocessing.connection import Client
import time
# Client 1
conn = Client(('localhost', 6000), authkey=b'secret password')
conn.send('foo')
time.sleep(1)
conn.send('close connection')
conn.close()
time.sleep(1)
# Client 2
conn = Client(('localhost', 6000), authkey=b'secret password')
conn.send('bar')
conn.send('close server')
conn.close()

From my experience, rpyc is by far the simplest and most elegant way to go about it.

I would use sockets; local communication was strongly optimized, so you shouldn't have performance problems and it gives you the ability to distribute your application to different physical nodes if the needs should arise.
With regard to the "low-level" approach, you're right. But you can always use an higher-level wrapper depending on your needs. XMLRPC could be a good candidate, but it is maybe overkill for the task you're trying to perform.
Twisted offers some good protocol simple implementations, such as LineReceiver (for simple line based messages) or the more elegant AMP (which was, by the way, standardized and implemented in different languages).

Check out a cross-platform library/server called RabbitMQ. Might be too heavy for two-process communication, but if you need multi-process or multi-codebase communication (with various different means, e.g. one-to-many, queues, etc), it is a good option.
Requirements:
$ pip install pika
$ pip install bson # for sending binary content
$ sudo apt-get rabbitmq-server # ubuntu, see rabbitmq installation instructions for other platforms
Publisher (sends data):
import pika, time, bson, os
connection = pika.BlockingConnection(pika.ConnectionParameters('localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='logs', type='fanout')
i = 0
while True:
data = {'msg': 'Hello %s' % i, b'data': os.urandom(2), 'some': bytes(bytearray(b'\x00\x0F\x98\x24'))}
channel.basic_publish(exchange='logs', routing_key='', body=bson.dumps(data))
print("Sent", data)
i = i + 1
time.sleep(1)
connection.close()
Subscriber (receives data, can be multiple):
import pika, bson
connection = pika.BlockingConnection(pika.ConnectionParameters(host='localhost'))
channel = connection.channel()
channel.exchange_declare(exchange='logs', type='fanout')
result = channel.queue_declare(exclusive=True)
queue_name = result.method.queue
channel.queue_bind(exchange='logs', queue=queue_name)
def callback(ch, method, properties, body):
data = bson.loads(body)
print("Received", data)
channel.basic_consume(callback, queue=queue_name, no_ack=True)
channel.start_consuming()
Examples based on https://www.rabbitmq.com/tutorials/tutorial-two-python.html

I would use sockets, but use Twisted to give you some abstraction, and to make things easy. Their Simple Echo Client / Server example is a good place to start.
You would just have to combine the files and instantiate and run either the client or server depending on the passed argument(s).