We Keep Coding

cannot keep a stable session open with bloomberg blpapi Python

I recently encountered an issue that I have not been able to solve, despite calling the Bloomberg helpdesk and researching thoroughly the internet for similar cases.
In short, I am using the official Python blpapi from Bloomberg (https://github.com/msitt/blpapi-python) and now am experiencing some connectivity issue: I cannot leave a session opened.
Here is the code I am running: https://github.com/msitt/blpapi-python/blob/master/examples/SimpleHistoryExample.py
I simply added a "while True loop" and a "time.sleep" in it so that I can keep the session open and refresh my data every 30 seconds (this is my use case).
This use to run perfectly fine for days, however, since last Friday, I am now getting those log messages:
22FEB2021_08:54:18.870 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
22FEB2021_08:54:23.755 29336:26880 WARN blpapi_platformcontroller.cpp:377 blpapi.session.platformcontroller.{1} Connectivity lost, no connected endpoints.
22FEB2021_08:54:31.867 29336:26880 WARN blpapi_platformcontroller.cpp:344 blpapi.session.platformcontroller.{1} Connectivity restored.
22FEB2021_08:54:32.731 29336:26880 WARN blpapi_subscriptionmanager.cpp:7437 blpapi.session.subscriptionmanager.{1} Could not find a service for serviceCode: 90.
which goes on and on and on, along with those responses as well:
SessionConnectionDown = {
server = "localhost:8194"
}
ServiceDown = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
SessionConnectionUp = {
server = "localhost:8194"
encryptionStatus = "Clear"
compressionStatus = "Uncompressed"
}
ServiceUp = {
serviceName = "//blp/refdata"
servicePart = {
publishing = {
}
}
}
I still can pull the data from the bloomberg API: I see the historical data request results just fine. However:
Those service/session status messages messes up my code (I could still ignore them)
For some reason the connect/reconnect also messes my Excel BBG in the background and prevent me from using the BBG excel add-in at all! I now have those "#N/A Connection" outputs in all of my workbooks using bloomberg formulas.
screenshot from excel
Has anyone ever encountered such cases? If yes, please do not hesitate to share your experience, any help is more than appreciated!
Wishing you all a great day,
Adrien

I cannot comment yet so I will try to "answer" it. I use blpapi everyday and pull data all day. I am a BBG Anywhere user and never have any session issues. If you log in from a different device it will kill your session for the Python app. Once you log back in where the python app is running it will connect again.
Why do you have another while loop and sleep to keep the session alive? You should create a separate session and always call it to run your request. You should not need any "keep alive" code inside the request. Just don't call session.stop(). This is what I ended up doing after much trial and error from not knowing what to do.
I run my model using Excel, trying to move away from any substantial code in Excel and use it as a GUI until I can migrate to a custom GUI. I also have BDP functions in my Excel and they work fine.
import blpapi
# removed optparse because it is deprecated.
from argparse import ArgumentParser
SERVICES = {}
def parseCmdLine():
parser = ArgumentParser(description='Retrieve reference data.')
parser.add_argument('-a',
'--ip',
dest='host',
help='server name or IP (default: %(default)s)',
metavar='ipAddress',
default='localhost')
parser.add_argument('-p',
dest='port',
type=int,
help='server port (default: %(default)s)',
metavar='tcpPort',
default=8194)
args = parser.parse_args()
return args
def start_session():
"""Standard session for synchronous refdata requests. Upon creation
the obj is held in SERVICES['session'].
Returns:
obj: A session object.
"""
args = parseCmdLine()
# Fill SessionOptions
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(args.host)
sessionOptions.setServerPort(args.port)
# Create a Session
session = blpapi.Session(sessionOptions)
# Start a Session
session.start()
SERVICES['session'] = session
return SERVICES['session']
def get_refDataService():
"""Create a refDataService object for functions to use. Upon creation
it is held in SERVICES['refDataService'].
Returns:
obj: refDataService object.
"""
# return the session and request because requests need session.send()
global SERVICES
if 'session' not in SERVICES:
start_session()
session = SERVICES['session']
# Check for SERVICES['refdata'] not needed because start_session()
# is called by this function and start_session() is never called on its own.
session.openService("//blp/refdata")
refDataService = session.getService("//blp/refdata")
SERVICES['refDataService'] = refDataService
session = SERVICES['session']
refDataService = SERVICES['refDataService']
print('get_refDataService called. Curious when this is called.')
return session, refDataService
# sample override request
def ytw_oride_muni(cusip_dict):
"""Requests the Price To Worst for a dict of cusips and Yield To Worst values.
The dict must be {'cusip' : YTW}. Overrides apply to each request, so this
function is designed for different overrides for each cusip. Although they could
all be the same as that is not a restriction.
Returns: Single level nested dict
{'cusip Muni': {'ID_BB_SEC_NUM_DES': 'val', 'PX_ASK': 'val1', 'YLD_CNV_ASK': 'val2'}}
"""
session, refDataService = get_refDataService()
fields1 = ["ID_BB_SEC_NUM_DES", "PX_ASK", "YLD_CNV_ASK"]
try:
values_dict = {}
# For different overrides you must send separate requests.
# This loops and creates separate messages.
for cusip, value in cusip_dict.items():
request = refDataService.createRequest("ReferenceDataRequest")
# append security to request
request.getElement("securities").appendValue(f"{cusip} Muni")
# append fields to request
request.getElement("fields").appendValue(fields1[0])
request.getElement("fields").appendValue(fields1[1])
request.getElement("fields").appendValue(fields1[2])
# add overrides
overrides = request.getElement("overrides")
override1 = overrides.appendElement()
override1.setElement("fieldId", "YLD_CNV_ASK")
override1.setElement("value", f"{value}")
session.sendRequest(request)
# Process received events
while(True):
# We provide timeout to give the chance to Ctrl+C handling:
ev = session.nextEvent(500)
# below msg.messageType == ReferenceDataResponse
for msg in ev:
if msg.messageType() == "ReferenceDataResponse":
if msg.hasElement("responseError"):
print(msg)
if msg.hasElement("securityData"):
data = msg.getElement("securityData")
num_cusips = data.numValues()
for i in range(num_cusips):
sec = data.getValue(i).getElement("security").getValue()
try:
des = data.getValue(i).getElement("fieldData").getElement("ID_BB_SEC_NUM_DES").getValue()
except:
des = None
try:
ptw = data.getValue(i).getElement("fieldData").getElement("PX_ASK").getValue()
except:
ptw = None
try:
ytw = data.getValue(i).getElement("fieldData").getElement("YLD_CNV_ASK").getValue()
except:
ytw = None
values = {'des': des, 'ptw': ptw, 'ytw': ytw}
# Response completly received, so we could exit
if ev.eventType() == blpapi.Event.RESPONSE:
values_dict.update({sec: values})
break
finally:
# Stop the session
# session.stop()
return values_dict

Python to Json DeserializerError

I am trying to read values from PLC using the IOTHub telemetry and stream them to PowerBI using azure.
I was able to connect to the PLC and read the values when I run the code and am able to see the hub receive the messages. However, the stream gives me an error that says 'InputDeserializerError.InvalidData'. I am not sure where I went wrong. Please have a look at the below code and suggest how I could serialize it. When I remove the string values and only run integers & float, the stream picks it up.
# Define the JSON message to send to IoT Hub.
MSG_TXT = "{\"Bin1Load\": %s,\"Bin1Grower\": %s,\"Bin1Variety\": %s,\"Bin1StatedTn\": %.2f,\"Bin1S\": %.3f,\"Bin1CalcV\": %.2f}"
def send_confirmation_callback(message, result, user_context):
print ( "IoT Hub responded to message with status: %s" % (result) )
def iothub_client_init():
# Create an IoT Hub client
client = IoTHubClient(CONNECTION_STRING, PROTOCOL)
return client
def iothub_client_telemetry_sample_run():
try:
client = iothub_client_init()
print ( "IoT Hub device sending periodic messages, press Ctrl-C to exit" )
while True:
# Build the message with simulated telemetry values.
b1load = comm.Read('Bin01.Content.Load')
b1grower = comm.Read('Bin01.Content.Grower')
b1variety = comm.Read('Bin01.Content.Variety')
b1statedton = comm.Read('Bin01.Content.Stated_Tn')
b1s = comm.Read('Bin01.Content.SG')
b1calcvol = comm.Read('Bin01.Content.Calc_Vol')
msg_txt_formatted = MSG_TXT % (b1loads, b1growers, b1varietys, b1statedton, b1s, b1calcvol)
message = IoTHubMessage(msg_txt_formatted)```

Solution:
Don't try to build JSON with string formatting. Create a dictionary and use json.dumps().

Configuring errbot presence in HipChat

I'm trying to use Errbot with HipChat server (not cloud). It looks like Errbot is confused between a room name and its xmpp jid. For example, testing with room named BotTest that has jid of 1_bottest#conf.btf.hipchat.com:
When I set CHATROOM_PRESENCE to the room name and send any message in that room, Errbot crashes with the error "Room '1_bottest#conf.btf.hipchat.com' not found".
When I set it instead to the room's jid, Errbot creates a new room with the same name as the jid, and a jid of '1_1_bottestconf.btf.hipchat.com#conf.btf.hipchat.com'. Then when I send a message in that new room it crashes with the error "Room '1_1_bottestconf.btf.hipchat.com#conf.btf.hipchat.com' not found".
My config.py is below:
import logging
BACKEND = 'XMPP' # defaults to XMPP
BOT_DATA_DIR = r'/auto/home.nas03/eeshel/work/errbot/data'
BOT_EXTRA_PLUGIN_DIR = '/auto/home.nas03/eeshel/work/errbot/plugins'
BOT_LOG_FILE = r'/auto/home.nas03/eeshel/work/errbot/errbot.log'
BOT_LOG_LEVEL = logging.DEBUG
TEXT_COLOR_THEME = 'dark'
BOT_ADMINS = ('1_8#chat.btf.hipchat.com', )
BOT_PREFIX = '\\'
BOT_ALT_PREFIXES = ('Hermes',)
BOT_ALT_PREFIX_SEPARATORS = (':', ',', ';')
BOT_ALT_PREFIX_CASEINSENSITIVE = True
CHATROOM_FN = 'Hermes the Bot'
CHATROOM_PRESENCE = ('1_bottest#conf.btf.hipchat.com',)
BOT_IDENTITY = {
'username' : '1_2#hipchat.eng.<ourdomain>',
'password' : '*****',
'token' : '*****',
'endpoint' : '10.18.0.185',
}
XMPP_KEEPALIVE_INTERVAL = 60
XMPP_USE_IPV6 = False
XMPP_CA_CERT_FILE = "/etc/ssl/certs/ca-bundle.crt"

You can refer below link ,which will help you http://errbot.io/en/latest/user_guide/configuration/hipchat.html

While the documentation is very good, I also struggled to get these things just right. Once I did, a re-read of the doc makes sense, but hindsight is 20/20!
Here is an example config section to have your errbot connect to Hipchat:
# Assuming you are #jdoe in your Hipchat and you want admin privs for your bot
BOT_ADMINS = ('#jdoe', '#mmouse', )
# Let's assume your bot's Hipchat account is "superbot" with
# full name of "Super Bot"
BACKEND = 'Hipchat'
if BACKEND == 'Hipchat':
# http://errbot.io/en/latest/user_guide/configuration/hipchat.html
BOT_IDENTITY = {
# You get username by logging into Hipchat with the account, then go to
# Profile / XMPP/Jabber Info and find the "Jabber ID"
'username': '123456_1234567#chat.hipchat.com',
# Simply the password of the Hipchat account
'password': 'password12345',
# You get token by logging into Hipchat with the account, then go to
# Profile / API Access and generate an API token--assigning it all
# scopes unless you know otherwise.
'token': 'azwx2BdVzGz0riEjqybWinLbUkBSIz6rpsa259HE',
## If you're using HipChat server (self-hosted HipChat) then you should set
## the endpoint below. If you don't use HipChat server but use the hosted version
## of HipChat then you may leave this commented out.
#'endpoint': 'https://api.hipchat.com'
}
CHATROOM_FN = 'Super Bot'
BOT_ALT_PREFIXES = ('#superbot', '#SuperBot', '#stinkfinger',)
# To "listen" for any mentions of #superbot:
CHATROOM_PRESENCE = ()
# To also join rooms to listen to all messages in those rooms,
# do this instead:
#CHATROOM_PRESENCE = ('Room 1', 'Lounge',)
IMPORTANT also to note that I had to downgrade these packages to get Hipchat integration working. I made these changes in my requirements.txt:
pyasn1==0.3.7
pyasn1-modules==0.1.5
sleekxmpp==1.3.2

How to send multicast message (multiple users) using xmpp and Python xmpppy (XEP-0033: Extended Stanza Addressing)

I am using xmpppy http://xmpppy.sourceforge.net/ to send Jabber notifications, it is working well for single destinations using the following code:
# pip install https://github.com/rochacbruno/xmpppy/tarball/master
import xmpp
JABBER_SETTINGS = {"USERNAME": None, "PASSWORD": None, "DOMAIN": None, "RESOURCE": None}
def get_jabber_client():
client = xmpp.Client(JABBER_SETTINGS.get('DOMAIN'))
client.connect(server=(JABBER_SETTINGS.get('DOMAIN'), '5222'))
client.auth(
JABBER_SETTINGS.get('USERNAME'),
JABBER_SETTINGS.get('PASSWORD'),
JABBER_SETTINGS.get('RESOURCE')
)
client.sendInitPresence()
return client
def send_message(to, message):
client = get_jabber_client()
xmpp_message = xmpp.Message(to, message)
client.send(xmpp_message)
client.disconnect()
send_message("single.destination#domain.com", "Hello World!")
But now I need to send the message to multiple destinations, for now I am doing.
for users in list_of_users:
send_message(user, "Hello World!")
Which works fine, but every time I call it starts the process of authentication and takes a lot of time.
I've tried to create a single client and use the same client to send the message.
def send_message(to, message):
if isinstance(to, basestring):
to = [to]
assert isinstance(to, (list, tuple))
client = get_jabber_client()
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)
client.disconnect()
send_message(['user1...', 'user2...'], "Hello World!")
The code above works, but only the first user in the list gets the message well formatted, the other users receives the message in pure XML.
I saw this code (in .net), mentioning XEP-0033: Extended Stanza Addressing http://forum.ag-software.net/thread/1482-Send-Message-To-all-users-in-contact-list
var addresses = new Addresses();
addresses.AddAddress(new Address
{
Type = Type.to,
Jid = "hildjj#jabber.org/Work",
Description = "Joe Hildebrand"
});
addresses.AddAddress(new Address
{
Type = Type.cc,
Jid = "jer#jabber.org/Home",
Description = "Jeremie Miller"
});
var msg = new Matrix.Xmpp.Client.Message();
msg.Add(addresses);
msg.To = "multicast.jabber.org";
msg.Body = "Hello, world!";
builds the following Xml:
<message to='multicast.jabber.org'>
<addresses xmlns='http://jabber.org/protocol/address'>
<address type='to' jid='hildjj#jabber.org/Work' desc='Joe Hildebrand'/>
<address type='cc' jid='jer#jabber.org/Home' desc='Jeremie Miller'/>
</addresses>
<body>Hello, world!</body>
</message>
But I did not found the way to do the same in Python using xmpppy, any idea on how to build the multicast stanza and send the message to multiple users using Python?
Thanks.

You could probably get this done without XEP-0033. In this loop, you overwrite the value of the variable message the first time, and subsequent messages will get garbled contents:
for destination in to:
message = xmpp.Message(destination, message)
client.send(message)
Try this instead:
for destination in to:
xmpp_message = xmpp.Message(destination, message)
client.send(xmpp_message)

XMPP on Python responds to Gtalk but not to Hangouts

I was trying out a Gtalk bot using python and XMPP.
When I ping the bot using iChat application, I could receive the response back.
But when I ping using Hangouts, I am not able to receive the response message. But still I could see my message at server side logs.
# -- coding: utf-8 -
import xmpp
user="BOTUSERNAME#gmail.com"
password="PASSWORD"
server=('talk.google.com', 5223)
def message_handler(connect_object, message_node):
us = str(message_node.getFrom()).split('/')[0]
if us == 'REALUSERNAME#gmail.com':
us = us[0:4]
print str(message_node)
message = "Welcome to my first Gtalk Bot :) " + us
s= str(message_node.getBody()).replace("\n", "\t")
if s <> 'None' :
print "MESSAGE: " + s
connect_object.send(xmpp.Message( message_node.getFrom() ,message))
jid = xmpp.JID(user)
connection = xmpp.Client(jid.getDomain())
connection.connect(server)
result = connection.auth(jid.getNode(), password )
connection.RegisterHandler('message', message_handler)
connection.sendInitPresence()
while connection.Process(1):
pass
Is this something to do with gtalk moving out of XMPP support?
My Bot is still able to receive message but my Hangouts Application is not receiving response

I was able to fix the issue.
You need to add typ = 'chat' attribute to xmpp.Message
connect_object.send(xmpp.Message( message_node.getFrom() ,message, typ='chat' ))
Now my gTalkBot reponds to my message from hangouts & ichat client.
Many thanks to this stack overflow answer

If you have extended sleekxmpp.ClientXMPP, then you can ensure messages are sent to hangouts by added mtype='chat' to send_message()
bot = MyBot([...])
bot.send_message(mto=JID,mbody=MSG,mtype='chat')