We Keep Coding

Attempting to Understand Functional Arguments

I recognize this may be a very 101 type question, but I'm still having trouble understanding functional programming in general, and have a particular code snippet that I can't make sense of:
Full code, but leaving out most of the function definitions:
import blpapi
import sys
SESSION_STARTED = blpapi.Name("SessionStarted")
SESSION_STARTUP_FAILURE = blpapi.Name("SessionStartupFailure")
SERVICE_OPENED = blpapi.Name("ServiceOpened")
SERVICE_OPEN_FAILURE = blpapi.Name("ServiceOpenFailure")
ERROR_INFO = blpapi.Name("ErrorInfo")
GET_FILLS_RESPONSE = blpapi.Name("GetFillsResponse")
d_service="//blp/emsx.history"
d_host="localhost"
d_port=8194
bEnd=False
class SessionEventHandler():
def processEvent(self, event, session):
try:
if event.eventType() == blpapi.Event.SESSION_STATUS:
self.processSessionStatusEvent(event,session)
elif event.eventType() == blpapi.Event.SERVICE_STATUS:
self.processServiceStatusEvent(event,session)
elif event.eventType() == blpapi.Event.RESPONSE:
self.processResponseEvent(event)
else:
self.processMiscEvents(event)
except:
print ("Exception: %s" % sys.exc_info()[0])
return False
def processSessionStatusEvent(self,event,session):
print ("Processing SESSION_STATUS event")
for msg in event:
pass
def processServiceStatusEvent(self,event,session):
print ("Processing SERVICE_STATUS event")
for msg in event:
pass
def processResponseEvent(self, event):
print ("Processing RESPONSE event")
for msg in event:
global bEnd
bEnd = True
def processMiscEvents(self, event):
print ("Processing " + event.eventType() + " event")
for msg in event:
print ("MESSAGE: %s" % (msg.tostring()))
def main():
sessionOptions = blpapi.SessionOptions()
sessionOptions.setServerHost(d_host)
sessionOptions.setServerPort(d_port)
print ("Connecting to %s:%d" % (d_host,d_port))
eventHandler = SessionEventHandler()
session = blpapi.Session(sessionOptions, eventHandler.processEvent)
if not session.startAsync():
print ("Failed to start session.")
return
global bEnd
while bEnd==False:
pass
session.stop()
I can follow the code up to here:
session = blpapi.Session(sessionOptions, eventHandler.processEvent)
Here, I see I'm calling "Session" from the blpapi library, and passing it some options as well as my eventHandler.processEvent. Here is where I get lost. I look at that particular function, and see:
def processEvent(self, event, session):
try:
if event.eventType() == blpapi.Event.SESSION_STATUS:
self.processSessionStatusEvent(event,session)
elif event.eventType() == blpapi.Event.SERVICE_STATUS:
self.processServiceStatusEvent(event,session)
elif event.eventType() == blpapi.Event.RESPONSE:
self.processResponseEvent(event)
else:
self.processMiscEvents(event)
except:
print ("Exception: %s" % sys.exc_info()[0])
return False
I see that the function is attempting to discern what type of event has been passed in, and will execute a different function within the class depending on that event type. The trouble is, I can't figure out where the event is ever specified! Where does "event" come from? I see it as an argument in that particular function, but no event argument was passed to:
session = blpapi.Session(sessionOptions, eventHandler.processEvent)
So how does it know what to do at this point? How did this "event" object magically appear?
Thanks for entertaining my dumb questions

session = blpapi.Session(sessionOptions, eventHandler.processEvent)
Note that processEvent here lacks parentheses () after it. This means you are passing the function itself as a parameter to the Session class. This class will later call processEvent with appropriate parameters.
Side Note:
I'm still having trouble understanding functional programming
"Functional programming" has a very specific definition and this example isn't it. If you are interested, you can google "functional programming" or read the Wikipedia article to find out more. However, this isn't really important at this stage in your learning process.

How to change "for" into a multithreaded pool in python

So I made this program that I want to loop for ever until closed. So at the moment I use this piece of code;
while True:
a = start();
for aaa in a:
check(a[aaa], 0)
But that is pretty slow. How can I multithread this using this (this is my try, it's incorrect ofcourse);
pool = ThreadPool(threads)
results = pool.map(check, a, 0)
I tried that code, with threads = 1. And it just gave nothing. Could anyone help me with this?
==== EDIT ====
Start function;
def start():
global a
url = "URL_WAS_HERE" // receives a json like {"a":56564356, "b":654653453} etc. etc.
r = requests.get(url)
a = json.loads(r.text)
return a
Check function;
def check(idd, tries):
global checked
global snipe
global notworking
if tries < 1:
checked = checked+1
url = "URL_WAS_HERE"+str(idd) // Receives json with extra information about the id
r = requests.get(url)
try:
b = json.loads(r.text)
if b['rap'] > b['best_price']:
difference = b['rap']-b['best_price'];
print(str(idd)+" has a "+str(difference)+ "R$ difference. Price: "+str(b['best_price'])+" //\\ Rap: "+str(b['rap']))
snipe = snipe+1
except:
time.sleep(1)
tries = tries+1
notworking = notworking+1
check(idd, tries)
settitle("Snipes; "+str(snipe)+" //\\ Checked; "+str(checked)+" //\\ Errors; "+str(notworking))
I hope this helps a bit

Perhaps start by using a documented object, ThreadPoolExecutor. ThreadPool is an undocumented language feature.
The docs offer minimal examples to get you started. For your example try the following construction:
from concurrent.futures import ThreadPoolExecutor, as_completed
values_to_test = a()
result_container = []
with ThreadPoolExecutor(max_workers=2) as executor: # set `max_workers` as appropriate
pool = {executor.submit(check, val, tries=0): val for val in values_to_test}
for future in as_completed(pool):
try:
result_container.append(future.result())
except:
pass # handle exceptions here
If you are set on using the map method, you cannot pass 0 as an argument because it is not an iterable; see the method signature.

Is there a standard approach to returning values from coroutine endpoints

My question:
I would like to know if there is a "best practice" pattern in Python for returning values from coroutine endpoints (aka the "sink" or "consumer"). More generally, how would you approach the following scenario?
My scenario:
I have my (producer) > (filter) > (consumer) coroutine pipeline to process a text-based table and to build a list of dictionaries from it. I would like the object that is built in consumer to be returned to the original caller of producer.
My approach:
My approach has been to set up a unique finish-processing signal that each coroutine checks for. If it hears the signal, then it passes on the signal to its child and yields the returned value. The consumer just yields its current value.
Alternative approaches:
I considered:
Using a global to hold the desired object to be "returned" to the caller.
A class-based approach with regular subroutines.
Reasons why I should maybe reconsider these for my scenario would also be welcome.
My implementation:
Here is a simplified version of what I have done, with all key components included.
import uuid
FINISH_PROCESSING_SIGNAL = uuid.uuid4()
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
return cr
return start
# Sink
#coroutine
def list_builder():
# accepts objects and adds them to a list
_list = []
try:
while True:
data = (yield)
if data is FINISH_PROCESSING_SIGNAL:
yield _list
break
_list.append(data)
except GeneratorExit:
pass
# Filter
#coroutine
def user_data_filter(target=None):
if target is None:
target = list_builder()
header = "-+-"
footer = "Transfer Packets"
username = "User Name"
fullname = "Full Name"
note = "Description"
try:
while True:
user = {}
data = (yield)
if data is FINISH_PROCESSING_SIGNAL:
yield target.send(FINISH_PROCESSING_SIGNAL)
break
line = data
if header in line:
while True:
line = (yield)
if footer in line:
target.send(user)
break
elif username in line:
user["username"] = line.split('|')[1]
elif fullname in line:
user["fullname"] = line.split('|')[1]
elif note in line:
user["note"] = line.split('|')[1]
except GeneratorExit:
target.close()
# Producer
def process_users_table(table, target=None):
if target is None:
target = user_data_filter()
lines = table.split('\r\n')
for line in lines:
target.send(line)
processed_data = target.send(FINISH_PROCESSING_SIGNAL)
return processed_data
if __name__ == '__main__':
test_users_table = \
"""
Item |Value\r\n
----------------+-----------------------\r\n
User Name |alice\r\n
Full Name |Alice Doe\r\n
Description |\r\n
Transfer Packets|0\r\n
----------------+-----------------------\r\n
User Name |bob\r\n
Full Name |Bob Tables\r\n
Description |\r\n
Transfer Packets|0\r\n
"""
users = process_users_table(test_users_table)
print users

Your method of signaling the consumer to terminate is fine and is in harmony with what you would do if using a multiprocessing or threaded Queue. However, generators also have a way to throw Exceptions (rather than sending values) and the purpose of throw is precisely to signal events or changes in state to the generator. Moreover, when an exception is thrown to a generator,
[i]f the
generator catches the exception and yields another value, that is
the return value of g.throw().
That seems perfectly suited for your use case. Instead of sending a FINISH_PROCESSING_SIGNAL value, you could throw a FINISH_PROCESSING_SIGNAL Exception, and use try..except to yield the final value.
class FINISH_PROCESSING_SIGNAL(Exception): pass
def coroutine(func):
def start(*args,**kwargs):
cr = func(*args,**kwargs)
cr.next()
return cr
return start
# Sink
#coroutine
def list_builder():
# accepts objects and adds them to a list
_list = []
try:
while True:
data = (yield)
_list.append(data)
except FINISH_PROCESSING_SIGNAL:
yield _list
# Filter
#coroutine
def user_data_filter(target=list_builder()):
header = "-+-"
footer = "Transfer Packets"
username = "User Name"
fullname = "Full Name"
note = "Description"
try:
while True:
user = {}
data = (yield)
line = data
if header in line:
while True:
line = (yield)
if footer in line:
target.send(user)
break
elif username in line:
user["username"] = line.split('|')[1]
elif fullname in line:
user["fullname"] = line.split('|')[1]
elif note in line:
user["note"] = line.split('|')[1]
except FINISH_PROCESSING_SIGNAL as err:
# Pass along the Exception to the target, and yield its result back
# to the caller
yield target.throw(err)
# Producer
def process_users_table(table, target=user_data_filter()):
lines = table.split('\r\n')
for line in lines:
target.send(line)
processed_data = target.throw(FINISH_PROCESSING_SIGNAL)
# processed_data = target.close()
return processed_data
if __name__ == '__main__':
test_users_table = \
"""
Item |Value\r\n
----------------+-----------------------\r\n
User Name |alice\r\n
Full Name |Alice Doe\r\n
Description |\r\n
Transfer Packets|0\r\n
----------------+-----------------------\r\n
User Name |bob\r\n
Full Name |Bob Tables\r\n
Description |\r\n
Transfer Packets|0\r\n
"""
users = process_users_table(test_users_table)
print users

Python SSH via Pseudo TTY Clear

I wrote the following python module to handle ssh connections in my program:
#!/usr/bin/env python
from vxpty import VX_PTY
class SSHError(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return repr(self.msg)
class SSHShell:
def __init__(self, host, port, user, password):
self.host = host
self.port = port
self.user = user
self.password = password
self.authenticated = False
def authenticate(self):
self.tty = VX_PTY(['/usr/bin/ssh', 'ssh', '-p'+str(self.port), self.user+'#'+self.host])
resp = self.tty.read()
if "authenticity of host" in resp:
self.tty.println('yes')
while 1:
resp = self.tty.read()
if "added" in resp:
break
resp = self.tty.read()
if "assword:" in resp:
self.tty.println(self.password)
tmp_resp = self.tty.read()
tmp_resp += self.tty.read()
if "denied" in tmp_resp or "assword:" in tmp_resp:
raise(SSHError("Authentication failed"))
else:
self.authenticated = True
self.tty.println("PS1=''")
return self.authenticated
def execute(self, os_cmd):
self.tty.println(os_cmd)
resp_buf = self.tty.read().replace(os_cmd+'\r\n', '')
return resp_buf
Which uses a pty module I wrote earlier:
#!/usr/bin/env python
import os,pty
class PTYError(Exception):
def __init__(self, msg):
self.msg = msg
def __str__(self):
return repr(self.msg)
class VX_PTY:
def __init__(self, execlp_args):
self.execlp_args = execlp_args
self.pty_execlp(execlp_args)
def pty_execlp(self, execlp_args):
(self.pid, self.f) = pty.fork()
if self.pid==0:
os.execlp(*execlp_args)
elif self.pid<0:
raise(PTYError("Failed to fork pty"))
def read(self):
data = None
try:
data = os.read(self.f, 1024)
except Exception:
raise(PTYError("Read failed"))
return data
def write(self, data):
try:
os.write(self.f, data)
except Exception:
raise(PTYError("Write failed"))
def fsync(self):
os.fsync(self.f)
def seek_end(self):
os.lseek(self.f, os.SEEK_END, os.SEEK_CUR)
def println(self, ln):
self.write(ln+'\n')
However, whenever I call the execute() method, I end up reading the output from the first line:
>>> import SSH;shell=SSH.SSHShell('localhost',22,'735tesla','notmypassword');shell.authenticate()
True
>>> shell.execute('whoami')
"\x1b[?1034hLaptop:~ 735Tesla$ PS1=''\r\n"
>>>
Then the second time I call read() I get the output:
>>> shell.tty.read()
'whoami\r\n735Tesla\r\n'
>>>
Removing whoami\r\n from the output is not problem but is there any way to clear the output so I don't have to call read twice with the first command?

I think your problem is deeper than you realize. Luckily, it's also easier to solve than you realize.
What you seem to want is for os.read to return the entirety of what the shell has to send to you in one call. That's not something you can ask for. Depending on several factors, including, but not limited to, the shell's implementation, network bandwidth and latency, and the behavior of the PTYs (yours and the remote host's), the amount of data you'll get back in each call to read can be as much as, well, everything, and as little as a single character.
If you want to receive just the output of your command, you should bracket it with unique markers, and don't worry about messing with PS1. What I mean is that you need to make the shell output a unique string before your command executes and another one after your command executes. Your tty.read method should then return all the text it finds in between these two marker strings. The easiest way to make the shell output these unique strings is just to use the echo command.
For multiline commands, you have to wrap the command in a shell function, and echo the markers before and after executing the function.
A simple implementation is as follows:
def execute(self, cmd):
if '\n' in cmd:
self.pty.println(
'__cmd_func__(){\n%s\n' % cmd +
'}; echo __"cmd_start"__; __cmd_func__; echo __"cmd_end"__; unset -f __cmd_func__'
)
else:
self.pty.println('echo __"cmd_start"__; %s; echo __"cmd_end"__' % cmd)
resp = ''
while not '__cmd_start__\r\n' in resp:
resp += self.pty.read()
resp = resp[resp.find('__cmd_start__\r\n') + 15:] # 15 == len('__cmd_start__\r\n')
while not '_cmd_end__' in resp:
resp += self.pty.read()
return resp[:resp.find('__cmd_end__')]

Handle multiple IO errors

I have several IO operation that I carry out on class init but they often fail with IOError. What I would like to do is delay a few hundred ms and try again until success or some defined timeout. How can I make sure each individual command succeeds before continuing/ending the loop? I assume there is a better way than an if statement for each item and a counter to check if all commands succeeded.
My current code below often fails with IOError and hangs the rest of the application.
def __init__(self):
print("Pressure init.")
self.readCoefficients()
def readCoefficients(self):
global a0_MSB;
global a0_LSB;
global b1_MSB;
global b1_LSB;
global b2_MSB;
global b2_LSB;
global c12_MSB;
global c12_LSB;
a0_MSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_A0_COEFF_MSB+0);
a0_LSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_A0_COEFF_LSB+0);
b1_MSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_B1_COEFF_MSB+0);
b1_LSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_B1_COEFF_LSB+0);
b2_MSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_B2_COEFF_MSB+0);
b2_LSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_B2_COEFF_LSB+0);
c12_MSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_C12_COEFF_MSB+0);
c12_LSB = Pressure.bus.read_byte_data(Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_C12_COEFF_LSB+0);

Are you wanting to retry each one of those last 8 lines independently or as a group? If independently you will want to make a little helper function:
def retry_function(tries, function, *args, **kwargs):
for try in range(tries):
try:
return function(*args, **kwargs)
except IOError as e:
time.sleep(.005)
raise e # will be the last error from inside the loop. be sure tries is at least 1 or this will be undefined!
Then call it like this:
a0_MSB = retry_function(5, Pressure.bus.read_byte_data, Pressure.MPL115A2_ADDRESS,Pressure.MPL115A2_REGISTER_A0_COEFF_MSB+0)
If not independently but as a group, you probably still want this helper function. But you'll have to rewrite it to handle a list of functions/arguments, or pass in another custom function

If it's OK for you that all the files are read one after the other, you can use a simple function.
import time
# ...
def readCoefficients(self):
global a0_MSB;
global a0_LSB;
global b1_MSB;
global b1_LSB;
global b2_MSB;
global b2_LSB;
global c12_MSB;
global c12_LSB;
max_retries = 15
a0_MSB = self.readretry(Pressure.MPL115A2_REGISTER_A0_COEFF_MSB+0, max_retries)
a0_LSB = self.readretry(Pressure.MPL115A2_REGISTER_A0_COEFF_LSB+0, max_retries)
b1_MSB = self.readretry(Pressure.MPL115A2_REGISTER_B1_COEFF_MSB+0, max_retries)
b1_LSB = self.readretry(Pressure.MPL115A2_REGISTER_B1_COEFF_LSB+0, max_retries)
b2_MSB = self.readretry(Pressure.MPL115A2_REGISTER_B2_COEFF_MSB+0, max_retries)
b2_LSB = self.readretry(Pressure.MPL115A2_REGISTER_B2_COEFF_LSB+0, max_retries)
c12_MSB = self.readretry(Pressure.MPL115A2_REGISTER_C12_COEFF_MSB+0, max_retries)
c12_LSB = self.readretry(Pressure.MPL115A2_REGISTER_C12_COEFF_LSB+0, max_retries)
def readretry(self, address, max_retries):
for i in range(max_retries):
try:
return Pressure.bus.read_byte_data(
Pressure.MPL115A2_ADDRESS,
address
)
except IOError as e:
# print(e)
time.sleep(0.1)
else:
raise IOError("Reading failed after multiple tries")
Note: You should not use globals, most specially in classes.

This is another way of doing it. this code tries to read all addresses, and saves the one that failed. Then waits a little and retries all the addresses that failed until all addresses have been read properly or the number of allowed retries exceeded.
def readCoefficients(self):
(
a0_MSB, a0_LSB,
b1_MSB, b1_LSB,
b2_MSB, b2_LSB,
c12_MSB, c12_LSB) = self.mio_read(15,
Pressure.MPL115A2_REGISTER_A0_COEFF_MSB+0,
Pressure.MPL115A2_REGISTER_A0_COEFF_LSB+0,
Pressure.MPL115A2_REGISTER_B1_COEFF_MSB+0,
Pressure.MPL115A2_REGISTER_B1_COEFF_LSB+0,
Pressure.MPL115A2_REGISTER_B2_COEFF_MSB+0,
Pressure.MPL115A2_REGISTER_B2_COEFF_LSB+0,
Pressure.MPL115A2_REGISTER_C12_COEFF_MSB+0,
Pressure.MPL115A2_REGISTER_C12_COEFF_LSB+0
)
def mio_read(self, max_retries, *addresses):
# Create storage for results
results = [None] * len(addresses)
# Keep track of the index of a particular address in the list of results
ios = list(enumerate(addresses))
for i in range(max_retries):
failedios = []
for index, address in ios:
try:
results[index] = Pressure.bus.read_byte_data(
Pressure.MPL115A2_ADDRESS,
address
)
except IOError as e:
# Place address in the queue for the next round
failedios.append((index, address))
# If all succeeded
if len(failedios) == 0:
return results
# Time may be reduced as so was spent checking other addresses
time.sleep(0.1)
ios = failedios
else:
raise IOError(",".join((addr for ind, addr in failedios)))