We Keep Coding

How can I improve the structure / design of my API?

I was wondering if I could get some software design advice. Hypothetically, let's say that I'm writing an unofficial API for Reddit that lets you log into a Reddit account and manage it, manage messages, view account information, etc., with a wide array of features.
This is the typical pattern I use. All of the code is simplified and dumbed down.
class HTTPClient:
def __init__(self, session: requests.Session = None):
if not self.session:
self.session = requests.Session()
else:
self.session = session
def _request(self, method, url, **kwargs):
"""Internal request handler"""
resp = self.session.request(method, url, **kwargs)
if resp.ok:
return resp
raise Exception(resp, resp.text)
class RedditSession(HTTPClient):
"""Establishes a Reddit session and does the authentication stuff"""
def __init__(self, username:str, password: str, session: requests.Session = None):
super().__init__(session)
self.username = username
self.password = password
def _authentication_stuff_here(self):
self._request("get", "https://example.com")
pass
def _login(self):
pass
class RedditClient(RedditSession)
def __init__(self, username:str, password: str, session: requests.Session = None):
super().__init__(username, password, session)
self._login(self):
# Profile intance contains methods to manage profile info
self.profile = Profile(self.session)
# Manage account info (not profile info)
def get_account_information(self):
pass
def change_account_password(self, password: str):
pass
def change_account_email(self, email: str):
pass
I was wondering if anyone knew if this structure was considered bad design, and if anyone had any ideas on how I could improve it.
What particularly bothers me is the multiple layers of indirection; i.e., it feels repetitive to pass an optional session instance (or some other variable) through multiple parents. Also, things begin to feel even more messy when I pass the session to multiple other classes when using composition.
For example, what if I want to separate methods in a way that a user can type user.profile.message.delete()? Would it be normal practice to pass session to the Profile instance, which then passes it to the Message instance? Or is there a better way of achieving this without so many layers of abstraction?
I have tried various design patterns, but everything always feels "messy" in the end. Maybe it's due to my lack of OOP knowledge. Thank you in advance for any help/advice.

How do i handle streaming messages with Python gRPC

I'm following this Route_Guide sample.
The sample in question fires off and reads messages without replying to a specific message. The latter is what i'm trying to achieve.
Here's what i have so far:
import grpc
...
channel = grpc.insecure_channel(conn_str)
try:
grpc.channel_ready_future(channel).result(timeout=5)
except grpc.FutureTimeoutError:
sys.exit('Error connecting to server')
else:
stub = MyService_pb2_grpc.MyServiceStub(channel)
print('Connected to gRPC server.')
this_is_just_read_maybe(stub)
def this_is_just_read_maybe(stub):
responses = stub.MyEventStream(stream())
for response in responses:
print(f'Received message: {response}')
if response.something:
# okay, now what? how do i send a message here?
def stream():
yield my_start_stream_msg
# this is fine, i receive this server-side
# but i can't check for incoming messages here
I don't seem to have a read() or write() on the stub, everything seems to be implemented with iterators.
How do i send a message from this_is_just_read_maybe(stub)?
Is that even the right approach?
My Proto is a bidirectional stream:
service MyService {
rpc MyEventStream (stream StreamingMessage) returns (stream StreamingMessage) {}
}

What you're trying to do is perfectly possible and will probably involve writing your own request iterator object that can be given responses as they arrive rather than using a simple generator as your request iterator. Perhaps something like
class MySmarterRequestIterator(object):
def __init__(self):
self._lock = threading.Lock()
self._responses_so_far = []
def __iter__(self):
return self
def _next(self):
# some logic that depends upon what responses have been seen
# before returning the next request message
return <your message value>
def __next__(self): # Python 3
return self._next()
def next(self): # Python 2
return self._next()
def add_response(self, response):
with self._lock:
self._responses.append(response)
that you then use like
my_smarter_request_iterator = MySmarterRequestIterator()
responses = stub.MyEventStream(my_smarter_request_iterator)
for response in responses:
my_smarter_request_iterator.add_response(response)
. There will probably be locking and blocking in your _next implementation to handle the situation of gRPC Python asking your object for the next request that it wants to send and your responding (in effect) "wait, hold on, I don't know what request I want to send until after I've seen how the next response turned out".

Instead of writing a custom iterator, you can also use a blocking queue to implement send and receive like behaviour for client stub:
import queue
...
send_queue = queue.SimpleQueue() # or Queue if using Python before 3.7
my_event_stream = stub.MyEventStream(iter(send_queue.get, None))
# send
send_queue.push(StreamingMessage())
# receive
response = next(my_event_stream) # type: StreamingMessage
This makes use of the sentinel form of iter, which converts a regular function into an iterator that stops when it reaches a sentinel value (in this case None).

Python twisted irc: Wait for a whois reply inside privmsg method

I'm trying to make an IRC bot using the twisted.words.protocols.irc module.
The bot will parse messages from a channel and parse them for command strings.
Everything works fine except when I need the bot to identify a nick by sending a whois command. The whois reply will not be handled until the privmsg method (the method from which I'm doing the parsing) returns.
example:
from twisted.words.protocols import irc
class MyBot(irc.IRClient):
..........
def privmsg(self, user, channel, msg):
"""This method is called when the client recieves a message"""
if msg.startswith(':whois '):
nick = msg.split()[1]
self.whois(nick)
print(self.whoislist)
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""This method is called when the client recieves a reply for whois"""
self.whoislist[prefix] = params
Is there a way to somehow make the bot wait for a reply after self.whois(nick)?
Perhaps use a thread (I don't have any experience with those).

Deferred is a core concept in Twisted, you must be familiar with it to use Twisted.
Basically, your whois checking function should return a Deferred that will be fired when you receive whois-reply.

I managed to fix this by running all handler methods as threads, and then setting a field, following
kirelagin's suggestion, before running a whois query, and modifying the method that recieves the data
to change the field when it recieves a reply. Its not the most elegant solution but it works.
Modified code:
class MyBot(irc.IRClient):
..........
def privmsg(self, user, channel, msg):
"""This method is called when the client recieves a message"""
if msg.startswith(':whois '):
nick = msg.split()[1]
self.whois_status = 'REQUEST'
self.whois(nick)
while not self.whois_status == 'ACK':
sleep(1)
print(self.whoislist)
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""This method is called when the client recieves a reply for whois"""
self.whoislist[prefix] = params
def handleCommand(self, command, prefix, params):
"""Determine the function to call for the given command and call
it with the given arguments.
"""
method = getattr(self, "irc_%s" % command, None)
try:
# all handler methods are now threaded.
if method is not None:
thread.start_new_thread(method, (prefix, params))
else:
thread.start_new_thread(self.irc_unknown, (prefix, command, params))
except:
irc.log.deferr()
def irc_RPL_WHOISCHANNELS(self, prefix, params):
"""docstring for irc_RPL_WHOISCHANNELS"""
self.whoislist[prefix] = params
def irc_RPL_ENDOFWHOIS(self, prefix, params):
self.whois_status = 'ACK'

Question about getting asyncmongo query result in Tornado web

Hi
I know this is a common issue for user who is not familiar with async method....
i want to query db with user id using asyncmongo to check if the user log on, but obviously this doesn't work and i don't want to use self.render in call back.
Thanks for your help.
class MainPage(BaseHandler):
def get(self):
if not self.current_user:
#### get no result here
.............
get_current_user function using asyncmongo method:
def get_current_user(self):
user_id = self.get_secure_cookie("user")
if not user_id: return None
self.db.users.find({'user_id': bson.ObjectId(str(user_id))}, limit=1, callback=self._on_response)
def _on_response(self, response, error):
if error:
raise tornado.web.HTTPError(500)
how to return the value of response instead of self.render('template',response) ?

see Tornado Asynchronous Handler

def _on_response(self, response, error):
if error:
raise tornado.web.HTTPError(500)
self.render(str(response))

Python Twisted: "wait" for a variable to be filled by another event

I know that twisted will not "wait"... I am working with an XMPP client to exchange data with an external process. I send an request and need to fetch the corresponding answer. I use a sendMessage to send my request to the server. When the server answers a onMessage method will receive it and check if it an answer to a request (not necessarily the one I am looking for) and puts any answer in a stack.
As return to my sendRequest I want to return the results, so I would like to pop the response to my request from the stack and return.
I read about threads, defers, callbacks and conditionals, tried a lot of the examples and none is working for me. So my example code here is very stripped down pseudo-code to illustrate my problem. Any advice is appreciated.
class Foo(FooMessageProtocol):
def __init__(self, *args, **kwargs):
self.response_stack = dict()
super(Foo, self).__init__(*args, **kwargs)
def sendRequest(self, data):
self.sendMessage(id, data)
# I know that this doesn't work, just to illustrate what I would like to do:
while 1:
if self.response_stack.has_key(id):
break
return self.response_stack.pop(id)
def receiveAnswers(self, msg):
response = parse(msg)
self.response_stack[response['id']] = response

you can't return the results to sendRequest, because sendRequest can't wait.
make sendRequest return a Deferred instead, and fire it when the result arrives.
So the code calling sendRequest can just add a callback to the deferred and it will be called when there's a response.
Something like this (pseudo-code):
class Foo(FooMessageProtocol):
def __init__(self, *args, **kwargs):
self._deferreds = {}
super(Foo, self).__init__(*args, **kwargs)
def sendRequest(self, data):
self.sendMessage(id, data)
d = self._deferreds[id] = defer.Deferred()
return d
def receiveAnswers(self, msg):
response = parse(msg)
id = response['id']
if id in self._deferreds:
self._deferreds.pop(id).callback(response)