I just wanted to hear ideas on correct usage of EventListener/EventSubscription provider in twisted. Most of the examples and the twisted source handle events through the specific methods with a pretty hard coupling. Dispatching target methods of those events are "hardcoded" in a specific Protocol class and then it is a duty of inheriting class to override these to receive the "event". This is very nice and transparent to use while we know of all of potential subscribers when creating the Protocol.
However in larger projects there is a need (perhaps I am in a wrong mindset) for a more dynamic event subscription and subscription removal: think of hundereds of object with a lifespan of a minute all interested in the same event.
What would be correct way to acheive this according to the "way of twisted". I currently have created an event subscription / dispatching mechanism, however there is a lingering thought that the lack of this pattern in twisted library might suggest that there is a better way.
Twisted havs a package "twisted.words.xish.utility.EventDispatcher", pydoc it to know the usage, it is simple. However, I think what make Twisted strong is its "Deferred". You can looks Deferred object as a Closure of related events (something OK, something failed), callback, fallback are registed observer function. Deferred has advanced feature, such as can be nested.
So in my opinion, you can use default EventDispatcher in Twisted, or invent some simple new. But If you introduce some complicated mechanism into Twisted, it dooms to lead a confusion and mess.
Related
I have a ROUTER socket, which accepts requests, does some work and sends back a response:
import zmq
context = zmq.Context()
socket = context.socket(zmq.ROUTER)
socket.bind("tcp://*:1234")
sender, _, content = socket.recv_multipart()
reply = do_some_work(content) # Do something with the message
socket.send_multipart((sender, "", reply))
When sending the response, that empty string in the middle feels pretty stupid, although I know it is needed by ZeroMQ to separate the identity from the content.
I would expect a high-level language binding, such as pyzmq, to have a function that wraps that and provides a cleaner, less error-prone API.
I can imagine something like:
socket.sendto(identity, content)
And it will add the empty frame for me.
So - is there anything like that?
Well, AFAIK, no, there is not.
Reason 1)
IMHO, the identity / security / ... concepts seem to me to be a sort of ex-post add-ons, easily re-using the concept of a multi-frame composition for these add-ons onto an already mature API + countless amount of such API language-bindings, some of which might, some of which might not have any similar levels of syntax freedom, such as to similarly beautify the way to call the original ZeroMQ core exposed API services. One ought somehow respect the long successful running evolution of the ZeroMQ, their indoctrinated organisation side of generating consensus over and obeying in implementation the RFC-specifications and manage the end-to-end development process, which yet spans from API v 2.1 over many years and having survived many design and technology inducted shifts up to recent days, having recently core API v 4.2.2 stable in production.
Reason 2)
OOD/OOP Evangelists would make you soon revise your view to expect others to do this for you, once your imaginated high-level view on a request for having asocket.sendto( identity, content ) custom-specific method argumentation is based on your own view of comfort ( ref. above ) - as it is the very obvious case for using an appropriate tool -- a subclassing -- and there adding your own, specialised method, that extends the generic superclass behaviour and adds your wished to have comfort, all that under your control.
Anyway, enjoy the powers of ZeroMQ, as it can help you in many aspects of designing advanced scalable distributed heterogeneous non-blocking signalling / messaging systems. G/L on that wild and innovative ride!
Am just getting my head round Twisted, threading, stackless, etc. etc. and would appreciate some high level advice.
Suppose I have remote clients 1 and 2, connected via a websocket running in a page on their browsers. Here is the ideal goal:
for cl in (1,2):
guess[cl] = show(cl, choice("Pick a number:", range(1,11)))
checkpoint()
if guess[1] == guess[2]:
show((1,2), display("You picked the same number!"))
Ignoring the mechanics of show, choice and display, the point is that I want the show call to be asynchronous. Each client gets shown the choice. The code waits at checkpoint() for all the threads (or whatever) to rejoin.
I would be interested in hearing answers even if they involve hairy things like rewriting the source code. I'd also be interested in less hairy answers which involve compromising a bit on the syntax.
The most simple solution code wise is to use a framework like Autobahn which support remote procdure calls (RPC). That means you can call some JavaScript in the browser and wait for the result.
If you want to call two clients, you will have to use threads.
You can also do it manually. The approach works along these lines:
You need to pass a callback to show().
show() needs to register the callback with some kind of string ID in a global dict
show() must send this ID to the client
When the client sends the answer, it must include the ID.
The Python handler can then remove the callback from the global dict and invoke it with the answer
The callback needs to collect the results.
When it has enough results (two in your case), it must send status updates to the client.
You can simplify the code using yield but the theory behind is a bit complex to understand: What does the "yield" keyword do in Python? and coroutines
In Python, the most widely-used approach to async/event-based network programming that hides that model from the programmer is probably gevent.
Beware: this kind of trickery works by making tasks yield control implicitly, which encourages the same sorts of surprising bugs that tend to appear when OS threads are involved. Local reasoning about such problems is significantly harder than with explicit yielding, and the convenience of avoiding callbacks might not be worth the trouble introduced by the inherent pitfalls. Perhaps just as important to a library author like yourself: this approach is not pure Python, and would force dependencies and interpreter restrictions on the users of your library.
A lot of discussion about this topic sprouted up (especially between the gevent and twisted camps) while Guido was working on the asyncio library, which was called tulip at the time. He summarized the main issues here.
In the Python docs I see:
concurrent.futures.Future... ...should not be created directly
except for testing.
And I want to use it as a promise in my code and I'm very surprised that it is not recommended to use it like this.
My use case:
I have a single thread that reads data packets coming from socket, and I have many callbacks that are called depending on some information contained in packets. Packets are responses to consumers requests, and all consumers use single connection. Each consumer receives a promise and adds some handlers to it, that are invoked when response arrives.
So I cant use Executor subclass here, because I have only one thread, but I need to create many Futures (promises).
Promise is pretty widespread programming technique and, I thought that Future is Python's promise implementation. But if it is not recommended to use it like promise, what pythonistas are commonly use for this purpose?
Note
I use Python 2.7 backport of concurrent.futures to 2.7
It's perfectly fine to use Future in order to wrap non-promise APIs into promises.
The reason it generally should not be created is because most times people create futures directly it's because they're doing the deferred anti pattern and wrapping an executor created future in another future.
It's worth mentioning that this future implementation is very weak, it's akin to Java's old futures, the cool stuff promises give you like chaining is simply missing. It's worth mentioning that languages like JavaScript got their promises from Python's Twisted, which has a better implementation, even if it's intertwined with other things.
Maybe something like Django signals that doesn't depend on Django.
Django signals can be used to clear cache on saving a model, I'm trying to do the same.
eventlet
Twisted
Tornado
gevent (either forked or based on eventlet's design)
Of the four, eventlet is probably the quickest to pick up and easiest to use - you don't have to modify a lot of your code to make it event-based in the model of eventlet. It basically does some wrapping of the built-in libraries of python, and can do some runtime monkey patching of your code to make it event-based.
It looks like you want a library like PyDispatcher for signal registration and dispatching rather than an event-loop for networking.
Twisted is for event-driven networking.
I'd suggest PySide/PyQt with the signal/slot paradigm. It's quite a big dependency but it's well documented, steel hard tested, thread safe and easy to use.
Our projects at work include synchronous applications (short lived) and asynchronous Twisted applications (long lived). We're re-factoring our database and are going to build an API module to decouple all of the SQL in that module. I'd like to create that API so both synchronous and asynchronous applications can use it. For the synchronous applications I'd like calls to the database API to just return data (blocking) just like using MySQLdb, but for the asynchronous applications I'd like calls to the same API functions/methods to be non-blocking, probably returning a deferred. Anyone have any hints, suggestions or help they might offer me to do this?
Thanks in advance,
Doug
twisted.enterprise.adbapi seems the way to go -- do you think it fails to match your requirements, and if so, can you please explain why?
Within Twisted, you basically want a wrapper around a function which returns a Deferred (such as the Twisted DB layer), waits for it's results, and returns them. However, you can't busy-wait, since that's using up your reactor cycles, and checking for a task to complete using the Twisted non-blocking wait is probably inefficient.
Will inlineCallbacks or deferredGenerator solve your problem? They require a modern Twisted. See the twistedmatrix docs.
def thingummy():
thing = yield makeSomeRequestResultingInDeferred()
print thing #the result! hoorj!
thingummy = inlineCallbacks(thingummy)
Another option would be to have two methods which execute the same SQL template, one which uses runInteraction, which blocks, and one which uses runQuery, which returns a Deferred, but that would involve more code paths which do the same thing.
Have you considered borrowing a page from continuation-passing style? Stackless Python supports continuations directly, if you're using it, and the approach appears to have gained some interest already.
All the database libraries I've seen seem to be stubbornly synchronous.
It appears that Twisted.enterprise.abapi solves this problem by using a threads to manage a connection pool and wrapping the underlying database libraries. This is obviously not ideal, but I suppose it would work, but I haven't actually tried it myself.
Ideally there would be some way to have sqlalchemy and twisted integrated. I found this project, nadbapi, which claims to do it, but it looks like it hasn't been updated since 2007.