We Keep Coding

How do I create an asynchronous socket in Python?

I've created a socket object for Telnet communication, and I'm using it to communicate with an API, sending and receiving data. I need to configure it in such a way that I can send and receive data at the same time. By that, I mean data should be sent as soon as the application tries to send it, and data should be processed immediately on receipt. Currently, I have a configuration which allows receipt to be instant, and sending to be second priority with a very short delay.
Currently the best way I have found to do this is by having an event queue, and pushing data to send into it, then having a response queue into which I put messages from the server. I have a thread which polls the buffer every .1 seconds to check for new data, if there isn't any, it then checks the request queue and processes anything there, and that's running in a continuous loop. I then have threads insert data into the request queue, and read data from the response queue. Everything is just about linear enough that this works fine.
This is not "asynchronous", in a sense that I've had to make it as asynchronous as possible without actually achieving it. Is there a proper way to do this? Or is anything under the hood going to be doing exactly the same as I am?
Other things I have investigated as a solution to this problem:
A callback system, where I might call socket.on_receipt(handle_message, args) to call the method handle_message with args as a parameter, passing the received data into the method. The only way I could find to achieve this is by implementing what I already have, then registering a callback for it (in fact, this is very close to what I do already have).
Please note: I am approaching this as a learning exercise to understand better how asynchronous systems work, not to understand how to use a particular library, so please do not suggest an existing library unless it contains very clear code which is simple to understand and answers the question fully and concisely.

This seems like a pretty straightforward use case for asyncio. I wouldn't consider using asyncio as "using a particular library" since socket programming paired with asyncio's event loop is pretty low-level and the concept is very transparent if you have experience with other languages and just want to see how async programming works in Python.
You can use this async chat as an example: https://gist.github.com/gregvish/7665915
Essentially, you create a non-blocking socket, see standard library reference on socket.setblocking(0):
https://docs.python.org/3/library/socket.html#socket.socket.setblocking
I'd also suggest this amazing session by David Beazley as a must-see for async Python programming. He explains the concurrency concepts in Python using sockets, exactly what you need: https://www.youtube.com/watch?v=MCs5OvhV9S4

Abort long running http operation

In my (python) code I have a thread listening for changes from a couchdb feed (continuous changes). The changes request has a timeout parameter which is too big in certain circumstances (for example when a user wants to interrupt the program manually with ^C).
How can I abort a long-running blocking http request?
Is this possible, or do I need to reduce the timeout to make my program more responsive?
This would be unfortunate, because having a timeout small enough to make the program really responsive (say, 1s), means that there are lots of connections being created (one per second!), which defeats the purpose of listening to changes, and makes it very difficult to make sure that we are not missing any changes (in the re-connecting timespan we can indeed miss changes, so that special code is needed to handle that case)
The other option is to forcefully abort the thread, but that is not really an option in python.

If I understand correctly it looks like you are waiting too long between requests before deciding whether to respond to the users or not. You are right continuously closing and creating new connections will defeat the purpose of changes feed.
A solution could be to use heartbeat query parameter in which couchdb will keep sending newlines to tell the client that the connection is still alive.
http://localhost:5984/hello/_changes?feed=continuous&heartbeat=1000&include_docs=true
as long as you are getting heartbeats (newlines) you can be sure that you are getting new changes. A new line will indicate that no changes have occurred. Where as an actual change will be reported back. No need to close the connection. Respond to your clients if resp!="/n"

Blocking the thread execution in general prevents the thread from beeing terminated. You need to wait until the request timed out. But this is already clear.
Using a library that supports non blocking requests is maybe a solution, but I don't know if there is any.
Anyway ... you've mentioned that reducing the timeout will lead to more connections. I'd suggest to implement a waiting loop between requests that can be interrupted by an external signal to terminate the thread. with this loop you can control the number of requests independent from the timeout.

Python: Interrupting sender with incoming messages

I'm working with socket, asynchronous event-driven programming. I would like to send a message, once I receive a response, send another message. But I may be doing something besides listening. That is, I want to get interrupted when socket.recv() actually receives a message.
Question 1: How can I let layer 3 interrupt layer 4? i.e. How can I handle the event of a non-null returning socket.recv() without actually dedicating "program time" to actually wait for a specific time to listen to incoming messages?

In asynchronous programming you don't interrupt an operation triggered by a message. All operations should be done in a short and fast fashion so you can process lots of messages per second. This way every operation is atomic and you don't suffer any race conditions so easily.
If you are in need to do more complex processing in parallel you could hand those problems over to a helper thread. Libraries like twisted are prepared for such use cases.

Do you need to use sockets directly? I would otherwise recommend looking into the excellent Twisted library for python. It handles all the async work for you, so you can focus on writing handlers and other logic your code needs. Twisted is relatively easy to get started with. Take a look at some examples at http://twistedmatrix.com/documents/current/core/howto/index.html.

how to unblock a blocked socket?

Synopsis:
My program occasionally runs into a condition where it wants to send data over a socket, but that socket is blocked waiting for a response to a previous command that never came. Is there any way to unblock the socket and pick back up with it when this happens? If not that, how could I test whether the socket is blocked so I could close it and open a new one? (I need blocking sockets in the first place)
Details:
I'm connecting to a server over two sockets. Socket 1 is for general command communication. Socket 2 is for aborting running commands. Aborts can come at any time and frequently. Every command sent over socket 1 gets a response, such as:
socket1 send: set command data
socket1 read: set command ack
There is always some time between the send and the read, as the server doesn't send anything back until the command is finished executing.
To interrupt commands in progress, I connect over a another socket and issue an abort command. I then use socket 1 to issue a new command.
I am finding that occasionally commands issued over socket 1 after an abort are hanging the program. It appears that socket 1 is blocked waiting for a response to a previously issued command that never returned (and that got interrupted). While usually it works sometimes it doesn't (I didn't write the server).
In these cases, is there any way for me to check to see if socket 1 is blocked waiting for a read, and if so, abandon that read and move on? Or even any way to check at all so I can close that socket and start again?
thx!
UPDATE 1: thanks for the answers. As for why I'm using blocking sockets, it's because I'm controlling a CNC-type machine with this code, and I need to know when the command I've asked it to execute is done executing. The server returns the ACK when it's done, so that seems like a good way to handle it. I like the idea of refactoring for non-blocking but can't envision a way to get info on when the command is done otherwise. I'll look at select and the other options.

Not meaning to seem disagreeable, but you say you need blocking sockets and then go on to describe some very good reasons for needing non-blocking sockets. I would recommend refactoring to use non-blocking.
Aside from that, the only method I'm aware of to know if a socket is blocked is the fact that your program called recv or one of its variants and has not yet returned. Someone else may know an API that I don't, but setting a "blocked" boolean before the recv call and clearing it afterward is probably the best hack to get you that info. But you don't want to do that. Trust me, the refactor will be worth it in the long run.

The traditional solution to this problem is to use select. Before writing, test whether the socket will support writing, and if not, do something else (such as waiting for a response first). One level above select, Python provides the asyncore module to enable such processing. Two level above, Twisted is an entire framework dealing with asynchronous processing of messages.

Sockets should be full duplex. If Python blocks a thread from writing to a socket while another thread is reading from the same socket I would regard it as a major bug in Python. This doesn't occur in any other programming language I've used.

What you really what is to block on a select() or poll(). The only way to unblock a blocked socket is to receive data or a signal which is probably not acceptable. A select() or poll() call can block waiting for one or more sockets, either on reading or writing (waiting for buffer space). They can also take a timeout if you want to wait periodically to check on other things. Take a look at my answer to Block Socket with Unix and C/C++ Help

How should a ZeroMQ worker safely "hang up"?

I started using ZeroMQ this week, and when using the Request-Response pattern I am not sure how to have a worker safely "hang up" and close his socket without possibly dropping a message and causing the customer who sent that message to never get a response. Imagine a worker written in Python who looks something like this:
import zmq
c = zmq.Context()
s = c.socket(zmq.REP)
s.connect('tcp://127.0.0.1:9999')
while i in range(8):
s.recv()
s.send('reply')
s.close()
I have been doing experiments and have found that a customer at 127.0.0.1:9999 of socket type zmq.REQ who makes a fair-queued request just might have the misfortune of having the fair-queuing algorithm choose the above worker right after the worker has done its last send() but before it runs the following close() method. In that case, it seems that the request is received and buffered by the ØMQ stack in the worker process, and that the request is then lost when close() throws out everything associated with the socket.
How can a worker detach "safely" — is there any way to signal "I don't want messages anymore", then (a) loop over any final messages that have arrived during transmission of the signal, (b) generate their replies, and then (c) execute close() with the guarantee that no messages are being thrown away?
Edit: I suppose the raw state that I would want to enter is a "half-closed" state, where no further requests could be received — and the sender would know that — but where the return path is still open so that I can check my incoming buffer for one last arrived message and respond to it if there is one sitting in the buffer.
Edit: In response to a good question, corrected the description to make the number of waiting messages plural, as there could be many connections waiting on replies.

You seem to think that you are trying to avoid a “simple” race condition such as in
... = zmq_recv(fd);
do_something();
zmq_send(fd, answer);
/* Let's hope a new request does not arrive just now, please close it quickly! */
zmq_close(fd);
but I think the problem is that fair queuing (round-robin) makes things even more difficult: you might already even have several queued requests on your worker. The sender will not wait for your worker to be free before sending a new request if it is its turn to receive one, so at the time you call zmq_send other requests might be waiting already.
In fact, it looks like you might have selected the wrong data direction. Instead of having a requests pool send requests to your workers (even when you would prefer not to receive new ones), you might want to have your workers fetch a new request from a requests queue, take care of it, then send the answer.
Of course, it means using XREP/XREQ, but I think it is worth it.
Edit: I wrote some code implementing the other direction to explain what I mean.

I think the problem is that your messaging architecture is wrong. Your workers should use a REQ socket to send a request for work and that way there is only ever one job queued at the worker. Then to acknowledge completion of the work, you could either use another REQ request that doubles as ack for the previous job and request for a new one, or you could have a second control socket.
Some people do this using PUB/SUB for the control so that each worker publishes acks and the master subscribes to them.
You have to remember that with ZeroMQ there are 0 message queues. None at all! Just messages buffered in either the sender or receiver depending on settings like High Water Mark, and type of socket. If you really do need message queues then you need to write a broker app to handle that, or simply switch to AMQP where all communication is through a 3rd party broker.

I've been thinking about this as well. You may want to implement a CLOSE message which notifies the customer that the worker is going away. You could then have the worker drain for a period of time before shutting down. Not ideal, of course, but might be workable.

There is a conflict of interest between sending requests as rapidly as possible to workers, and getting reliability in case a worked crashes or dies. There is an entire section of the ZeroMQ Guide that explains different answers to this question of reliability. Read that, it'll help a lot.
tl;dr workers can/will crash and clients need a resend functionality. The Guide provides reusable code for that, in many languages.

Wouldn't the simplest solution be to have the customer timeout when waiting for the reply and then retry if no reply is received?

Try sleeping before the call to close. This is fixed in 2.1 but not in 2.0 yet.