First, of, I've read around a fair amount of time including many threads on this site, however I still need some clarification on Sockets, TCP and Networking in Python, as I feel like I don't fully understand what's happening in my program.
I'm sending data from a server to a client via an Unix Domain Socket (AF_UNIX) using TCP (SOCK_STREAM).
On the server side, a process is continuously putting items on a Queue.Queue and another process is sending items to the client by running
while True:
conn.sendall(queue.get())
On the client side, data is read by running
while True:
conn.recv(1024)
# time.sleep(10)
Now, I emulate a slow client by sending the client process to sleep after every call on recv(). What I expect is that the queue on the server side is filled up, since send() should block because the client can't read off data fast enough.
I monitor the number of items send to the client as well as the queue size. What I notice is that several dozen messages (roughly depending on the size of the messages, but slightly different message sizes might behave the same) are sent to the client (which are received by the client with delay, due to time.seep()) before the queue starts to fill up.
What is happening here? Why is send() not blocking immediately?
I suspect that some sort of network or file buffer is involved, which queues the send items and fills up before my implemented queue.
There are a number of buffers in various places in the system, on both the sender and the receiver. Your call to a sending function won't block until all those buffers are filled up. When the receiver drains some of the buffers, data will flow again and eventually it will unblock the send call.
Typically there's a buffer in the sender holding data waiting to be put on the wire, a buffer "in flight" allowing a certain number of bytes to be send before having to wait for the receiver to acknowledge, and lastly receive buffers holding data that has been acknowledged but not yet delivered to the receiving application.
Were this not so, forward progress would be extremely limited. The sender would be stuck waiting to send until the receiver called receive. Then, whichever one finishes first would have to wait for the other one. Even if the sender was finished first, it couldn't make any forward progress at all until the receiver finished processing the previous chunk of data. That would be quite sub-optimal for most applications.
Related
I am building a socket server with Python.
This server
receives data from client
does something here (approximately it takes about 10 sec in maximum depending on input data)
sends back data after some works done above
This system works fine unless client doesn't send data simultaneously in a row. For example, say server takes 5 seconds to process data and client sends data every 10 seconds. The problem, however, is that client send multiple requests at a time, thus causing a delay. Currently, client cannot send data to server unless server is ready to receive data which means that server is not doing any work. Below are what I want to build.
a) build a queue at socket server whose main task is to make a queue of input data so that client can send data to server even when server is busy
b) make a thread(here, I'm bit confused with concurrency and parallelism. Work in socket focused on computation rather than system call) at socket server so that server can do work 'simultaneously'.
c) send back data to client socket
My questions are as follows.
Is it Queue that I need to use in order to achieve a) ?
Is it thread or something else that I need to use in order to achieve b)?
Thanks in advance
Best
Gee
Yeah something like this could work.
First, you'll need a thread to receive and send data. If you have limited amount of clients, you can create a thread per client, but it's not an option for a more or less robust system. In order to be able to serve multiple clients in a single thread, the sockets should be nonblocking. Otherwise one long transmission would block other transmissions. Nonblocking code has more sophisticated structure that uses select, so I would advice to spend some time reading about it.
Then you'll need a thread to do the math. Or several threads/processes if "the math" is taking long to execute.
Last but not least, these socket threads and a "math" thread should use two queues to exchange data. Simple lists's are enough, but make sure they are synchronized. Guard them with mutexes, or locks. This is another vast topic that is worth reading about.
I have a producer sending data using PULL / PUSH to multiple workers. All the workers need to receive all their data before performing a computation task.
I tried a sync using a PUB / SUB socket sending a "go" but as the PUSH socket are non-blocking, the go is received before the end of the datastream...
Sender :
context = zmq.Context()
push_socket = self.context.socket(zmq.PUSH)
push_socket.bind("tcp://127.0.0.1:5557")
pull_socket = self.context.socket(zmq.PULL)
pull_socket.bind("tcp://127.0.0.1:5558")
for index, data in range(100):
push_socket.send_json({"data": data, "id": index})
pub_socket.send_json({"command": "map"})
Receiver :
# recieve work
consumer_receiver = context.socket(zmq.PULL)
consumer_receiver.connect("tcp://127.0.0.1:5557")
# receive commands
consumer_command = context.socket(zmq.SUB)
consumer_command.subscribe("")
consumer_command.connect("tcp://127.0.0.1:5559")
poller = zmq.Poller()
poller.register(consumer_receiver, zmq.POLLIN)
poller.register(consumer_command, zmq.POLLIN)
while True:
events = dict(poller.poll(100))
if consumer_command in events:
received = consumer_command.recv_json()
command = received["command"]
print("received command : ", command)
if consumer_receiver in events:
received = consumer_receiver.recv_json()
print("received data", received)
Receiver output :
received data {'data': ['Hi'], 'id': 0}
received command : map
received data {'data': ['hi'], 'id': 1}
...
I would like to have:
received data {'data': ['Hi'], 'id': 0}
received data {'data': ['hi'], 'id': 1}
...
received command : map
I tried to set a HWM of 1 for the PUSH socket but it didn't work.
How can I send a synchronization message to all workers after the PUSH is finished ?
You are seeking to implement a barrier.
ZeroMQ is all about Actor model programming, and one characteristic is that there is no explicit rendevous implied in sending and receiving messages. That is, a send will return regardless of whether or not the other end has read the message.
So this means that a barrier (a type of rendevous) has to be synthesised on top of ZeroMQ's Actor model.
Use a PUSH / PULL socket pair to get the data to the workers.
Use a separate PUSH / PULL socket pair for the workers to send back a "I have the data and am ready to proceed" message to the producer.
Have the producer wait for these "I can proceed" messages,
When it has received one from every worker, send a "go" message on the PUB / SUB socket to the workers.
Communicating Sequential Processes
Simply out of interest you may wish to compare Actor model programming with Communicating Sequential Processes (which in Rust, Erlang, and (I think?) Go is making something of a comeback). In CSP sending / receiving message is a rendevous. This has several benefits;
the sender knows that a message has been received and not just queued,
it forces one to properly address architecture and resource allocation if one has performance and latency goals. You can't hide messages in transit. So if one had not supplied enough workers the producer would very obviously not be able to offload messages; the deficiency cannot be temporarily hidden by increased latency.
if you have managed to construct an architecture that can deadlock, livelock, etc it always will. Whereas an Actor model architecture can appear to be perfectly fine for years until that one day when the network gets a little busier.
To do what you want with CSP, you'd be able to omit steps 2 and 3 above. The Producer would know that every worker had received its data when the send to the last worker returned, and the "go" can be sent out immediately.
Personally speaking I really wish ZeroMQ would have the option to be CSP, not Actor. Then it would be fabulous, instead of being just pretty tremendous. What makes it really good is that it doesn't matter whether it's tcp, ipc, inproc, etc. it all behaves the same (with speed variations obviously).
AFAIK Rust, Erlang and Go CSP channels go no further than the process. ZMQ can be inter and/or intra process and/or inter computer, which makes it highly suitable for developing systems that may outgrow one computer. Need to offload a thread to another computer? Change the connection string, no other code changes required. Very nice.
You are using separate streams for command and data - this will always guarantee synchronization problems. On recipient side, you will have two stream buffers - first with a lot of data to handle, second with only command and poll() will make sure you are notified that both are ready to be read.
I see two ways to handle this problem:
1) Keep it simple: use only one stream. Everything you send on the end will be received on the end. TCP guarantees that. If you're using json, you can just add to it 'type': 'command' or 'type': 'data' to discriminate between message types.
2) If, for some reason, you really need two streams (e.g. you really want to play with publisher/subscriber pattern), receiver should acknowledge to the sender reception of the last data batch, before sender can send its command. This option would also be the choice if all workers need to receive their data before any of them is started with the command.
My problem is kind of trying to half-close a zmq socket.
In simple terms I have a pair of PUSH/PULL sockets in Python.
The PUSH socket never stops sending, but the PULL socket should be able to clean itself up in a following way:
Stop accepting any additional messages to the queue
Process the messages still in the queue
Close the socket etc.
I don't want to affect the PUSH socket in any way, it can keep accumulating its own queue until another PULL socket comes around or that might be there already. The LINGER option doesn't seem to work with recv() (just with send()).
One option might be to have a broker in between with the broker PUSH and receiver PULL HWM set to zero. Then the broker's PULL would accumulate the messages. However, I'd rather not do this. Is there any other way?
I believe you are confusing which socket type will queue messages. According to the zmq_socket docs, a PUSH socket will queue its messages but a PULL socket doesn't have any type of queuing mechanism.
So what you're asking to be able to do would be something of the following:
1) Stop recv'ing any additional messages to the PULL socket.
2) Close the socket etc.
The PUSH socket will continue to 'queue' its messages automatically until either the HWM is met (at which it will then block and not queue any more messages) or a PULL socket comes along and starts recv'ing messages.
The case I think you're really concerned about is a slow PULL reader. In which you would like to get all of the currently queued messages in the PUSH socket (at once?) and then quit. This isn't how zmq works, you get one message at a time.
To implement something of this sort, you'll have to wrap the PULL capability with your own queue. You 'continually' PULL the messages into your personal queue (in a different thread?) until you want to stop, then process those messages and quit.
Consider the following scenario: A process on the server is used to handle data from a network connection. Twisted makes this very easy with spawnProcess and you can easily connect the ProcessTransport with your protocol on the network side.
However, I was unable to determine how Twisted handles a situation where the data from the network is available faster than the process performs reads on its standard input. As far as I can see, Twisted code mostly uses an internal buffer (self._buffer or similar) to store unconsumed data. Doesn't this mean that concurrent requests from a fast connection (eg. over local gigabit LAN) could fill up main memory and induce heavy swapping, making the situation even worse? How can this be prevented?
Ideally, the internal buffer would have an upper bound. As I understand it, the OS's networking code would automatically stall the connection/start dropping packets if the OS's buffers are full, which would slow down the client. (Yes I know, DoS on the network level is still possible, but this is a different problem). This is also the approach I would take if implementing it myself: just don't read from the socket if the internal buffer is full.
Restricting the maximum request size is also not an option in my case, as the service should be able to process files of arbitrary size.
The solution has two parts.
One part is called producers. Producers are objects that data comes out of. A TCP transport is a producer. Producers have a couple useful methods: pauseProducing and resumeProducing. pauseProducing causes the transport to stop reading data from the network. resumeProducing causes it to start reading again. This gives you a way to avoid building up an unbounded amount of data in memory that you haven't processed yet. When you start to fall behind, just pause the transport. When you catch up, resume it.
The other part is called consumers. Consumers are objects that data goes in to. A TCP transport is also a consumer. More importantly for your case, though, a child process transport is also a consumer. Consumers have a few methods, one in particular is useful to you: registerProducer. This tells the consumer which producer data is coming to it from. The consumer can them call pauseProducing and resumeProducing according to its ability to process the data. When a transport (TCP or process) cannot send data as fast as a producer is asking it to send data, it will pause the producer. When it catches up, it will resume it again.
You can read more about producers and consumers in the Twisted documentation.
I have a "manager" process on a node, and several worker processes. The manager is the actual server who holds all of the connections to the clients. The manager accepts all incoming packets and puts them into a queue, and then the worker processes pull the packets out of the queue, process them, and generate a result. They send the result back to the manager (by putting them into another queue which is read by the manager), but here is where I get stuck: how do I send the result to a specific socket? When dealing with the processing of the packets on a single process, it's easy, because when you receive a packet you can reply to it by just grabbing the "transport" object in-context. But how would I do this with the method I'm using?
It sounds like you might need to keep a reference to the transport (or protocol) along with the bytes the just came in on that protocol in your 'event' object. That way responses that came in on a connection go out on the same connection.
If things don't need to be processed serially perhaps you should think about setting up functors that can handle the data in parallel to remove the need for queueing. Just keep in mind that you will need to protect critical sections of your code.
Edit:
Judging from your other question about evaluating your server design it would seem that processing in parallel may not be possible for your situation, so my first suggestion stands.