We Keep Coding

Python - are sockets and recv() blocking by default

I see a lot of examples online regarding socket recv() that follow this format:
Example from here.
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost', 50000))
s.listen(1)
conn, addr = s.accept()
while 1:
data = conn.recv(1024)
if not data:
break
The code here does not set the socket as non-blocking; nor does it set a timeout. My understanding is that recv() is blocking by default. If so, how does the code get to the break since, if there is no data then it stays on the recv()?
EDIT: what is the proper way to handle 'no more data'? Is it to set the socket to have a timeout or be non-blocking; or is there another way?

According to this, recv could return an empty string (therefore falsey) if the peer shuts down or disconnects. In other words, keep receiving data using the blocking recv function until the peer disconnects, then leave the loop.
In response to your edit: generally, if no more data is to be sent, the machine that is no longer sending data would send one last message to tell the server there's no more data to receive, or otherwise just end the connection and you would get your empty string.

non-blocking i/o using select

I have a sample client-server program that does non-blocking I/O for several sockets not using processes or threads. It uses select. Unfortunately, the server just shows lots of blank lines and that's all. Where is the mistake?
Running on MacOS.
Thanks in advance.
Server:
import socket
import select
sock = socket.socket()
sock.bind(('', 10001))
sock.listen()
conn1, _ = sock.accept()
conn2, _ = sock.accept()
conn1.setblocking(0)
conn2.setblocking(0)
epoll = select.poll()
epoll.register(conn1.fileno(), select.POLLIN | select.POLLOUT)
epoll.register(conn2.fileno(), select.POLLIN | select.POLLOUT)
conn_map = {
conn1.fileno(): conn1,
conn2.fileno(): conn2,
}
while True:
events = epoll.poll(1)
for fileno, event in events:
if event & select.POLLIN:
data = conn_map[fileno].recv(1024)
print(data.decode('utf8'))
elif event & select.POLLOUT:
conn_map[fileno].send('ping'.encode('utf8'))
Client:
import socket
from multiprocessing import Pool
def create_socket_and_send_data(number):
with socket.create_connection(('127.0.0.1', 10001)) as sock:
try:
sock.sendall(f'client {number}\n'.encode('utf8'))
except socket.error as ex:
print('data sending error', ex)
print(f'data for {number} has been sent')
if __name__ == '__main__':
with Pool(processes=2) as pool:
pool.map(create_socket_and_send_data, range(2))

Unfortunately, the server just shows lots of blank lines and that's all.
Actually this is not true.
The server prints at the beginning the lines it got from the clients. After they've send these lines the client close the connection which means that select.POLLIN gets triggered again on the socket and recv returns empty data.
This empty data is the sign that the peer has closed the connection. Once it got this sign the server should close the connection to the client and remove the fileno from the select. Instead your server prints the empty string with a newline and continues to expect new POLLIN events. These will come again and again and will always an empty buffer, thus leading to all the empty lines you see.

select is paradoxically easier to use for input than for output. For input, you receive an event each time new data arrives on a socket, so you always ask for all the sockets and have something to process for every new event.
For output, select will just say that a socket if ready to accept new data. Which is almost always true except if you have just filled a buffer. So you should only poll for an output socket when you have something to write there.
So you should register your sockets with select.POLLIN only. For the write part, you should either directly write to a socket without polling if you can hope that the peer should always be able to receive, or set up a queue with pending output per socket, modify the polling state of a socket with select.POLLIN | select.POLLOUT when there is something in its queue and modify it back with select.POLLIN back when the queue is empty again.

Python socket recv data in while loop not stopping

While im trying to recv data with a while loop the loop not stopping even when there is no data
import socket
class Connect:
connect = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
def __init__(self, server_ip, server_port):
self.connect.connect((server_ip, server_port))
def recv(self):
data_ls = []
while True:
data = self.connect.recv(2048)
if not data: # after getting the first data
break # Python wont come to this "if" so it wont break!
data = data.decode('utf-8')
data_ls.append(data)
return data_ls

Because socket.recv is a blocking call. This means that your program will be paused until it receives data.
You can set a time limit on how long to wait for data:
socket.settimeout(seconds_to_wait_for_data)
Or, you can make the socket not block:
sock.setblocking(False)
Note that under your current implementation, your code will probably busy wait for data to be available, potentially using more system resources than necessary. You can prevent this by:
looking for a signal for when there isn't any more data from the server at the start (such as a Content-Length header for HTTP) while setting a timeout (in case of network issues)
using a library implementing a higher level protocol

Python sockets: Server waits for nothing when asked to 'recv' and then 'sendall'

I am experimenting with python sockets to try to understand the whole concept better, but I have run into a problem. I have a simple server and a client, where the client sends a list to the server, and then waits for the server to send a string signaling the process is complete.
This is the client file:
import socket
import json
host = '192.168.1.102'
port = 14314
def request():
print 'Connecting'
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect((host, port))
print 'Sending request'
clientsocket.sendall(json.dumps([1, 2, 3, 4, 5, 6, 7, 8, 9]))
print 'Receiving data'
data = clientsocket.recv(512)
print 'Received: {}'.format(data)
request()
and here is the server file:
import socket
import json
host = '192.168.1.102'
port = 14314
def run():
print 'Binding socket'
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serversocket.bind((host, port))
print 'Waiting for client'
serversocket.listen(1)
clientsocket, addr = serversocket.accept()
print 'Receiving data'
raw_data = ''
while True:
tmp = clientsocket.recv(1024)
if not tmp:
break
raw_data += tmp
data = json.loads(raw_data)
print 'Received: {}'.format(data)
print 'Sending data'
clientsocket.sendall('done')
run()
The problem is that while the client is done sending the list, the server is stuck in the recv loop, waiting for nothing. The whole data has been received in the first iteration and in the second iteration there is nothing to be received because the client has moved on to the receiving part.
The weird part is that if I comment out the receive part from the client and the send part from the server, the process completes successfully. So, what am I doing wrong? Why is this not working?
Thanks.

The Docs for socket.recv talk about additional flags being able to be passed in to the recv function described in the unix documentation. So turning to that documentation, I found the following message:
If no messages are available at the socket, the receive calls wait for
a message to arrive, unless the socket is nonblocking (see fcntl(2)),
in which case the value -1 is returned
So once again, we're directed to another page. The documentation for fcntl says
Performs one of the operations described below on the open file
descriptor
So, normally the socket.recv function is blocking (it will wait indefinitely for new data), unless we use a file descriptor. How do we do that? Well there is a socket.makefile function that gives us a file descriptor attached to the socket. Cool. This SO question gives us an example of how we can read and write to a socket, using a file descriptor.
Well what if we don't want to use a file descriptor. Reading further into the unix documentation for the recv function, I see that I can use the MSG_DONTWAIT flag. This doesn't work in Windows, but I did find out that we can use socket.setbocking(False) to permamently change the socket to non-blocking mode. You would then need to ignore any "A non-blocking socket operation could not be completed immediately" errors. Those are normal and non-fatal(error #10035 of this page mentions it is non-fatal).
Another possible implementation would be to multi-thread your program, you can implement a receiving and a sending thread for your socket. This might give you the best performance, but it would be a lot of work to setup.
Python is awesome. I just found some libraries Python has that does asynchronous sockets too. There's asyncore, asynchat which have both been deprecated in favor of asyncio if that is available in the version of Python you are using.
Sorry for throwing so much out there. I don't know a whole lot about sockets. I used them once with the Paramiko library, and that was it. But it looks like there are a lot of ways of implementing them.

I need the server to send messages to all clients (Python, sockets)

This is my server program, how can it send the data received from each client to every other client?
import socket
import os
from threading import Thread
import thread
def listener(client, address):
print "Accepted connection from: ", address
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
client.send(data)
client.close()
host = socket.gethostname()
port = 10016
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host,port))
s.listen(3)
th = []
while True:
print "Server is listening for connections..."
client, address = s.accept()
th.append(Thread(target=listener, args = (client,address)).start())
s.close()

If you need to send a message to all clients, you need to keep a collection of all clients in some way. For example:
clients = set()
clients_lock = threading.Lock()
def listener(client, address):
print "Accepted connection from: ", address
with clients_lock:
clients.add(client)
try:
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
with clients_lock:
for c in clients:
c.sendall(data)
finally:
with clients_lock:
clients.remove(client)
client.close()
It would probably be clearer to factor parts of this out into separate functions, like a broadcast function that did all the sends.
Anyway, this is the simplest way to do it, but it has problems:
If one client has a slow connection, everyone else could bog down writing to it. And while they're blocking on their turn to write, they're not reading anything, so you could overflow the buffers and start disconnecting everyone.
If one client has an error, the client whose thread is writing to that client could get the exception, meaning you'll end up disconnecting the wrong user.
So, a better solution is to give each client a queue, and a writer thread servicing that queue, alongside the reader thread. (You can then extend this in all kinds of ways—put limits on the queue so that people stop trying to talk to someone who's too far behind, etc.)
As Anzel points out, there's a different way to design servers besides using a thread (or two) per client: using a reactor that multiplexes all of the clients' events.
Python 3.x has some great libraries for this built in, but 2.7 only has the clunky and out-of-date asyncore/asynchat and the low-level select.
As Anzel says, Python SocketServer: sending to multiple clients has an answer using asyncore, which is worth reading. But I wouldn't actually use that. If you want to write a reactor-based server in Python 2.x, I'd either use a better third-party framework like Twisted, or find or write a very simple one that sits directly on select.