We Keep Coding

python3 - broken pipe error when using socket.send()

I am programming a client-server instant message program. I created a similar program in Python 2, and am trying to program it in Python 3. The problem is when the server takes the message and tries to send it to the other client, it gives me "[Errno 32] Broken Pipe" and exits.
I have done some research, and found that this occurs when the client disconnects, so I did some more testing but could not find when the client disconnects. (I am using Ubuntu 14.04 and Python 3.4)
Here is the server code:
import socket, select, sys
def broadcast(sock, messaged):
for socket in connection_list:
if socket != s and socket != sock:
# Here is where it gives me the broken pipe error
try:
s.send(messaged.encode("utf-8"))
except BrokenPipeError as e:
print(e)
sys.exit()
connection_list = []
host = ''
port = 5558
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host,port))
s.listen(5)
connection_list.append(s)
read_sockets,write_sockets,error_sockets = select.select(connection_list,[],[])
while True:
for sock in read_sockets:
if sock == s:
conn, addr = s.accept()
connection_list.append(conn)
client = "Client (%s,%s) connected" % addr
print(client)
broadcast(sock,client)
else:
try:
data = sock.recv(2048)
decodeddata = data.decode("utf-8")
if data:
broadcast(sock, decodeddata)
except:
offline = "Client " + addr + "is offline"
broadcast(sock, offline)
print(offline)
connection_list.remove(sock)
sock.close()
continue
And the client code:
import socket, select, string, sys, time
def prompt(data) :
print("<You> " + data)
def Person(data) :
print("<Receiver> " + data)
if __name__ == "__main__":
host = "localhost"
port = 5558
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(2)
try:
s.connect((host,port))
except:
print('Unable to connect')
sys.exit()
print('Connected.')
socket_list = [s]
read_sockets,write_sockets,error_sockets = select.select(socket_list,[],[])
while 1:
for sock in read_sockets:
if sock == s:
try:
time.sleep(1)
data = sock.recv(1024)
Person(data.decode("utf-8"))
except:
msg = input("Send a message: ")
try:
s.send(str.encode(msg))
except:
print("Server is offline")
sys.exit()
else:
print("Server is offline")
sys.exit()

There are two problems that you have to fix to make this work.
First, on both the client side and the server side, you have to put the select inside the loop, not outside. Otherwise, if there was something to read before you got to the loop, you'll recv over and over, and if there wasn't, you'll never recv. Once you fix this, you can get rid of the time.sleep(1). (You should never need a sleep to solve a problem like this; at best it masks the problem, and usually introduces new ones.)
Meanwhile, on the server side, inside broadcast, you're doing s.send. But s is your listener socket, not a connected client socket. You want socket.send here, because socket is each socket in connection_list.
There are a number of unrelated problems in your code as well. For example:
I'm not sure what the except: in the client is supposed to be catching. What it mainly seems to catch is that, about 50% of the time, hitting ^C to end the program triggers the send prompt. But of course, like any bare except:, it also masks any other problems with your code.
There's no way to send any data back and forth other than the "connected" message except for that except: clause.
addr is a tuple of host and port, so when someone goes offline, the server raises a TypeError from trying to format the offline message.
addr is always the last client who connected, not the one who's disconnecting.
You're not setting your sockets to nonblocking mode.
You're not checking for EOF on the recv. This means that you don't actually detect that a client has gone offline until you get an error. Which normally happens only after you try to send them a message (e.g., because someone else has connected or disconnected).

Python Socket WebChat, I don't understand Select.select() and how it works

I don't want to make this post long so i get straight to the part i don't understand
this is the code i got from here
i aleady made my own app, but i made it the most awful way!
# Tcp Chat server
import socket, select
#Function to broadcast chat messages to all connected clients
def broadcast_data (sock, message):
#Do not send the message to master socket and the client who has send us the message
for socket in CONNECTION_LIST:
if socket != server_socket and socket != sock :
try :
socket.send(message)
except :
# broken socket connection may be, chat client pressed ctrl+c for example
socket.close()
CONNECTION_LIST.remove(socket)
if __name__ == "__main__":
# List to keep track of socket descriptors
CONNECTION_LIST = []
RECV_BUFFER = 4096 # Advisable to keep it as an exponent of 2
PORT = 5000
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# this has no effect, why ?
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind(('', PORT))
6.listen(10)
# Add server socket to the list of readable connections
CONNECTION_LIST.append(server_socket)
print "Chat server started on port " + str(PORT)
while 1:
# Get the list sockets which are ready to be read through select
read_sockets,write_sockets,error_sockets = select.select(CONNECTION_LIST,[],[])
for sock in read_sockets:
#New connection
if sock == server_socket:
# Handle the case in which there is a new connection recieved through server_socket
sockfd, addr = server_socket.accept()
CONNECTION_LIST.append(sockfd)
print "Client (%s, %s) connected" % addr
broadcast_data(sockfd, "[%s:%s] entered room\n" % addr)
#Some incoming message from a client
else:
# Data recieved from client, process it
try:
#In Windows, sometimes when a TCP program closes abruptly,
# a "Connection reset by peer" exception will be thrown
data = sock.recv(RECV_BUFFER)
if data:
broadcast_data(sock, "\r" + '<' + str(sock.getpeername()) + '> ' + data)
except:
broadcast_data(sock, "Client (%s, %s) is offline" % addr)
print "Client (%s, %s) is offline" % addr
sock.close()
CONNECTION_LIST.remove(sock)
continue
server_socket.close()
i don't understand the part which uses select and adds a new connections, i guess every time an app tries to connect(myport,myip) a new descriptor is created and this method tries to get connect()ed people? can you explain me?
and i have no idea why the server socket is appended to the socket and why its passed to select! I already read this but i still don't understand
i also don't understand why if sock == server_socket: means there is a new connection, i think its because i still haven't understood select

Select() (like the select() function in C) is somewhat magic. It enables you to wait for a events, without occupying the computer's CPU.
The events, in your case, are the input and output operations of the socket. Imagine you're waiting before closed doors ('you' are the CPU). The (correct) door will open if something useful can be done. Each door leads to the an I/O operation which is added to the CONNECTION_LIST.
While waiting, the CPU doesn't spend time on your program. Whichever door opens first, will wake the CPU, and lets you process the data.
Note: In Linux, select() works on all file operations (such as, in this case sockets). In Windows, this only works on sockets.
Note2: Though sockets work fine, I rather prefer a callback mechanism, such as iowatch from the gio library (part of glib). These will call functions when an operation is necessary.

Send/receive Packets with TCP sockets

Recently, I managed to create sockets on my PC and my Raspberry Pi to enable communication between both devices. Currently, the client is able to automatically send messages to the server. I was wondering, if it is possible to modify the scripts to send tcp data packets instead of purely text messages, as I would very much like to control the raspberry pi using my PC in the future without having the need to ssh/etc.
I've looked at some examples, but as I don't have much experience in writing my own scripts/codes, I'm not very sure how to go about doing this. I would appreciate if someone could guide me in the right direction with explanation and some examples if possible.
Anyway here is the server/client script I'm running at the moment:
Client:
import socket
import sys
import struct
import time
#main function
if __name__ == "__main__":
if(len(sys.argv) < 2) :
print 'Usage : python client.py hostname'
sys.exit()
host = sys.argv[1]
port = 8888
#create an INET, STREAMing socket
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print 'Socket Created'
try:
remote_ip = socket.gethostbyname( host )
s.connect((host, port))
except socket.gaierror:
print 'Hostname could not be resolved. Exiting'
sys.exit()
print 'Socket Connected to ' + host + ' on ip ' + remote_ip
#Send some data to remote server
message = "Test"
try :
#Set the whole string
while True:
s.send(message)
print 'Message sent successfully'
time.sleep(1)
print 'Sending...'
except socket.error:
#Send failed
print 'Send failed'
sys.exit()
def recv_timeout(the_socket,timeout=2):
#make socket non blocking
the_socket.setblocking(0)
#total data partwise in an array
total_data=[];
data='';
#beginning time
begin=time.time()
while 1:
#if you got some data, then break after timeout
if total_data and time.time()-begin > timeout:
break
#if you got no data at all, wait a little longer, twice the timeout
elif time.time()-begin > timeout*2:
break
#recv something
try:
data = the_socket.recv(8192)
if data:
total_data.append(data)
#change the beginning time for measurement
begin=time.time()
else:
#sleep for sometime to indicate a gap
time.sleep(0.1)
except:
pass
#join all parts to make final string
return ''.join(total_data)
#get reply and print
print recv_timeout(s)
s.close()
Server:
import socket
import sys
from thread import *
HOST = '' # Symbolic name meaning all available interfaces
PORT = 8888
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print 'Socket created'
try:
s.bind((HOST, PORT))
except socket.error , msg:
print 'Bind failed. Error Code : ' + str(msg[0]) + ' Message ' + msg[1]
sys.exit()
print 'Socket bind complete'
s.listen(10)
print 'Socket now listening'
#Function for handling connections
def clientthread(conn):
#Sending message to connected client
conn.send('Welcome to the server. Receving Data...\n') #send only takes string
#infinite loop so that function do not terminate and thread do not end.
while True:
#Receiving from client
data = conn.recv(1024)
reply = 'Message Received at the server!\n'
print data
if not data:
break
conn.sendall(reply)
conn.close()
#now keep talking with the client
while 1:
#wait to accept a connection
conn, addr = s.accept()
print 'Connected with ' + addr[0] + ':' + str(addr[1])
#start new thread
start_new_thread(clientthread ,(conn,))
s.close()

socket.socket(socket.AF_INET, socket.SOCK_STREAM) already creates a connection that provides a reliable stream of bytes between two machines. This uses TCP, which is on top of IP and Ethernet. The latter two are package-based, while TCP creates a stream of continuous bytes on top of it. It also adds some error checking and error correction, so it is pretty reliable.
I honestly don't understand what you want to achieve with what you call "send packets". What you don't want to do is to create an implementation of TCP yourself, as that's a non-trivial task, so sending RAW packets is out. In general, even using TCP is already relatively low-level and should be avoided unless really necessary.
Using e.g. ZeroMQ you get a message-based interface that does all the transmission for you. It does so on top of TCP (or other transports) and adds more error correction for e.g. disconnects. There, you also have something like "packets", but those are independent of how many TCP or IP packets were required to send it underneath. If you don't want to implement a specific protocol, I'd suggest you use this framework instead of lowlevel TCP sockets.
Another simple alternative is to use HTTP, for which there is also existing code in Python. The downside is that it is always one side that initiates some communication and the other side only replies. If you want some kind of active notification, you either have to poll or use hacks like delaying an answer.

You are already sending data packets - those packets juts happen to contain text data at the moment. Try looking into pickle in the standard libraries and into pyro.

How do I abort a socket.recv() from another thread in Python

I have a main thread that waits for connection. It spawns client threads that will echo the response from the client (telnet in this case). But say that I want to close down all sockets and all threads after some time, like after 1 connection.
How would I do it? If I do clientSocket.close() from the main thread, it won't stop doing the recv. It will only stop if I first send something through telnet, then it will fail doing further sends and recvs.
My code looks like this:
# Echo server program
import socket
from threading import Thread
import time
class ClientThread(Thread):
def __init__(self, clientSocket):
Thread.__init__(self)
self.clientSocket = clientSocket
def run(self):
while 1:
try:
# It will hang here, even if I do close on the socket
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
except:
break
self.clientSocket.close()
HOST = ''
PORT = 6000
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serverSocket.bind((HOST, PORT))
serverSocket.listen(1)
clientSocket, addr = serverSocket.accept()
print 'Got a new connection from: ', addr
clientThread = ClientThread(clientSocket)
clientThread.start()
time.sleep(1)
# This won't make the recv in the clientThread to stop immediately,
# nor will it generate an exception
clientSocket.close()

I know this is an old thread and that Samuel probably fixed his issue a long time ago. However, I had the same problem and came across this post while google'ing. Found a solution and think it is worthwhile to add.
You can use the shutdown method on the socket class. It can prevent further sends, receives or both.
socket.shutdown(socket.SHUT_WR)
The above prevents future sends, as an example.
See Python docs for more info.

I don't know if it's possible to do what you're asking, but it shouldn't be necessary. Just don't read from the socket if there is nothing to read; use select.select to check the socket for data.
change:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
to something more like this:
r, _, _ = select.select([self.clientSocket], [], [])
if r:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
EDIT: If you want to guard against the possibility that the socket has been closed, catch socket.error.
do_read = False
try:
r, _, _ = select.select([self.clientSocket], [], [])
do_read = bool(r)
except socket.error:
pass
if do_read:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)

I found a solution using timeouts. That will interrupt the recv (actually before the timeout has expired which is nice):
# Echo server program
import socket
from threading import Thread
import time
class ClientThread(Thread):
def __init__(self, clientSocke):
Thread.__init__(self)
self.clientSocket = clientSocket
def run(self):
while 1:
try:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
except socket.timeout:
# If it was a timeout, we want to continue with recv
continue
except:
break
self.clientSocket.close()
HOST = ''
PORT = 6000
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serverSocket.bind((HOST, PORT))
serverSocket.listen(1)
clientSocket, addr = serverSocket.accept()
clientSocket.settimeout(1)
print 'Got a new connection from: ', addr
clientThread = ClientThread(clientSocket)
clientThread.start()
# Close it down immediatly
clientSocket.close()

I must apologize for the comments below. The earlier comment by #Matt Anderson works. I had made a mistake when trying it out which led to my post below.
Using timeout is not a very good solution. It may seem that waking up for an instant and then going back to sleep is no big deal, but I have seen it greatly affect the performance of an application. You have an operation that for the most part wants to block until data is available and thus sleep forever. However, if you want to abort for some reason, like shutting down your application, then the trick is how to get out. For sockets, you can use select and listen on two sockets. Your primary one, and a special shutdown one. Creating the shutdown one though is a bit of a pain. You have to create it. You have to get the listening socket to accept it. You have to keep track of both ends of this pipe. I have the same issue with the Synchronized Queue class. There however, you can at least insert a dummy object into the queue to wake up the get(). This requires that the dummy object not look like your normal data though. I sometimes wish Python had something like the Windows API WaitForMultipleObjects.

How to make server accepting connections from multiple ports?

How can I make a simple server(simple as in accepting a connection and print to terminal whatever is received) accept connection from multiple ports or a port range?
Do I have to use multiple threads, one for each bind call. Or is there another solution?
The simple server can look something like this.
def server():
import sys, os, socket
port = 11116
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
try:
listening_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listening_socket.bind((host, port))
listening_socket.listen(backlog)
except socket.error, (value, message):
if listening_socket:
listening_socket.close()
print 'Could not open socket: ' + message
sys.exit(1)
while True:
accepted_socket, adress = listening_socket.accept()
data = accepted_socket.recv(buf_size)
if data:
accepted_socket.send('Hello, and goodbye.')
accepted_socket.close()
server()
EDIT:
This is an example of how it can be done. Thanks everyone.
import socket, select
def server():
import sys, os, socket
port_wan = 11111
port_mob = 11112
port_sat = 11113
sock_lst = []
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
try:
for item in port_wan, port_mob, port_sat:
sock_lst.append(socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sock_lst[-1].setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
sock_lst[-1].bind((host, item))
sock_lst[-1].listen(backlog)
except socket.error, (value, message):
if sock_lst[-1]:
sock_lst[-1].close()
sock_lst = sock_lst[:-1]
print 'Could not open socket: ' + message
sys.exit(1)
while True:
read, write, error = select.select(sock_lst,[],[])
for r in read:
for item in sock_lst:
if r == item:
accepted_socket, adress = item.accept()
print 'We have a connection with ', adress
data = accepted_socket.recv(buf_size)
if data:
print data
accepted_socket.send('Hello, and goodbye.')
accepted_socket.close()
server()

I'm not a python guy, but the function you are interested in is "select". This will allow you to watch multiple sockets and breaks out when activity occurs on any one of them.
Here's a python example that uses select.

Since Python's got so much overhead, multithreaded apps are a big point of debate. Then there's the whole blocking-operation-GIL issue too. Luckily, the Python motto of "If it seems like a big issue, someone's probably already come up with a solution (or several!)" holds true here. My favorite solution tends to be the microthread model, specifically gevent.
Gevent is an event-driven single-thread concurrency library that handles most issues for you out of the box via monkey-patching. gevent.monkey.patch_socket() is a function that replaces the normal socket calls with non-blocking variants, polling and sleeping to allow the switch to other greenlets as need be. If you want more control, or it's not cutting it for you, you can easily manage the switching with select and gevent's cooperative yield.
Here's a simple example.
import gevent
import socket
import gevent.monkey; gevent.monkey.patch_socket()
ALL_PORTS=[i for i in xrange(1024, 2048)]
MY_ADDRESS = "127.0.0.1"
def init_server_sock(port):
try:
s=socket.socket()
s.setblocking(0)
s.bind((MY_ADDRESS, port))
s.listen(5)
return s
except Exception, e:
print "Exception creating socket at port %i: %s" % (port, str(e))
return False
def interact(port, sock):
while 1:
try:
csock, addr = sock.accept()
except:
continue
data = ""
while not data:
try:
data=csock.recv(1024)
print data
except:
gevent.sleep(0) #this is the cooperative yield
csock.send("Port %i got your message!" % port)
csock.close()
gevent.sleep(0)
def main():
socks = {p:init_server_sock(p) for p in ALL_PORTS}
greenlets = []
for k,v in socks.items():
if not v:
socks.pop(k)
else:
greenlets.append(gevent.spawn(interact, k, v))
#now we've got our sockets, let's start accepting
gevent.joinall(greenlets)
That would be a super-simple, completely untested server serving plain text We got your message! on ports 1024-2048. Involving select is a little harder; you'd have to have a manager greenlet which calls select and then starts up the active ones; but that's not massively hard to implement.
Hope this helps! The nice part of the greenlet-based philosophy is that the select call is actually part of their hub module, as I recall, which will allow you to create a much more scalable and complex server more easily. It's pretty efficient too; there are a couple benchmarks floating around.

If you really wanted to be lazy (from a programmer standpoint, not an evaluation standpoint), you could set a timeout on your blocking read and just loop through all your sockets; if a timeout occurs, there wasn't any data available. Functionally, this is similar to what the select is doing, but it is taking that control away from the OS and putting it in your application.
Of course, this implies that as your sleep time gets smaller, your program will approach 100% CPU usage, so you wouldn't use it on a production app. It's fine for a toy though.
It would go something like this: (not tested)
def server():
import sys, os, socket
port = 11116
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
NUM_SOCKETS = 10
START_PORT = 2000
try:
socket.setdefaulttimeout(0.5) # raise a socket.timeout error after a half second
listening_sockets = []
for i in range(NUM_SOCKETS):
listening_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listening_socket.bind((host, START_PORT + i))
listening_socket.listen(backlog)
listening_sockets.append(listening_socket)
except socket.error, (value, message):
if listening_socket:
listening_socket.close()
print 'Could not open socket: ' + message
sys.exit(1)
while True:
for sock in listening_sockets:
try:
accepted_socket, adress = sock_socket.accept()
data = sock.recv(buf_size)
if data:
sock_socket.send('Hello, and goodbye.')
sock.close()
except socket.timeout:
pass
server()