I am testing a Python web server. It works as expected using localhost as the server and client, but when I test on different computers, I am getting
[Errno 54] Connection reset by peer about 20% - 80% of the time, depending on how many client threads I spawn at once. Why?
Code Snippets
Server listens:
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((self.interface, self.port))
sock.listen(5)
Server loops forever, accepts client connection, spawns new thread:
while True:
(clientsock, (ip, port)) = self.sock.accept()
newthread = ClientThread(ip, port, clientsock)
newthread.start()
Spawn a bunch of client threads which connect with server, send message which requests a file, and then closes connection
Server sends message to client when ready
self.socket.sendall(message.encode())
After message is sent, close the write end of connection:
self.socket.shutdown(socket.SHUT_WR)
Client receives message (error occurs here)
def receive(self):
data_string = ''
bytes = self.sock.recv(self.bufsize)
while len(bytes) > 0:
bytes_str = bytes.decode('UTF-8')
data_string += bytes_str
bytes = self.sock.recv(self.bufsize)
return data_string
After client thread has received message, close the connection:
self.socket.close()
Receive function had errors. Changed to this:
def receive(self):
data_string = ''
while True:
bytes = self.sock.recv(self.bufsize)
bytes_str = bytes.decode('UTF-8')
data_string += bytes_str
if not bytes:
break
return data_string
Old receive function would try to call recv a second time when server had already closed socket. New one only calls once.
Also did not know you could increase listening socket backlog > 5 since Python docs say generally 5 is max, when on OS X it is 128. Increasing backlog to 128 helped.
Related
I am trying to send and receive data from a server. When I run the below python program from eclipse or python IDLE I get the response from the server. But, when I run the program from Raspberry Pi I am getting an error. Below is the simple code and error.
import socket,time
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ip = '127.0.0.1'
port = 1883
address = (ip,port)
client.connect(address)
print("connected")
def close():
global client
client.close()
client=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect(address)
while 1:
client.send(b'\x01\x04\x00\x00\xxxxxx')
print("sent")
try:
raw_data = client.recv(1024)
except:
close()
raw_data = client.recv(1024)
received_data = " ".join("{:02x}".format(byte) for byte in
(raw_data))
print(received_data)
time.sleep(1)
When I run this code in raspberry pi,I get BrokenPipe error. why it is happening in raspberry pi?
Error:
connected
sent
received
sent
Traceback (most recent call last):
File "exdu.py", line 18, in <module>
client.send(b'\x01\x04\x00\x00\xxxxxxxxx')
socket.error: [Errno 32] Broken pipe
A broken pipe is caused when a pipe (in this case a TCP socket) is closed unexpectedly by the other side and recv gets back an empty response or send tries to write to a closed connection.
There are quite a couple pieces of not so best practice code may be causing a broken pipe error in your code.
I'll try and point out a few things that stand out to me using comments:
import socket,time
# it would be nice to have a function wrapping this but as it is this is not the issue.
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ip = '127.0.0.1'
port = 1883
address = (ip,port)
client.connect(address)
print("connected")
def close():
global client
# If you read the python socket documentation https://docs.python.org/2/howto/sockets.html
# you will see that they advise sending a shutdown packet to the other side of the connection.
# In my experience this is actually extremely important when properly handling TCP connections and may be causing this issue.
# you must do this from the side that closes the connection, this includes when the server closes a connection.
client.shutdown(SHUT_RDWR) # this is what you want to do here.
client.close()
# connecting to a new socket here makes no sense and may also cause an issue.
client=socket.socket(socket.AF_INET,socket.SOCK_STREAM)
client.connect(address)
raw_data=client.recv(1024)
while 1:
client.send(b'\x01\x04\x00\x00\xxxxxx')
print("sent")
try:
raw_data = client.recv(1024)
except:
# this except statement may cause an exception itself.
# Not only that but you are probably getting your broken pipe error
# here from the recv in close().
close()
raw_data = client.recv(1024)
received_data = " ".join("{:02x}".format(byte) for byte in
(raw_data))
print(received_data)
time.sleep(1)
Apologies for the code comments, but i find they're useful when the segment is copy pasted when you are trying to experiment with a solution.
As an extension i would write the code you have to be much more like the python docs example:
class mysocket:
def __init__(self, sock=None):
if sock is None:
self.sock = socket.socket(
socket.AF_INET, socket.SOCK_STREAM)
else:
self.sock = sock
def connect(self, host, port):
self.sock.connect((host, port))
# important bit of missing code
def disconnect(self):
self.sock.shutdown(SHUT_RDWR)
self.sock.close()
def mysend(self, msg):
totalsent = 0
while totalsent < MSGLEN:
sent = self.sock.send(msg[totalsent:])
if sent == 0:
raise RuntimeError("socket connection broken")
totalsent = totalsent + sent
def myreceive(self):
chunks = []
bytes_recd = 0
while bytes_recd < MSGLEN:
chunk = self.sock.recv(min(MSGLEN - bytes_recd, 2048))
if chunk == '':
raise RuntimeError("socket connection broken")
chunks.append(chunk)
bytes_recd = bytes_recd + len(chunk)
return ''.join(chunks)
You can rewrite the send and recieve functions to do your bidding and then call it like:
socket = mysocket()
socket.connect('127.0.0.1', 1883)
try:
socket.mysend(b'\x01\x04\x00\x00\xxxxxx')
raw_data = socket.myrecieve()
# do more stuff
except:
socket.disconnect()
This is example code but may set you on the right path.
Broken pipe implies something isn't going through exactly as you want it to. Are you sure the server is running correctly on the raspberry pi on localhost?
Otherwise you might want to either run the server seperately(maybe heroku or digitalocean is relevant) or check your pi's firewall(I highly doubt this is the problem for localhost -> localhost connections)
I have the following case:
SERVER
sock = socket.socket()
sock.bind((hostaddr, port))
sock.listen(backlog)
print(f'Server listenning on {hostaddr}:{port}')
while True:
client_sock, client_address = self.sock.accept()
print(f'Incoming connection from {client_address[0]}:{client_address[1]}')
while True:
data = client_socket.recv(buffer_size)
if not data:
break
print(f'Received "{data.decode()}" from {client_address[0]}:{client_address[1]}')
reply = f'Server: I got the message "{data.decode()}"'.encode()
client_socket.sendall(reply)
client_socket.close()
CLIENT
sock = socket.socket()
sock.connect(server_address)
sock.sendall('Lorem Ipsum'.encode())
while True:
data = sock.recv(buffer_size)
if not data:
break
print(data.decode())
sock.close()
I first start the server, then I start the client, and I get the following logs:
SERVER
Server listening on 172.16.0.110:8081
Incoming connection from 172.16.0.110:62388
Received "Lorem Ipsum" from 172.16.0.110:62388
CLIENT
Server reply: I got the message "Lorem Ipsum"
I wanted to get the server reply and then the client should finish, but both server and client enter an infinite loop and keep running forever. Why and how to fix that? I'm using Python 3.6.0 on Windows 10 x64 in an IPv4 network.
You must define a protocol, which is just the rules for how messages are exchanged and formatted, and how message boundaries are communicated. It appears you simply want the client to send some data and read the server response. You can achieve this by closing the write half of the client connection, in your case by calling sock.shutdown(socket.SHUT_WR), after the sock.sendall(...).
On the server side this is the read half of the same connection, and the servers detects this as EOF, causing socket.recv() to return a zero-length bytes object.
For more complicated protocols for which you want to send multiple messages on the same connection, a different strategy must be used. One simple example for a binary protocol would be to send 4 bytes representing the length in bytes of the message, then send that many bytes for the subsequent message itself.
One way is to set a timeout for the socket so it doesn't block forever when waiting for reply using socket.settimeout() as the following:
sock = socket.socket()
sock.connect(server_address)
sock.sendall('Lorem Ipsum'.encode())
sock.settimeout(5.0) # sets timeout to 5 seconds
while True:
data = sock.recv(buffer_size)
if not data:
break
print(data.decode())
sock.close()
I'm learning socket programming in python,
Server Code:
import socket
srvsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
srvsock.bind(('', 23000))
srvsock.listen(5)
while True:
clisock, (rem_host, rem_port) = srvsock.accept()
print "conection established with host %s and port %s" % (rem_host, rem_port)
while True:
strg = clisock.recv(20)
if not strg:
print 'conection closed'
clisock.close()
break
clisock.send(strg)
Client Code:
import socket
clisock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clisock.connect(('', 23000))
clisock.send("Hello World rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr dsadsadsa tttttt\n")
while True:
data = clisock.recv(20)
print type(data)
if not data:
clisock.close()
break
print data
I'm sending data stream from client to server and at the same time receiving data from server, after successful data transmission, the server not closing client connection. Did I miss any thing ?
The issue is caused because the server keeps reading data from the client until it reads no data. This only happens when the connected client closes its connection. Until then, the server socket will block (i.e. temporarily suspend operations) until the client sends more data.
Bottom line: either the client or the server has to indicate that it no longer intends to send data over the connection.
You can fix the client by adding the line
clisock.shutdown(socket.SHUT_WR)
before the for loop in the client. This indicates that no more data will be sent.
server code:
while True:
clisock, (rem_host, rem_port) = srvsock.accept()
print "conection established with host %s and port %s" % (rem_host, rem_port)
while True:
strg = clisock.recv(20)
print '[message from client:] %s'%strg
clisock.send(strg)
print 'about to close with client'
clisock.close()
print 'connection closed with client'
break
client code :
clisock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clisock.connect(('', 23000))
clisock.send("Hello World rrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrrr dsadsadsa tttttt\n")
while True:
data = clisock.recv(20)
print '[Message from server :]%s'%data
print 'about to close server connection'
clisock.close()
print 'server connection closed'
break
This will work out in your case, holdenweb has proper a proper answer why your code is not behaving as expected in above code client only sends one message and closes the connection as well as server listens only for one message per client and closes connection to that client single client -- single connection ---- single message
I am trying to write a simple multithreaded web server in Python.
When a new client socket connects with the server socket, I create a new thread to handle the client socket, like so:
(clientsock, (ip, port)) = self.sock.accept()
newthread = ClientThread(ip, port, clientsock)
newthread.start()
where the ClientThread class has a run method, which in turn calls the following bit of code:
data_string = ''
bytes = self.socket.recv(2046)
bytes_str = bytes.decode('UTF-8')
data_string += bytes_str
where self.socket is the clientsock referenced above.
This code works as expected: 1 thread is created when the server socket accepts a connection from the client socket, and receives the client socket request.
But if I change the above code in order to loop while there is more data, such as:
data_string = ''
bytes = self.socket.recv(2046)
while len(bytes) > 0:
bytes_str = bytes.decode('UTF-8')
data_string += bytes_str
print 'data received: %s' % (bytes_str)
bytes = self.socket.recv(2046)
Then I have a strange issue. It seems that accept is being called multiple times, thereby creating a new, unnecessary thread each time.
I.E - if I start up a client and try to access a file, I get the following output from the server:
We have accepted a connection from (127.0.0.1, 59983)
New thread started for (127.0.0.1, 59983)
We have accepted a connection from (127.0.0.1, 59984)
New thread started for (127.0.0.1, 59984)
Why is accept being called multiple times if only one client has connected to the server?
Edit
Is this expected behavior?
I have server and client code in python in which client sends a request message to server and wait for the response. I have the server code to close the connection when the client doesn't send a proper request. When the server closes the request, the client still is listening forever for the response.
Below is the code
server.py
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind((host, port))
s.listen(1)
while True:
c, addr = s.accept()
# Receive request
data = c.recv(1024)
if data != 'something'
c.close()
else
c.sendall("message"+"End")
c.close()
s.close()
Client.py
End='End'
def recv_end(the_socket):
# Snippet from http://code.activestate.com/recipes/408859-socketrecv-three-ways-to-turn-it-into-recvall/
total_data=[];data=''
while True:
data=the_socket.recv(8192)
if End in data:
total_data.append(data[:data.find(End)])
break
total_data.append(data)
if len(total_data)>1:
#check if end_of_data was split
last_pair=total_data[-2]+total_data[-1]
if End in last_pair:
total_data[-2]=last_pair[:last_pair.find(End)]
total_data.pop()
break
return ''.join(total_data)
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((host, port))
s.send("some_request")
data = s.recv_end(1024)
print "<---- " + str(data)
s.close()
I'm new to python and wondering if there is a way for the client to know that the server closed the connection and terminate.
I understand that the client would disconnect if I use normal s.recv(1024). But in my case, I need to send large data to the client so I have used a different function I found from http://code.activestate.com/recipes/408859-socketrecv-three-ways-to-turn-it-into-recvall/.
Is it like the server sends a empty string reply during the close of a connection and in my case, it would send a empty string without the end limiter and hence the client is listening forever ?
When you have a loop with recv or anything that reads from a socket or a pipe, you should stop reading as soon as you get a buffer with len 0 :
while True:
data=the_socket.recv(8192)
if len(data) == 0: break
...