We Keep Coding

error 10054 in a multithreaded server program in python

I am trying to design a multithreaded server. client1 send data to server and server forwards data to client2 for processing. client2 sends processed data to server and then server forwards it to client1.
Data from client1 is received by client2 through server while execution, but my server program after sending data to client2 terminates with error
[10054] An existing connection was forcibly closed by remote host.
client1.py
from socket import *
import thread
ip='127.1.1.2'
port=5554
s=socket(AF_INET, SOCK_STREAM)
s.connect((ip,port))
data=raw_input("Input lowercase :: ")
while data!="#close":
s.send(data)
data=raw_input("Input next word or type #close to exit: ")
s.close()
client2.py
from socket import *
import thread
ip='127.1.1.2'
port=5554
s=socket(AF_INET, SOCK_STREAM)
s.connect((ip,port))
m=s.recv(1024)
k=m.upper()
print "Uppercase of ",m," in System--> ",k
s.close()
server.py
from socket import *
import thread
ip='127.1.1.2'
port=5554
data=''
def multi_threading(c,a):
while True:
global data
print "Inside client 1 thread"
data= c.recv(1024)
if not data:
break
if "close"==data.rstrip():
break
c.close()
print a,"--terminates connection"
def multi_threading2(c,a):
while True:
print "Inside client 2 thread"
c.send(data)
if not data:
break
if "close"==data.rstrip():
break
c.close()
print a,"--terminates connection"
if __name__=='__main__':
s=socket(AF_INET, SOCK_STREAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR,1)
s.bind((ip,port))
s.listen(4)
count=0
while True:
print "Server is listening on port number ",port
c,a=s.accept()
count=count+1
print "--------------------------------------------------"
print "Server connected to system having ip address --> ",a
if(count==1):
thread.start_new_thread(multi_threading,(c,a))
else:
thread.start_new_thread(multi_threading2,(c,a))
What i might be doing wrong?

I'm trying to help by changing your code as little as possible. It isn't how I would write it but should allow you to follow along. This is also not very idomatic Python. I just want you to see how to get the basic interaction going.
Some comments about my changes:
Remove the indent in client1.py so you don't always close the
socket with s.close(). I suspect this was a typo in your post?
In server.py it isn't safe to access the global data between two
threads. At the very least you need a semaphore to protect it.
Every time around multi_threading loop you destroy the last value of
data. You need to append the incoming bytes to a data structure
like a list. I renamed the threads to make it easier to understand.
You are expecting to see close in the server but you don't ever send this because your loop terminates when you type #close. You need to adjust for this.
Your code is order dependent because both client 'sender' and client 'receiver' use the same port. You need to put something into the protocol to distinguish between these two roles so you can start them in either order. This makes the system more robust.
Due to the sender not necessarily having any data I do a crude sleep. This can be avoided by using a shared queue between the threads which would block one end
You need some framing because you can't guarantee the speed at which the independent processes send and receive to the socket. I use \n for this.
There is an edge case if you sent very large strings (and removed the 1024 read) you could not get a full word with the \n through. I ignore this case.
I adjust the loopback address
I wrote this in PyCharm CE. I recommend using an IDE (PyCharm is excellent) with breakpoints. You can step through all the processes bit by bit. Put breakpoints in the threads and step through the actions.
Result:
server.py
from socket import *
import thread
from threading import Lock
import time
ip = '127.0.0.1'
port = 5554
data = []
data_lock = Lock()
def multi_threading_recv(sock, a):
should_run = True
while should_run:
global data
print "Inside client 1 thread"
_data = sock.recv(1024)
if _data:
with data_lock:
for word in _data.split('\n'):
data.append(word)
if "#close" == word:
should_run = False
break
else:
break
sock.close()
print a, "--terminates connection"
def multi_threading_send(sock, a):
while True:
print "Inside client 2 thread"
with data_lock:
if len(data) == 0:
time.sleep(1) # Not ideal. Ideally block on a shared queue
continue
else:
_data = data.pop(0)
if _data == "#close":
break
sock.send(_data + '\n')
c.close()
print a, "--terminates connection"
if __name__ == '__main__':
s = socket(AF_INET, SOCK_STREAM)
s.setsockopt(SOL_SOCKET, SO_REUSEADDR,1)
s.bind((ip,port))
s.listen(4)
count = 0
while True:
print "Server is listening on port number ", port
c, a = s.accept()
print "--------------------------------------------------"
print "Server connected to system having ip address --> ", a
role = c.recv(4)
if role == 'PUSH':
thread.start_new_thread(multi_threading_recv, (c, a))
elif role == 'PULL':
thread.start_new_thread(multi_threading_send, (c,a))
else:
print('invalid role: ' + role)
c.close()
client1.py
from socket import *
ip = '127.0.0.1'
port = 5554
s = socket(AF_INET, SOCK_STREAM)
s.connect((ip,port))
s.send('PUSH')
while True:
data = raw_input("Input word or type #close to exit: ")
s.send(data + '\n')
if data == "#close":
break
s.close()
client2.py
from socket import *
ip = '127.0.0.1'
port = 5554
s = socket(AF_INET, SOCK_STREAM)
s.connect((ip, port))
s.send('PULL')
while True:
data = s.recv(1024)
if data:
words = data.split('\n')
for w in words:
print(w)
else:
break
s.close()
Output:
client1
Input word or type #close to exit: hello
Input word or type #close to exit: world
Input word or type #close to exit: #close
client2:
hello
world
server:
Server is listening on port number 5554
--------------------------------------------------
Server connected to system having ip address --> ('127.0.0.1', 62605)
Server is listening on port number 5554
Inside client 1 thread
('127.0.0.1', 62605) --terminates connection
--------------------------------------------------
Server connected to system having ip address --> ('127.0.0.1', 62614)
Server is listening on port number 5554
Inside client 2 thread
Inside client 2 thread
Inside client 2 thread
('127.0.0.1', 62614) --terminates connection
There is one final comment that the server will go around the loop again and keep on accepting clients. The code doesn't cope with this and I leave it as an exercise for you. You can multiplex multiple senders and receivers if you add a channel number to the PUSH and PULL initiation of the protocol (e.g. PUSH3, PULL3). This will then allow you to store the incoming data in a dictionary keyed by this integer and then send it out to the correct client.

TCP Proxy Using Python

I am studying Black Hat Python and trying to understand the TCP proxy code.
I now almost understand it, but it doesn't quite work when I try to test it with
python proxy.py localhost 21 ftp.target.ca 21 True
in one terminal and
ftp ftp.target.ca 21
in another.
In the first terminal, I get only listen on localhost on port 21 and nothing else; and in the second terminal, it's happen connection between me and server and I wrote username and password.
The packages that transfer between me and server should appear in the first terminal.
What am I doing wrong?
Here is the code:
import sys
import socket
import threading
# this is a pretty hex dumping function directly taken from
# http://code.activestate.com/recipes/142812-hex-dumper/
def hexdump(src, length=16):
result = []
digits = 4 if isinstance(src, unicode) else 2
for i in xrange(0, len(src), length):
s = src[i:i+length]
hexa = b' '.join(["%0*X" % (digits, ord(x)) for x in s])
text = b''.join([x if 0x20 <= ord(x) < 0x7F else b'.' for x in s])
result.append( b"%04X %-*s %s" % (i, length*(digits + 1), hexa, text) )
print b'\n'.join(result)
def receive_from(connection):
buffer = ""
# We set a 2 second time out depending on your
# target this may need to be adjusted
connection.settimeout(2)
try:
# keep reading into the buffer until there's no more data
# or we time out
while True:
data = connection.recv(4096)
if not data:
break
buffer += data
except:
pass
return buffer
# modify any requests destined for the remote host
def request_handler(buffer):
# perform packet modifications
return buffer
# modify any responses destined for the local host
def response_handler(buffer):
# perform packet modifications
return buffer
def proxy_handler(client_socket, remote_host, remote_port, receive_first):
# connect to the remote host
remote_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
remote_socket.connect((remote_host,remote_port))
# receive data from the remote end if necessary
if receive_first:
remote_buffer = receive_from(remote_socket)
hexdump(remote_buffer)
# send it to our response handler
remote_buffer = response_handler(remote_buffer)
# if we have data to send to our local client send it
if len(remote_buffer):
print "[<==] Sending %d bytes to localhost." % len(remote_buffer)
client_socket.send(remote_buffer)
# now let's loop and reading from local, send to remote, send to local
# rinse wash repeat
while True:
# read from local host
local_buffer = receive_from(client_socket)
if len(local_buffer):
print "[==>] Received %d bytes from localhost." % len(local_buffer)
hexdump(local_buffer)
# send it to our request handler
local_buffer = request_handler(local_buffer)
# send off the data to the remote host
remote_socket.send(local_buffer)
print "[==>] Sent to remote."
# receive back the response
remote_buffer = receive_from(remote_socket)
if len(remote_buffer):
print "[<==] Received %d bytes from remote." % len(remote_buffer)
hexdump(remote_buffer)
# send to our response handler
remote_buffer = response_handler(remote_buffer)
# send the response to the local socket
client_socket.send(remote_buffer)
print "[<==] Sent to localhost."
# if no more data on either side close the connections
if not len(local_buffer) or not len(remote_buffer):
client_socket.close()
remote_socket.close()
print "[*] No more data. Closing connections."
break
def server_loop(local_host,local_port,remote_host,remote_port,receive_first):
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
server.bind((local_host,local_port))
except:
print "[!!] Failed to listen on %s:%d" % (local_host,local_port)
print "[!!] Check for other listening sockets or correct permissions."
sys.exit(0)
print "[*] Listening on %s:%d" % (local_host,local_port)
server.listen(5)
while True:
client_socket, addr = server.accept()
# print out the local connection information
print "[==>] Received incoming connection from %s:%d" % (addr[0],addr[1])
# start a thread to talk to the remote host
proxy_thread = threading.Thread(target=proxy_handler,args=(client_socket,remote_host,remote_port,receive_first))
proxy_thread.start()
def main():
# no fancy command line parsing here
if len(sys.argv[1:]) != 5:
print "Usage: ./proxy.py [localhost] [localport] [remotehost] [remoteport] [receive_first]"
print "Example: ./proxy.py 127.0.0.1 9000 10.12.132.1 9000 True"
sys.exit(0)
# setup local listening parameters
local_host = sys.argv[1]
local_port = int(sys.argv[2])
# setup remote target
remote_host = sys.argv[3]
remote_port = int(sys.argv[4])
# this tells our proxy to connect and receive data
# before sending to the remote host
receive_first = sys.argv[5]
if "True" in receive_first:
receive_first = True
else:
receive_first = False
# now spin up our listening socket
server_loop(local_host,local_port,remote_host,remote_port,receive_first)
main()

Simply running a proxy server on port 21 won't make the ftp client use it - so you shouldn't be trying to connect to the remote host. if the commandline is running the proxy on for example 9000, then ftp to that port i.e. localhost 9000 and the proxy will forward communication to/from that remote host.

Python client disconnect if server closes connection

I have server and client code in python in which client sends a request message to server and wait for the response. I have the server code to close the connection when the client doesn't send a proper request. When the server closes the request, the client still is listening forever for the response.
Below is the code
server.py
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.bind((host, port))
s.listen(1)
while True:
c, addr = s.accept()
# Receive request
data = c.recv(1024)
if data != 'something'
c.close()
else
c.sendall("message"+"End")
c.close()
s.close()
Client.py
End='End'
def recv_end(the_socket):
# Snippet from http://code.activestate.com/recipes/408859-socketrecv-three-ways-to-turn-it-into-recvall/
total_data=[];data=''
while True:
data=the_socket.recv(8192)
if End in data:
total_data.append(data[:data.find(End)])
break
total_data.append(data)
if len(total_data)>1:
#check if end_of_data was split
last_pair=total_data[-2]+total_data[-1]
if End in last_pair:
total_data[-2]=last_pair[:last_pair.find(End)]
total_data.pop()
break
return ''.join(total_data)
s = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
s.connect((host, port))
s.send("some_request")
data = s.recv_end(1024)
print "<---- " + str(data)
s.close()
I'm new to python and wondering if there is a way for the client to know that the server closed the connection and terminate.
I understand that the client would disconnect if I use normal s.recv(1024). But in my case, I need to send large data to the client so I have used a different function I found from http://code.activestate.com/recipes/408859-socketrecv-three-ways-to-turn-it-into-recvall/.
Is it like the server sends a empty string reply during the close of a connection and in my case, it would send a empty string without the end limiter and hence the client is listening forever ?

When you have a loop with recv or anything that reads from a socket or a pipe, you should stop reading as soon as you get a buffer with len 0 :
while True:
data=the_socket.recv(8192)
if len(data) == 0: break
...

select.select in python need 1 - 3 arguments sequence

I keep on receiving this error of 1 - 3 arguments needing to be sequences
import socket # Import socket module
import sys
import select
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM) # Create a socket object
host = "127.0.0.1" # Get local machine name
port = 50001 # Reserve a port for your service.
s.bind((host, port)) # Bind to the port
a = []
b = []
s.listen(1) # Now wait for client connection.
c, addr = s.accept() # Establish connection with client.
s.setblocking(0)
ready = select.select(s, s, s, 1) # i believe the error lies in here
while True:
print "reached"
if ready[0]:
print "reached1"
data = mysocket.recv(4096)
print 'Got connection from', addr
c.send('Thank you for connecting \r\n') #all strings have to end with /r/n!!!
print "sent"
c.close() # Close the connection
Error
Select.select arguments 1 - 3 has to be sequences.
I am new to python and hence i am unsure of what the error is. I googled the select code from another post as i wanted my recv socket to be non blocking

select.select takes three lists as arguments, rlist, wlist and xlist:
rlist: wait until ready for reading
wlist: wait until ready for writing
xlist: wait for an “exceptional condition” (see the manual page for what your system considers such a condition)
You're not passing lists but single sockets.
Try this:
ready = select.select([s], [s], [s], 1)
The return value will, again, be a tuple of three lists, the first containig sockets ready for reading, the second sockets ready for writing and the third sockets in 'exceptional condition'.
Note also that in your while loop you never update ready, so you will always use the same lists of sockets. Also, you should have a break somewhere, otherwise you'll end up calling c.send in an endless loop.

Python sockets - keep socket alive?

I'm having a little trouble with sockets in Python. Whenever someone connects it works fine but if they disconnect the server program closes. I want the server program to remain open after the client closes. I'm using a while True loop to keep the connection alive but once the client closes the connection the server closes it's connection.
Here is the client:
import socket, sys
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = sys.argv[1]
port = int(sys.argv[2])
conn.connect((host, port))
print("Connected to host " + sys.argv[1])
td = 1
while td == 1:
msg = raw_input('MSG: ')
Here is the server:
import socket, sys
socket.setdefaulttimeout(150)
host = ''
port = 50005
socksize = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
print("Server started on port: %s" % port)
s.listen(1)
print("Now listening...\n")
conn, addr = s.accept()
while True:
print 'New connection from %s:%d' % (addr[0], addr[1])
data = conn.recv(socksize)
if not data:
break
elif data == 'killsrv':
conn.close()
sys.exit()
else:
print(data)

If a client closes a connection, you want it to close the socket.
It seems like there's a bit of a disconnect here that I'll try to elaborate on. When you create a socket, bind, and listen, you've established an open door for others to come and make connections to you.
Once a client connects to you, and you use the accept() call to accept the connection and get a new socket (conn), which is returned for you to interact with the client. Your original listening socket is still there and active, and you can still use it to accept more new connections.
Looking at your code, you probably want to do something like this:
while True:
print("Now listening...\n")
conn, addr = s.accept()
print 'New connection from %s:%d' % (addr[0], addr[1])
data = conn.recv(socksize)
if not data:
break
elif data == 'killsrv':
conn.close()
sys.exit()
else:
print(data)
Please note that this is just a starting point, and as others have suggested you probably want to use select() along with forking off processes or spawning threads to service each client.

Your code is only accepting a single connection - the loop only deals with the first accepted connection and terminates as soon as it lost. This is way your server exists:
data = conn.recv(socksize)
if not data:
break
What you will need to do is to accept several connections, while handling each of those in it's own loop. Note that it does not have to be a real loop for each socket, you can use a select-based approach to query which of the sockets has an event associated with it (data available, connection lost etc.) and then process only those sockets, all in the same loop.
You can also use a multi threaded / multi process approach, dealing with each client in it's own thread or process - I guess you won't run into scaling issues when playing around.
See:
http://docs.python.org/library/select.html
http://docs.python.org/library/multiprocessing.html