We Keep Coding

multiclient server that receives a file from few sources

I'm building a torrent like program for a project. I want to get a file from few sources and then merge it together. the code below is my server and client for the data transfer (note: in the main program these codes are threads). variables such as address, port, directories, filename, numofclients etc are just placeholders.
The problem I'm having is that half of the time, when a client connects, the server doesn't register him (meaning it doesn't give the client info it needs to continue the sending process). when this happens the client side claims its connected but I dont know if this issue is serverbased or clientbased. If anyone can help me find the issue that would be great, I've been trying to fix it for days.
Another side issue is that when the client does send the data, it send it significantly slower than when using a normal basic send loop with one client. is the select bottlenecking my speed?
note: I'm using python 2.7. prints are for monitoring.
server:
import random
import select
import socket
portnum=3500
filename="Testvid.avi"
numofclients=2
datalist=[]
for i in range(0,numofclients):
datalist.append(open("C:/Users/Nitai/Desktop/Metorrent/"+filename+"-tmp"+str(i+1),'wb'))
server_socket = socket.socket()
server_socket.bind(('0.0.0.0', portnum))
server_socket.listen(5)
s, address = server_socket.accept()
open_client_sockets = []
print "receivefile initiated"
def send_waiting_messages (wlist):
for message in messages_to_send:
(client_socket, data) = message
if client_socket in wlist:
client_socket.send(data)
messages_to_send.remove(message)
messages_to_send = []
dataindex=0
socketindex=[]
finishedcount=0
while finishedcount<numofclients:
rlist, wlist, xlist = select.select( [server_socket] + open_client_sockets, open_client_sockets, [] )
for current_socket in rlist:
if current_socket is server_socket:
print "new client"
(new_socket,address)=server_socket.accept()
open_client_sockets.append(new_socket)
socketindex.append(new_socket)
print open_client_sockets
datatosend="IDP "+str(dataindex)+"%"+str(numofclients)
print datatosend
messages_to_send.append((new_socket,datatosend))
print "data sent"
print dataindex
dataindex+=1
else:
data=current_socket.recv(1024)
if data.find("EndPacket")!=-1:
print "connection finished"
finishedcount+=1
open_client_sockets.remove(current_socket)
else:
datalist[socketindex.index(current_socket)].write(data)
send_waiting_messages(wlist)
s.close()
print "select exited"
filewriter=open("C:/Users/Nitai/Desktop/Metorrent/"+filename+"-final",'wb')
for i in range(0,numofclients):
filewriter.write(datalist[i].read())
print "File received"
filewriter.close()
print "transfer finished"
client:
import random
import socket
import os
portnum=3500
filename="Testvid.avi"
address='10.0.0.5'
s= socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((address, portnum))
print "connected"
data=s.recv(1024)
print "data received"
print data
index=int(data[4:data.find("%")])
print index
numofclients=int(data[data.find("%")+1:len(data)])
print numofclients
filetosend=open("C:/Users/Nitai/Desktop/"+filename,'rb')
filelength=int(os.stat("C:/Users/Nitai/Desktop/"+filename).st_size)
startpoint=(filelength/numofclients)*index
if numofclients==index+1:
print "last part sender"
endpoint=filelength
else:
endpoint=(filelength/numofclients)*(index+1)
filetosend.seek(startpoint)
print startpoint
print endpoint
while startpoint+1024<endpoint:
a=filetosend.read(1024)
s.send(a)
startpoint+=1024
l=filetosend.read(endpoint-filetosend.tell())
s.send(l)
filetosend.close()
time.sleep(3)
endpacketdata="EndPacket"
s.send(endpacketdata)
print "File sent"
s.close()
print "data transfer complete"
thanks for the help!

After the server_socket.listen(5), you have s, address = server_socket.accept() which is out of place, since the accept is to be done in the while loop.

multithreaded server in python

I have a server-client application written in python. Everything worked fine but I wanted to make the server multi threaded and everything crashed.
Here is part of the code for the server:
host = 'localhost'
port = 10001
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
class ClientThread(threading.Thread):
def __init__(self, ip, port, socket):
print '5'
threading.Thread.__init__(self)
self.ip = ip
self.port = port
self.socket = socket
print "[+] New thread started for "+ip+":"+str(port)
def __run__(self):
while True:
try:
#conn, addr = sock.accept()
print >>sys.stderr, "Connection from : "+ip+":"+str(port)
print '6'
#reqCommand = conn.recv(1024)
reqCommand = self.recv(1024)
print '7'
command = reqCommand.split(' ', 1) # get <<filename>>
print '8'
reqFile = command[1] # takes the name of the file
reqCommand = command[0]
print '9'
encFile = reqFile + "_enc"
print >>sys.stderr, 'Client> %s' % (reqCommand)
if (reqCommand == 'get'):
pass
try:
os.remove(encFile) # removes the encrypted file
except OSError, e:
print ("Error: %s - %s." % (e.filename,e.strerror))
print >>sys.stderr, 'successfully finished'
print >>sys.stderr, 'waiting for new connections...'
finally:
# clean up connection
self.close()
while True:
sock.listen(4)
print "\nListening for incoming connections..."
(conn, (ip, port)) = sock.accept()
print '1'
newthread = ClientThread(ip, port, conn)
print '2'
newthread.start()
print '3'
threads.append(newthread)
print '4'
When I type in the client: "get " it sends the message to the client but it doesn't receive anything back. In the server as you can see I have a lot of prints to see where it crashes. It prints in the following order: 1 5 2 3 4. + it also prints [+] new thread...
As you can also see I've used self.recv instead of conn.recv (this was a solution that I found on stackoverflow, but it didn't work)
Does anyone have a clue what I'm doing wrong? I mention again that before I've added threads and the class ClientThread everything worked fine. Thanks in advance!

You have a lot of errors in the code shown.
E.g. self.recv(1024) should probably be replaced with self.socket.recv(1024), and self.close() with self.socket.close()? (since self is an instance of ClientThread/Thread, not a socket).
I also think run method should be named just run (not __run__), and if you do a close() in the finally in run() the second time while True is executed connction will be already closed.
In addition to that, large chunks are missing, e.g. all the imports, and a call to bind() - e.g. something like sock.bind((socket.gethostname(), port))
Other than that and assuming all these errors are fixed, it seems that it should do what it is supposed to.

How to run a "try/finally"-like method when closing a cmd window running Python script?

I am creating a Python Server/Client chat program (like AOL instant messenger) using sockets.
I run into a problem when I force quit the server or the client because my socketInstance.recv() method is sent a huge amount of "" blank strings. I am wondering how to run a closing method, like a deconstructor, when the cmd window is force quit so I can exit gracefully.
I've included my code, just in case (shouldn't be necessary tho):
Echo server program
import socket
import sys
import time
HOST = '' # Symbolic name meaning all available interfaces
PORT = 50007 # Arbitrary non-privileged port
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#s = socket.socket() # Create a socket object
HOST = socket.gethostname() # Get local machine name
print "Socket hostname: ", HOST
s.bind((HOST, PORT))
conn = None
def check_for_messages():
print 'Check for messages...'
global conn
while 1:
try:
data = conn.recv(1024)
if not data: break
conn.sendall(data)
print data
except socket.error:
#[Errno 10054] Client Closes
pass
print "END OF CHECK MESS"
def check_for_client():
print 'Listening...'
global conn
s.listen(1)
conn, addr = s.accept()
print 'Connected by', addr
check_for_client()
check_for_messages()
Echo client program
import threading
import socket
import time
PORT = 50007 # The same port as used by the server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
HOST = socket.gethostname() # The local host
s.connect((HOST, PORT))
getUserInputThread = None
getMessThread = None
receivedMessages = []
def getMessage():
print "getMessageThread running"
while 1:
data = s.recv(1024)
if not data:
print "OVERLORD: Possible server disconnect"
break
receivedMessages.append(data)
print "getMessageThread ending\n"
def getUserInput():
print "getUserInputThread running"
while 1:
message = raw_input("type: ")
s.sendall(message)
#print messages in list:
while len(receivedMessages) > 0:
print "Received Messages Length: %d" % len(receivedMessages)
print receivedMessages.pop(0)
print "getUserInputThread ending"
getUserInputThread = threading.Thread(target=getUserInput)
getUserInputThread.start()
getMessThread = threading.Thread(target=getMessage)
getMessThread.start()
Thanks,
Jordan

You could use the atexit module to perform final cleanup before terminating:
import atexit
#atexit.register
def do_cleanup():
print "doing some cleanup..."
#close sockets, put chairs on tables, etc
print "goodbye"
#loop forever, until the user hits ctrl-c
while True:
pass
Result:
C:\Users\kevin\Desktop>test.py
Traceback (most recent call last):
File "C:\Users\kevin\Desktop\test.py", line 12, in <module>
while True:
KeyboardInterrupt
doing some cleanup...
goodbye
This will work if your user force quits with a KeyboardInterrupt, but not if he closes the command window with the X button, terminates the task with task manager, unplugs the computer, etc.

Python Socket Listening

All of the below mentioned is on windows machines using python 2.7
Hello,
I am currently attempting to listen on a socket for data send by a remote program. This data is then printed to the screen and user input is requested that is then returned to remote program. In testing I have been able to have the remote program send me a menu of command line programs (cmd, ipconfig, whoami, ftp) and then my program returns with a number as a selection of the menu option.
The remote program receives my response and sends the output of the selected command. ipconfig and whoami work perfectly, but cmd and ftp only returns the output of the terminal once. (I.E. I can enter one command into the FTP program and send that too the remote program before I never hear back)
The part of my code that fails is that
if ready[0]: never becomes ready a second time after the first conversation.
I know the remote program is functioning correctly as I can use netcat to act in lieu of my code and operate the cmd terminal indefinitely.
How do I go about properly implementing a python socket listener that can account for this type of connection?
My "program" in its entirety:
import socket, sys, struct, time, select
host = ''
port = 50000
connectionSevered=0
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port '+str(port)+'.'
s.bind((host,port))
s.listen(5)
def recvall(the_socket,timeout=2):
global connectionSevered
data=''; # Data found by recv
total_data=[]; # Finally list of everything
s.setblocking(0) #make socket non blocking
begin=time.time() #beginning time
while 1:
ready = select.select([client], [], [], .2)
if time.time()-begin > timeout:
print 'Timeout reached'
#Leave loop, timer has reached its threshold
break
if ready[0]:
print 'In ready loop!'
try:
data = client.recv(4096) #attempt to fetch data
if data:
begin=time.time() #reset timeout timer
total_data.append(data)
data='';
except socket.error:
print '[+] Lost connection to client. Printing buffer...'
connectionSevered=1 # Let main loop know connection has errored
pass
time.sleep(1)
#join all parts to make final string
return ''.join(total_data)
client, address = s.accept()
print '[+] Client connected!'
while (connectionSevered==0): # While connection hasn't errored
print "connectionSevered="+str(connectionSevered) # DEBUG
recvall(s)
response = raw_input() #take user input
client.sendto(response) #send input
client.close(0)
Please let me know if you need more information, any help would be greatly appreciated, I am very new to this and eager to learn.

Playing around with this for a while finally got it working nice with a telnet session locally using python 2.7.
What it does is it sets up a thread that runs when the client connects listening for client stuff.
When the client sends a return ("\r\n" might have to change that if your interacting with a Linux system?) the message gets printed to the server, while this is happening if there is a raw input at the server side this will get sent to the client:
import socket
import threading
host = ''
port = 50000
connectionSevered=0
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port: {0}'.format(port)
conn, address = s.accept()
c = client(conn)
c.start()
print '[+] Client connected: {0}'.format(address[0])
c.send_msg(u"\r\n")
print "connectionSevered:{0}".format(connectionSevered)
while (connectionSevered==0):
try:
response = raw_input()
c.send_msg(response + u"\r\n")
except:
c.close()
The above answer will not work for more than a single connection. I have updated it by adding another thread for taking connections. It it now possible to have more than a single user connect.
import socket
import threading
import sys
host = ''
port = 50000
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
class connectionThread(threading.Thread):
def __init__(self, host, port):
super(connectionThread, self).__init__()
try:
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.bind((host,port))
self.s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
self.clients = []
def run(self):
while True:
conn, address = self.s.accept()
c = client(conn)
c.start()
c.send_msg(u"\r\n")
self.clients.append(c)
print '[+] Client connected: {0}'.format(address[0])
def main():
get_conns = connectionThread(host, port)
get_conns.start()
while True:
try:
response = raw_input()
for c in get_conns.clients:
c.send_msg(response + u"\r\n")
except KeyboardInterrupt:
sys.exit()
if __name__ == '__main__':
main()
Clients are not able to see what other clients say, messages from the server will be sent to all clients. I will leave that as an exercise for the reader.

If you're in Python 3 by now and still wondering about sockets, here's a basic way of using them:
server.py
import time
import socket
# creating a socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
# get local Host machine name
host = socket.gethostname() # or just use (host == '')
port = 9999
# bind to pot
s.bind((host, port))
# Que up to 5 requests
s.listen(5)
while True:
# establish connection
clientSocket, addr = s.accept()
print("got a connection from %s" % str(addr))
currentTime = time.ctime(time.time()) + "\r\n"
clientSocket.send(currentTime.encode('ascii'))
clientSocket.close()
client.py
import socket
# creates socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
host = socket.gethostname() # or just use (host = '')
port = 9999
s.connect((host, port))
tm = s.recv(1024) # msg can only be 1024 bytes long
s.close()
print("the time we got from the server is %s" % tm.decode('ascii'))
Run server.py first, then run client.py.
This is just send and receive the currentTime.
What's new in Python 3.4 sockets?
A major difference between python 2.7 sockets and python 3.4 sockets is the sending messages. you have to .encode() (usually using 'ascii' or blank as parameters/arguments)
and then using .decode()
For example use .encode() to send, and use .decode() to receive.
Extra info: client/server socket tutorial

Netcat implementation in Python

I found this and am using it as my base, but it wasn't working right out of the box. My goal is also to treat it as a package instead of a command line utility, so my code changes will reflect that.
class Netcat:
def __init__(self, hostname, port):
self.hostname = hostname
self.port = port
def send(self, content):
self.socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.socket.connect((self.hostname, self.port))
self.socket.setblocking(0)
result = '';
read_ready, write_ready, in_error = select.select([self.socket], [], [self.socket], 5)
if(self.socket.sendall(content) != None):
return
while(1):
buffer = ''
try:
buffer = self.socket.recv(128)
while(buffer != ''):
result += buffer
try:
buffer = self.socket.recv(128)
except socket.error as err:
print (err, type(err))
buffer = ''
if(buffer == ''):
break
except socket.error as err:
print (err, type(err))
if(buffer == ''):
break
return result
When I send a basic command to my device, it returns the following.
50PMA-019 Connection Open
Atten #1 = 63dB
My code reads the first line, but then I get an error saying that the connection is temporarily unavailable and it does not get the second line. If I change it to blocking, it just blocks and never returns. Any thoughts?

Does it work if you just use nc?
I think you should try something a little simpler:
import socket
def netcat(hostname, port, content):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((hostname, port))
s.sendall(content)
s.shutdown(socket.SHUT_WR)
while 1:
data = s.recv(1024)
if len(data) == 0:
break
print("Received:", repr(data))
print("Connection closed.")
s.close()
I added the shutdown call because maybe your device is waiting for you to say you're done sending data. (That would be a little weird, but it's possible.)

The following is a working implementation on python3:
import socket
def netcat(host, port, content):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host, int(port)))
s.sendall(content.encode())
s.shutdown(socket.SHUT_WR)
while True:
data = s.recv(4096)
if not data:
break
print(repr(data))
s.close()
It can be used to send "content" to a "host" on "port" (which all might be entered as sting).
Regards

if you don't mind scrapping that code altogether, you might like to look at scapy -- it's basically the swiss army knife of packet tools in python. take a look at the interactive tutorial to see if it fits your needs.
if you'd like something higher-level than packets twisted is the go-to library for networking in python... unfortunately the learning curve is a tad steep.