We Keep Coding

python socket / how do you detect disconnection from clients and detect number of connections per user

this is for server authentication for my application.
(im working on login function so dont mind about it)
what i wanna do is to make server receive heartbeat from client
and close client socket if its doesnt respond in a few min
also i want to detect number of connections per user.
for receiving heartbeat, i can make the client send heartbeat constantly but
how do you make the server decect it? i know time measurement is needed but
if i put time.perf_counter() right before 'client_socket.recv(1024)' the counter function wont be executed because its waiting on receiving. so how would i solve this?
and im also trying to make it detect number of connections per user. (5 maximum connections per user) for detection, i give username + 1 when a user is connected and give -1 when the user disconnects but im not sure if the method im doing is correct or a good way to do so.
i'd be appreciated if you could help me out
------------------------server----------------------------
import socket
from _thread import *
import sys
import time
username = ['test123', 'hongengi']
userconnect= 0
def threaded(client_socket, addr):
print('Connected by :', addr[0], ':', addr[1])
while True:
try:
data = client_socket.recv(1024)
print (data.decode())
print('Received from ' + addr[0],':',addr[1] , data.decode())
if data.decode() == ".": # heartbeat
heartbeat = time.perf_counter()
print ("heartbeat")
if data.decode() == "test123":
print ("login success")
userconnect == userconnect + 1
if not data:
print ("no data / disconnect ")
print('Disconnected by ' + addr[0],':',addr[1])
userconnect == userconnect - 1
break
client_socket.send(data)
except (ConnectionResetError, socket.error) as e:
print ("error occurs")
print('Disconnected by ' + addr[0],':',addr[1])
userconnect == userconnect - 1
break
client_socket.close()
HOST = '127.0.0.1'
PORT = 5000
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen()
print('server start')
while True:
print('wait')
client_socket, addr = server_socket.accept()
start_new_thread(threaded, (client_socket, addr))
server_socket.close()
------------------------client----------------------------
import socket
SERVER_IP = 'localhost'
SERVER_PORT = 5000
SIZE = 100
SERVER_ADDR = (SERVER_IP, SERVER_PORT)
heartbeat = "."
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(SERVER_ADDR)
#username = "test123"
#userpass = "123123"
while True:
client_socket.send(heartbeat.encode())
msg = client_socket.recv(SIZE)
print (msg.decode())

One end of a socket is never "notified" when the other socket closes. There is no direct connection, so the only way to tell this is to time out. You can use socket.timeout to establish a timeout time. Your recv will then return with 0 bytes, and that's an indication that your timeout expired.
How to set timeout on python's socket recv method?

How to receive all data after TCP connection has been closed by the peer?

Running some production code I faced the following problem:
When sending HTTP requests to some server, server immediately closes the connection after sending response, which, for some reason, results in data loss.
Analyzing TCP dumps i can see that conversation goes as this:
client request
server ack
server push
server fin, ack (sent after ~0.000020 secs after previous push)
As the result my code can't get data sent by the server, (i'm guessing because of the small delay after push POLLHUP event might go before POLLIN?)
Trying to mimic the problem I've written the following code:
(It mimics the client behaviour on my side)
client:
import time
import socket
from errno import EAGAIN
from select import poll, POLLIN, POLLPRI, POLLERR, POLLHUP, POLLNVAL
def main(buf=""):
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.setblocking(False)
client.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
client.setsockopt(socket.SOL_SOCKET, socket.SO_KEEPALIVE, 1)
polling_object = poll()
polling_object.register(client, POLLPRI | POLLIN | POLLERR | POLLHUP)
in_buf = ""
sock_closed = False
try:
client.connect(("127.0.0.1", 8877))
except socket.error, e:
pass
while True and not sock_closed:
events = polling_object.poll(0)
for _, e in events:
if e & (POLLIN | POLLPRI):
while True:
try:
data = client.recv(1024)
if data:
in_buf += data
elif data == "":
client.close()
sock_closed = True
break
except socket.error, e:
if e.args[0] == EAGAIN:
break
else:
raise
elif e & (POLLERR|POLLHUP|POLLNVAL):
client.close()
sock_closed = True
if buf and not sock_closed:
try:
b_sent = client.send(buf)
if b_sent == len(buf):
buf = ""
else:
buf = buf[b_sent:]
except socket.error, e:
if e.args[0] != EAGAIN:
client.close()
sock_closed = True
time.sleep(0.5)
if sock_closed:
return in_buf
if __name__ == '__main__':
import sys
if len(sys.argv) > 1:
buf = sys.argv[1]
else:
buf = 'hello'
print main(buf)
server
import datetime
import time
import socket
def main():
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server.bind(("127.0.0.1", 8877))
server.listen(0)
client, _ = server.accept()
t1 = time.time()
data = ""
while not data:
data += client.recv(1024)
print "recv data %s" % data
client.sendall('{"ok": 1}')
t2 = time.time()
client.close()
t3 = time.time()
server.close()
return t1, t2, t3
if __name__ == '__main__':
c_r, d_s, c_c = main()
print "Connection received at ", datetime.datetime.fromtimestamp(c_r)
print "All Data sent after %.12f secs" % (d_s - c_r)
print "Connection closed after %.12f secs" % (c_c - d_s)
Running this code won't help me reproduce the problem because my client still can get data from socket buffer, which is kind of obviously by just following the code. The only difference is that in tcp dump it goes like this:
client request
server ack
server push
client ack
server fin, ack
I'm wondering is there a way to send fin, ack right after push without "letting" client to sent ack? Can it be done using python?

How do i get the socket to continue listening to the client and continuosly print information from the client

Server Side (server prints the first line of information sent from the client then it JUST STAYS open and doesn't seem to continue listening it just stays open. Is there a way to get the server to listen to the client more and print?)
import time
import socket
import signal
from datetime import datetime
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(1024) # become a server socket, maximum 5 connectionn
def clientsocketentry():
while True:
connection, addr = serversocket.accept()
buf = connection.recv(64)
if not buf:
break
elif buf == 'killsrv':
connection.close()
sys.exit()
else:
print (buf)
buf = buf.decode("utf-8")
buf = buf.split(',')
serverLong = buf[0]
print('Longitude:' + '' + serverLong)
serverLat = buf[1]
print('Lattitude:' + '' + serverLat)
serverAlt = buf[2]
print('Altitude:' + '' + serverAlt)
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
connection.close()
clientsocketentry()
Client Side (I am only able to send one of the strings of information then the server stays open ut does not take more information from the client)
import socket
import time
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))
a = '39.163100,-76.899428,0'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '4.2,2.2415,0'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '43454,354354,35435'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '435742.,35.452,52434'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
clientsocket.close()

If you accept one single connection at a time (no need for a 1024 backlog then...) you can simply nest 2 loops: the outer one waiting for new connections the inner one processing input from the only one established connection. If you need to process more than one connection, you will have to use select or threads.
Here is an example for one single connection:
def clientsocketentry():
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(5) # become a server socket, maximum 5 connectionn
cont = True
while cont:
connection, addr = serversocket.accept()
while True:
buf = connection.recv(64)
if len(buf) == 0: # end of connection
connection.close()
break
elif buf == b'killsrv': # request for closing server (beware of the byte string)
connection.close()
serversocket.close()
cont = False
break
else:
print (buf)
buf = buf.decode("utf-8")
buf = buf.split(',')
serverLong = buf[0]
print('Longitude:' + '' + serverLong)
serverLat = buf[1]
print('Lattitude:' + '' + serverLat)
serverAlt = buf[2]
print('Altitude:' + '' + serverAlt)
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
# connection.close() # wait for client to close

You are closing the socket at the end of your print logic in the ClientSocketEntry function.
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
connection.close()
Instead of closing the connection there only close it when the user sends killsrv
Because every time you close the connection on the socket it is saying that you are expecting another client to connect to the server. So maybe before going into the while statement accept the connection and then pass it into the while statement, because the way you have it structured at the moment is expecting multiple connections from different clients.

CPU reaches 100 percent with starting of my python script which handles HTTP requests through socket programming?

I am writing a script for handling HTTP request through socket programming. My Python Script just reads each HTTP response, search for few keywords and increment the counters.
Only starting the script takes CPU upto 90-99% when there is no incoming messages. How should i handle this?
HOST = ''
SOCKET_LIST = []
RECV_BUFFER = 40966
PORT=int(sys.argv[1])
serviceInitiatedEvent=0
deliveredEvent=0
EstablishedEvent=0
ConnectionClearedEvent=0
def chat_server():
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)
SOCKET_LIST.append(server_socket)
print "Chat server started on port " + str(PORT)
try:
while 1:
ready_to_read,ready_to_write,in_error = select.select(SOCKET_LIST,[],[],0)
for sock in ready_to_read:
if sock == server_socket:
sockfd, addr = server_socket.accept()
SOCKET_LIST.append(sockfd)
else:
try:
data = sock.recv(RECV_BUFFER)
if data:
if re.search('serviceInitiatedEvent></SOAP-ENV',data):
global serviceInitiatedEvent
serviceInitiatedEvent+=1
if re.search('deliveredEvent></SOAP-ENV',data):
global deliveredEvent
deliveredEvent+=1
else:
if sock in SOCKET_LIST:
SOCKET_LIST.remove(sock)
except:
broadcast(server_socket, sock, "Client (%s, %s) is offline\n" % addr)
continue
except KeyboardInterrupt:
print "service Initiated Event:%s" % (serviceInitiatedEvent)
print "delivered Event: %s" % (deliveredEvent)
server_socket.close()
if __name__ == "__main__":
sys.exit(chat_server())

If you have code with while 1 loop utilizing 100%, that's probably the culprit. It's called busy waiting.
select function has timeout parameter that specifies how long it should wait for events. In your code, you set it to 0, so that when there is no data available in sockets, control returns immediately, causing busy waiting loop.
Specify some larger timeout, based on your needs, so that your code won't spin when there's nothing to do:
ready_to_read,ready_to_write,in_error = select.select(SOCKET_LIST,[],[], 1)
# ^^^ here

Sending, receiving with python socket

I'm currently trying to write process that embeds a sequence of n IPs into packets and send it off to n server. Each server remove the outermost IP and then forward it to said IP. This is exactly like tunneling I know. During the process I also want the server to do a traceroute to where it's forwarding the packet and send that back to the previous server.
My code currently will forward the packets but it's stuck on performing the traceroute and getting it. I believe it's currently stuck in the while loop in the intermediate server. I think it's having something to do with me not closing the sockets properly. Any suggestion?
Client
#!/usr/bin/env python
import socket # Import socket module
import sys
import os
s = socket.socket() # Create a socket object
host = socket.gethostname() # Get local machine name
port = 17353 # Reserve a port
FILE = raw_input("Enter filename: \n ")
NIP = raw_input("Enter Number of IPs: ")
accepted_IP = 0
IP= []
while accepted_IP < int(NIP):
IP.append(raw_input("Enter destination IP: \n"))
accepted_IP +=1
#cIP = raw_input("Enter intemediate IP: \n ")
ipv = raw_input("Enter IP version... 4/6")
try:
s.connect((host, port))
print "Connection sucessful!"
except socket.error as err:
print "Connection failed. Error: %s" %err
quit()
raw = open(FILE,"rb")
size = os.stat(FILE).st_size
ls = ""
buf = 0
for i in IP:
while len(i) < 15:
i += "$"
ls += i
header = ipv+NIP+ls+FILE
print ls
s.sendall(header + "\n")
print "Sent header"
data = raw.read(56) +ipv + NIP + ls
print "Begin sending file"
while buf <= size:
s.send(data)
print data
buf += 56
data = raw.read(56) + ipv + NIP + ls
raw.close()
print "Begin receiving traceroute"
with open("trace_log.txt","w") as tracert:
trace = s.recv(1024)
while trace:
treacert.write(trace)
if not trace: break
trace = s.recv(1024)
print "finished forwarding"
s.close()
Intermediate server
#!/usr/bin/env python
import socket
import subprocess
srvsock = socket.socket( socket.AF_INET, socket.SOCK_STREAM )
srvsock.bind( (socket.gethostname(), 17353) )
srvsock.listen( 5 ) # Begin listening with backlog of 5
# Run server
while True:
clisock, (remhost, remport) = srvsock.accept() #Accept connection
print
d = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
header = ""
while True:
b = clisock.recv(1)
if b == "\n":
break
header += b
num = 15 * int(header[1]) + 2
file_name = header[num:]
nheader = header[0]+ str(int(header[1])-1) + header[17:]
d.connect((socket.gethostname(), 12355))
d.sendall(nheader+'\n')
print "begin forwarding"
while True:
raw = clisock.recv(56 + num) # recieve data
ip = raw[-15:] # extract IP
ipv, NIP = raw[57] , str(int(raw[57])-1)
if NIP == "0":
while (raw):
print "stuck in this loop"
d.send(raw[:56])
raw=clisock.recv(56+num)
if not raw: break
else:
while (raw):
print raw[:57] + NIP + raw[59:-15]
print "\n"
d.send(raw[:57] + NIP + raw[59:-15])
raw = clisock.recv(56+num)
if not raw :break
print "Finish forwarding"
d.close()
break
print "Begin traceroute"
tracrt = subprocess.Popen(['traceroute','google.com'], stdout=subprocess.PIPE)
s.sendall(tracrt.communicate()[0])
print "Finished"
clisock.close()
s.close()
Destination server
import socket
s = socket.socket()
host = socket.gethostname()
port = 12355
s.bind((host,port))
s.listen(5)
while True:
csock, (client, cport) = s.accept()
print client
header = ""
while True:
b = csock.recv(1)
if b == "\n":
break
header += b
file_name = header[2:]
r = open("File_test_"+file_name,"wb")
print 'Opening file for writing'
while True:
print "Begin writing file" + " " + file_name
raw = csock.recv(56)
while (raw):
print raw
r.write(raw)
raw = csock.recv(56)
r.flush()
r.close()
print "finish writing"
break
print "closing connection"
csock.close()
s.close()

The intermediate server is stuck in clisock.recv() in this loop because the break condition not raw isn't met before the connection is closed by the client, and the client doesn't close the connection before receiving the traceroute from the intermediate server, so they are waiting on each other.
To remedy this, you might consider sending the file size to the intermediate server, so that it can be used to determine when the receive loop is done. Or, if your platform supports shutting down one half of the connection, you can use
s.shutdown(socket.SHUT_WR)
in the client after sending the file.