We Keep Coding

python socket / how do you detect disconnection from clients and detect number of connections per user

this is for server authentication for my application.
(im working on login function so dont mind about it)
what i wanna do is to make server receive heartbeat from client
and close client socket if its doesnt respond in a few min
also i want to detect number of connections per user.
for receiving heartbeat, i can make the client send heartbeat constantly but
how do you make the server decect it? i know time measurement is needed but
if i put time.perf_counter() right before 'client_socket.recv(1024)' the counter function wont be executed because its waiting on receiving. so how would i solve this?
and im also trying to make it detect number of connections per user. (5 maximum connections per user) for detection, i give username + 1 when a user is connected and give -1 when the user disconnects but im not sure if the method im doing is correct or a good way to do so.
i'd be appreciated if you could help me out
------------------------server----------------------------
import socket
from _thread import *
import sys
import time
username = ['test123', 'hongengi']
userconnect= 0
def threaded(client_socket, addr):
print('Connected by :', addr[0], ':', addr[1])
while True:
try:
data = client_socket.recv(1024)
print (data.decode())
print('Received from ' + addr[0],':',addr[1] , data.decode())
if data.decode() == ".": # heartbeat
heartbeat = time.perf_counter()
print ("heartbeat")
if data.decode() == "test123":
print ("login success")
userconnect == userconnect + 1
if not data:
print ("no data / disconnect ")
print('Disconnected by ' + addr[0],':',addr[1])
userconnect == userconnect - 1
break
client_socket.send(data)
except (ConnectionResetError, socket.error) as e:
print ("error occurs")
print('Disconnected by ' + addr[0],':',addr[1])
userconnect == userconnect - 1
break
client_socket.close()
HOST = '127.0.0.1'
PORT = 5000
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen()
print('server start')
while True:
print('wait')
client_socket, addr = server_socket.accept()
start_new_thread(threaded, (client_socket, addr))
server_socket.close()
------------------------client----------------------------
import socket
SERVER_IP = 'localhost'
SERVER_PORT = 5000
SIZE = 100
SERVER_ADDR = (SERVER_IP, SERVER_PORT)
heartbeat = "."
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect(SERVER_ADDR)
#username = "test123"
#userpass = "123123"
while True:
client_socket.send(heartbeat.encode())
msg = client_socket.recv(SIZE)
print (msg.decode())

One end of a socket is never "notified" when the other socket closes. There is no direct connection, so the only way to tell this is to time out. You can use socket.timeout to establish a timeout time. Your recv will then return with 0 bytes, and that's an indication that your timeout expired.
How to set timeout on python's socket recv method?

How do i get the socket to continue listening to the client and continuosly print information from the client

Server Side (server prints the first line of information sent from the client then it JUST STAYS open and doesn't seem to continue listening it just stays open. Is there a way to get the server to listen to the client more and print?)
import time
import socket
import signal
from datetime import datetime
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(1024) # become a server socket, maximum 5 connectionn
def clientsocketentry():
while True:
connection, addr = serversocket.accept()
buf = connection.recv(64)
if not buf:
break
elif buf == 'killsrv':
connection.close()
sys.exit()
else:
print (buf)
buf = buf.decode("utf-8")
buf = buf.split(',')
serverLong = buf[0]
print('Longitude:' + '' + serverLong)
serverLat = buf[1]
print('Lattitude:' + '' + serverLat)
serverAlt = buf[2]
print('Altitude:' + '' + serverAlt)
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
connection.close()
clientsocketentry()
Client Side (I am only able to send one of the strings of information then the server stays open ut does not take more information from the client)
import socket
import time
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('localhost', 8089))
a = '39.163100,-76.899428,0'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '4.2,2.2415,0'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '43454,354354,35435'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
a = '435742.,35.452,52434'
clientsocket.send(a.encode('utf-8'))
time.sleep(5)
clientsocket.close()

If you accept one single connection at a time (no need for a 1024 backlog then...) you can simply nest 2 loops: the outer one waiting for new connections the inner one processing input from the only one established connection. If you need to process more than one connection, you will have to use select or threads.
Here is an example for one single connection:
def clientsocketentry():
serversocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serversocket.bind(('localhost', 8089))
serversocket.listen(5) # become a server socket, maximum 5 connectionn
cont = True
while cont:
connection, addr = serversocket.accept()
while True:
buf = connection.recv(64)
if len(buf) == 0: # end of connection
connection.close()
break
elif buf == b'killsrv': # request for closing server (beware of the byte string)
connection.close()
serversocket.close()
cont = False
break
else:
print (buf)
buf = buf.decode("utf-8")
buf = buf.split(',')
serverLong = buf[0]
print('Longitude:' + '' + serverLong)
serverLat = buf[1]
print('Lattitude:' + '' + serverLat)
serverAlt = buf[2]
print('Altitude:' + '' + serverAlt)
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
# connection.close() # wait for client to close

You are closing the socket at the end of your print logic in the ClientSocketEntry function.
serverTime = datetime.now().strftime('%Y-%m-%d %H:%M:%S')
print('Time of Entry:' + ' ' + serverTime)
connection.close()
Instead of closing the connection there only close it when the user sends killsrv
Because every time you close the connection on the socket it is saying that you are expecting another client to connect to the server. So maybe before going into the while statement accept the connection and then pass it into the while statement, because the way you have it structured at the moment is expecting multiple connections from different clients.

Python - Stream - How to send data to server faster

I want to send strings from a text file to my local port but i have to open connection and close it for each string. Therefore, data flow is very slow.(Almost in two seconds 1 string). How can i make it faster?
while 1:
conn, addr = s.accept()
line_number = random.randint(1,2261074)
liste.append(line_number)
line = linecache.getline(filename,line_number)
sendit = line.split(" ")[1]
print type(sendit)
print "sending: " + sendit
conn.send(sendit)
conn.close()
print('End Of Stream.')

The answer suggests sending 10 messages to Spark on each connection, separating each message by 1 second, then closing the connection.
It might be better to keep the connection open and use a non-blocking socket at the server end.
The server code below keeps the connection open, sending messages in batches on a non-blocking socket, with an idle delay between each batch.
This can be used to test how fast Spark can receive messages. I've set it to send in batches of 50 messages, then wait 1 second before sending the next 50.
Spark receives all messages OK on my machine, even if I set the idle delay to zero.
You can experiment and adjust as needed for your application.
Server code:
import socket
import time
import select
def do_send(sock, msg, timeout):
readers = []
writers = [sock]
excepts = [sock]
rxs, txs, exs = select.select(readers, writers, excepts, timeout)
if sock in exs:
return False
elif sock in txs:
sock.send(msg)
return True
else:
return False
host = 'localhost'
port = 9999
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host, port))
s.listen(1)
print "waiting for client"
conn, addr = s.accept()
print "client connected"
conn.setblocking(0)
batchSize = 50
idle = 1.
count = 0
running = 1
while running:
try:
sc = 0
while (sc < batchSize):
if do_send (conn, 'Hello World ' + repr(count) + '\n', 1.0):
sc += 1
count += 1
print "sent " + repr(batchSize) + ", waiting " + repr(idle) + " seconds"
time.sleep(idle)
except socket.error:
conn.close()
running = 0
print "done"
Simple Spark code:
from pyspark import SparkContext
from pyspark.streaming import StreamingContext
sc = SparkContext("local[2]","test")
ssc = StreamingContext(sc, 1)
ststr = ssc.socketTextStream("localhost", 9999)
lines = ststr.flatMap(lambda line: line.split('\n'))
lines.pprint()
ssc.start()
ssc.awaitTermination()
Hope this may be useful.

CPU reaches 100 percent with starting of my python script which handles HTTP requests through socket programming?

I am writing a script for handling HTTP request through socket programming. My Python Script just reads each HTTP response, search for few keywords and increment the counters.
Only starting the script takes CPU upto 90-99% when there is no incoming messages. How should i handle this?
HOST = ''
SOCKET_LIST = []
RECV_BUFFER = 40966
PORT=int(sys.argv[1])
serviceInitiatedEvent=0
deliveredEvent=0
EstablishedEvent=0
ConnectionClearedEvent=0
def chat_server():
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((HOST, PORT))
server_socket.listen(10)
SOCKET_LIST.append(server_socket)
print "Chat server started on port " + str(PORT)
try:
while 1:
ready_to_read,ready_to_write,in_error = select.select(SOCKET_LIST,[],[],0)
for sock in ready_to_read:
if sock == server_socket:
sockfd, addr = server_socket.accept()
SOCKET_LIST.append(sockfd)
else:
try:
data = sock.recv(RECV_BUFFER)
if data:
if re.search('serviceInitiatedEvent></SOAP-ENV',data):
global serviceInitiatedEvent
serviceInitiatedEvent+=1
if re.search('deliveredEvent></SOAP-ENV',data):
global deliveredEvent
deliveredEvent+=1
else:
if sock in SOCKET_LIST:
SOCKET_LIST.remove(sock)
except:
broadcast(server_socket, sock, "Client (%s, %s) is offline\n" % addr)
continue
except KeyboardInterrupt:
print "service Initiated Event:%s" % (serviceInitiatedEvent)
print "delivered Event: %s" % (deliveredEvent)
server_socket.close()
if __name__ == "__main__":
sys.exit(chat_server())

If you have code with while 1 loop utilizing 100%, that's probably the culprit. It's called busy waiting.
select function has timeout parameter that specifies how long it should wait for events. In your code, you set it to 0, so that when there is no data available in sockets, control returns immediately, causing busy waiting loop.
Specify some larger timeout, based on your needs, so that your code won't spin when there's nothing to do:
ready_to_read,ready_to_write,in_error = select.select(SOCKET_LIST,[],[], 1)
# ^^^ here

Network Bandwidth testing or speed testing in python 2.7

I am needing to test network bandwidth between a client and a server. I can do a simple drag and drop of the file and I get somewhere in the 800meg range according to the windows network monitor. The goal is to perform this same test using a python app, something like speedtest.net for inside the network. Here is the code I have been using but the results do not give me anything like I am seeing and I may just not understand them. The code comes from this site https://svn.python.org/projects/python/trunk/Demo/sockets/throughput.py
#! /usr/bin/env python
# Test network throughput.
#
# Usage:
# 1) on host_A: throughput -s [port] # start a server
# 2) on host_B: throughput -c count host_A [port] # start a client
#
# The server will service multiple clients until it is killed.
#
# The client performs one transfer of count*BUFSIZE bytes and
# measures the time it takes (roundtrip!).
import sys, time
from socket import *
MY_PORT = 50000 + 42
BUFSIZE = 1024
def main():
if len(sys.argv) < 2:
usage()
if sys.argv[1] == '-s':
server()
elif sys.argv[1] == '-c':
client()
else:
usage()
def usage():
sys.stdout = sys.stderr
print 'Usage: (on host_A) throughput -s [port]'
print 'and then: (on host_B) throughput -c count host_A [port]'
sys.exit(2)
def server():
if len(sys.argv) > 2:
port = eval(sys.argv[2])
else:
port = MY_PORT
s = socket(AF_INET, SOCK_STREAM)
s.bind(('', port))
s.listen(1)
print 'Server ready...'
while 1:
conn, (host, remoteport) = s.accept()
while 1:
data = conn.recv(BUFSIZE)
if not data:
break
del data
conn.send('OK\n')
conn.close()
print 'Done with', host, 'port', remoteport
def client():
if len(sys.argv) < 4:
usage()
count = int(eval(sys.argv[2]))
host = sys.argv[3]
if len(sys.argv) > 4:
port = eval(sys.argv[4])
else:
port = MY_PORT
testdata = 'x' * (BUFSIZE-1) + '\n'
t1 = time.time()
s = socket(AF_INET, SOCK_STREAM)
t2 = time.time()
s.connect((host, port))
t3 = time.time()
i = 0
while i < count:
i = i+1
s.send(testdata)
s.shutdown(1) # Send EOF
t4 = time.time()
data = s.recv(BUFSIZE)
t5 = time.time()
print data
print 'Raw timers:', t1, t2, t3, t4, t5
print 'Intervals:', t2-t1, t3-t2, t4-t3, t5-t4
print 'Total:', t5-t1
print 'Throughput:', round((BUFSIZE*count*0.001) / (t5-t1), 3),
print 'K/sec.'
main()
Here is a sample output
OK
Raw timers: 1497614245.55 1497614245.55 1497614245.55 1497614268.85 1497614268.85
Intervals: 0.000999927520752 0.000999927520752 23.2929999828 0.00300002098083
Total: 23.2979998589
Throughput: 43952.271 K/sec.

I'm doing the similar test and looks like the result could be better if the size of test data is increased -- but still can't make use of the full available bandwidth (1Gbps in my case), don't know why.
For more detail, I was testing the bandwidth from a Win7 client to a Win7 server, If I changes the testdata to be 4 times of receiving buffer size, the network usage could be up to more than 80% on a 1Gbps link. If the testdata's size is similar with the buffer size, the usage would only be a little more than 30%.
When I do the test between a Debian8 client and the same Win7 server, the usage could be up to near 100%. Also, when I just copy a large file between the same Win7 machines through file sharing, it's also 100% usage. Looks like the problem lies in the client side. Any suggestion would be appreciated.
Code for server (Python3.6):
from __future__ import print_function
import datetime
import socket
HOST = '0.0.0.0'
PORT = 50000
BUFFER = 4096
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind((HOST,PORT))
sock.listen(0)
print('listening at %s:%s\n\r' %(HOST, PORT))
while True:
client_sock, client_addr = sock.accept()
starttime = datetime.datetime.now()
print(starttime, end="")
print('%s:%s connected\n\r' % client_addr)
count = 0
while True:
data = client_sock.recv(BUFFER)
if data:
count += len(data)
del data
continue
client_sock.close()
endtime = datetime.datetime.now()
print(endtime)
print('%s:%s disconnected\n\r' % client_addr)
print('bytes transferred: %d' % count)
delta = endtime - starttime
delta = delta.seconds + delta.microseconds / 1000000.0
print('time used (seconds): %f' % delta)
print('averaged speed (MB/s): %f\n\r' % (count / 1024 / 1024 / delta))
break
sock.close()
Code for client (Python3.6):
import datetime
import socket
HOST = 'a.b.c.d'
PORT = 50000
BUFFER = 4096
testdata = b'x' * BUFFER * 4
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((HOST,PORT))
for i in range(1, 1000000):
sock.send(testdata)
sock.close()

I have been trying to achieve the same and to an extent i have using this code
Speedtest.py. They do even provide a API if you wish to render the test results on a webpage, which would require a python framework. I'd recommend flask.
Speedtest.net - Uses sockets for tests instead of https which is used in this code.
ps - If you have already achieved a better approach, i'd be very nice of you to tell us all.