I want a thread to wait for a message from another script.
I don't want to use time.sleep() as it creates time gaps and if I need my thread to wake up and continue running, it might delay too much and I'm aiming for fastest performance. I don't won't to use while(NOT_BEING_CALLED_BY_THE_OTHER_THREAD) because it will eat up my CPU and I'm also aiming to keep my CPU usage as low as possible (as there will be more thread doing the same at the same time).
In Pseudo-code it should look like this:
do_stuff()
wait_for_being_called() #Rise immediately after being called (or as soon as possible)
do_more_stuff()
The purpose of this is to use data that wasn't available before being called, there is a script that checks for the data availability (a single thread running) and many which await for the data they need to be available (the single script checks it, and should call them if the data is available). It's kind of like std::condition_variable in c++, only I want my other, external script to be able to wake the awaiting script.
How can I achieve something like this? What should check_for_events.py contain?
#check_for_events.py
for data_node in data_list:
"""
What do I do here, assuming I have the thread id?
"""
If you have two different scripts, probably the best thing to use is select. Here's an example of what I mean:
from __future__ import print_function
import select
import socket
import sys
import time
from random import randint
def serve():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
port = randint(10000, 50000)
with open('.test_port', 'w') as f:
f.write('{}'.format(port))
sock.bind(('127.0.0.1', port))
sock.listen(1)
not_finished = True
while not_finished:
try:
print('*'*40)
print('Waiting for connection...')
conn, addr = sock.accept()
print('Waiting forever for data')
select.select([conn], [], [])
data = conn.recv(2048)
print('got some data, so now I can go to work!')
print('-'*40)
print('Doing some work, doo da doo...')
print('Counting to 20!')
for x in range(20):
print(x, end='\r')
time.sleep(0.5)
print('** Done with work! **')
print('-'*40)
conn.close()
except KeyboardInterrupt:
print('^C caught, quitting!')
not_finished = False
def call():
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print('Connecting')
with open('.test_port') as f:
port = int(f.read())
sock.connect(('127.0.0.1', port))
sock.sendall(b'This is a message')
sock.close()
print('Done')
if __name__ == '__main__':
if 'serve' in sys.argv:
serve()
elif 'call' in sys.argv:
call()
This allows the caller to actually communicate information with the runner. You could also set it up to listen for multiple incoming connections and toss them in the pool to select from, if that's something that you need.
But if you really just want to block until another program calls you, then you could make this even more simple by removing the parts between conn, add = sock.accept() and conn.close() (other than your own work, of course).
Related
I'm trying to write a simple daemon that listens for orders on a Unix socket. The following works, but the connection.recv(1024) line blocks, meaning I can't kill the server gracefully:
import socket, os
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server:
server.bind("/tmp/sock")
server.listen()
connection, __ = server.accept()
with connection:
while True:
data = connection.recv(1024)
print("Hi!") # This line isn't executed 'til data is sent
if data:
print(data.decode())
Ideally, I'd like to place all of this inside a Thread that checks a self.should_stop property every self.LOOP_TIME seconds, and if that value is set to True, then exit. However, as that .recv() line blocks, there's no way for my program to be doing anything other than waiting at any given time.
Surely there's a proper way to do this, but as I'm new to sockets, I have no idea what that is.
Edit
Jeremy Friesner's answer put me on the right track. I realised that I could allow the thread to block and simply set .should_stop then pass an b"" to the socket so that it'd un-block, see that it should stop, and then exit cleanly. Here's the end result:
import os
import socket
from pathlib import Path
from shutil import rmtree
from threading import Thread
class MyThreadThing(Thread):
RUNTIME_DIR = Path(os.getenv("XDG_RUNTIME_DIR", "/tmp")) / "my-project-name"
def __init__(self):
super().__init__(daemon=True)
self.should_stop = False
if self.RUNTIME_DIR.exists():
rmtree(self.RUNTIME_DIR)
self.RUNTIME_DIR.mkdir(0o700)
self.socket_path = self.RUNTIME_DIR / "my-project.sock"
def run(self) -> None:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.bind(self.socket_path.as_posix())
s.listen()
while True:
connection, __ = s.accept()
action = ""
with connection:
while True:
received = connection.recv(1024).decode()
action += received
if not received:
break
# Handle whatever is in `action`
if self.should_stop:
break
self.socket_path.unlink()
def stop(self):
"""
Trigger this when you want to stop the listener.
"""
self.should_stop = True
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as s:
s.connect(self.socket_path.as_posix())
s.send(b"")
Using arbitrary-length timeouts is always a bit unsatisfactory -- either you set the timeout-value to a relatively long time, in which case your program becomes slow to react to the quit-request, because it is pointlessly waiting for timeout period to expire... or you set the timeout-value to a relatively short time, in which case your program is constantly waking up to see if it should quit, wasting CPU power 24/7 to check for an event which might never arrive.
A more elegant way to deal with the problem is to create a pipe, and send a byte on the pipe when you want your event-loop to exit. Your event loop can simultaneously "watch" both the pipe's reading-end file-descriptor and your networking-socket(s) via select(), and when that file-descriptor indicates it is ready-for-read, your event loop can respond by exiting. This approach is entirely event-driven, so it requires no CPU wakeups except when there is actually something to do.
Below is an example version of your program that implements a signal-handler for SIGINT (aka pressing Control-C) to sends the please-quit-now byte on the pipe:
import socket, os
import select
import signal, sys
# Any bytes written to (writePipeFD) will become available for reading on (readPipeFD)
readPipeFD, writePipeFD = os.pipe()
# Set up a signal-handler to handle SIGINT (aka Ctrl+C) events by writing a byte to the pipe
def signal_handler(sig, frame):
print("signal_handler() is executing -- SIGINT detected!")
os.write(writePipeFD, b"\0") # doesn't matter what we write; a single 0-byte will do
signal.signal(signal.SIGINT, signal_handler)
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as serverSock:
serverSock.bind("/tmp/sock")
serverSock.listen()
# Wait for incoming connection (or the please-quit signal, whichever comes first)
connection = None
while True:
readReady,writeReady,exceptReady = select.select([readPipeFD,serverSock], [], [])
if (readPipeFD in readReady):
print("accept-loop: Someone wrote a byte to the pipe; time to go away!");
break
if (connection in readReady):
connection, __ = serverSock.accept()
break
# Read data from incoming connection (or the please-quit signal, whichever comes first)
if connection:
with connection:
while True:
readReady,writeReady,exceptReady = select.select([readPipeFD,connection], [], [])
if (readPipeFD in readReady):
print("Connection-loop: Someone wrote a byte to the pipe; time to go away!");
break
if (connection in readReady):
data = connection.recv(1024)
print("Hi!") # This line isn't executed 'til data is sent
if data:
print(data.decode())
print("Bye!")
Use a timeout identical to your LOOP_TIME like so:
import socket, os
LOOP_TIME = 10
should_stop = False
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as server:
server.bind("/tmp/sock")
server.listen()
connection, __ = server.accept()
connection.settimeout(LOOP_TIME)
with connection:
while not should_stop:
try:
data = connection.recv(1024)
except socket.timeout:
continue
print("Hi!") # This line isn't executed 'til data is sent
if data:
print(data.decode())
You may use select, but if it's only a single simple socket, this way is a bit less complicated.
You can choose to place it in a different thread with a self.should_stop or just at the main - it will now listen to the KeyboardInterrupt.
This is my code:
socketcheck.py
import time
import subprocess
subprocess.Popen(["python", "server.py"])
for i in range(10):
time.sleep(2)
print i
def print_from_server(data):
print data
server.py
import socket
from socketcheck import print_from_server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost',3005))
client_connected = 1
while 1:
s.listen(1)
conn, addr = s.accept()
data = conn.recv(1024)
if data:
client_connected = 0
else: break
if client_connected == 0:
print 'data received'
print_from_server(data)
client_connected = 1
conn.sendall(data)
client.py
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost',3005))
s.sendall('Hello, world')
data = s.recv(1024)
#s.close()
print 'Received', repr(data)
What I am trying to do here is, run socketcheck.py which runs server.py in background and listens for a client connection. So whatever data the client sends, I want to pass it on to socketcheck.py. Is this valid? If so, then how do I achieve it?
Now when I try and run socketcheck.py, the for loop is running indefinitely.
Thanks :)
EDIT:
This initially I tried as a single program, but until the client gets connected, the rest of the program doesn't execute(blocking), with the setblocking(0) the program flow wouldn't stop but when the client connects to server it doesn't print(do anything). The server code looked something like this:
import socket
from socketcheck import print_from_server
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('localhost',3005))
s.setblocking(0)
while 1:
try:
s.listen(1)
conn, addr = s.accept()
conn.setblocking(0)
data = conn.recv(1024)
if not data: break
print 'data received'
conn.sendall(data)
except:
print 'non blocking'
print 'the lengthy program continues from here'
The reason why your program crashes your computer is simple:
You have a while loop which calls print_from_server(data), and each time it calls it, a new subprocess gets created via subprocess.Popen(["python", "server.py"]).
The reason for creating a new popen each time is a bit more complicated: You open a new server.py program in socketcheck.py. If this new server.py program calls print_from_server(data), this is the first time print_from_server(data) gets called (for the new server.py program). So the global commands (such as popen) are executed, since they are always executed once.
The number of processes running will explode quickly and you computer crashes.
One additional remark: You cannot print to console with a print command in a subprocess, since there is no console attached to that subprocess, you can only print to file. If you do that, you'll see that this output explodes quickly from all the processes.
Put socketcheck.py and server.py into one program and everything works fine, or explain why you need two programs.
The functionality can be easily achieved with multithreading :)
I am currently working on a school project where the assignment, among other things, is to set up a threaded server/client system. Each client in the system is supposed to be assigned its own thread on the server when connecting to it. In addition i would like the server to run other threads, one concerning input from the command line and another concerning broadcasting messages to all clients. However, I can't get this to run as i want to. It seems like the threads are blocking each other. I would like my program to take inputs from the command line, at the "same time" as the server listens to connected clients, and so on.
I am new to python programming and multithreading, and allthough I think my idea is good, I'm not suprised my code doesn't work. Thing is I'm not exactly sure how I'm going to implement the message passing between the different threads. Nor am I sure exactly how to implement the resource lock commands properly. I'm going to post the code for my server file and my client file here, and I hope someone could help me with this. I think this actually should be two relative simple scripts. I have tried to comment on my code as good as possible to some extend.
import select
import socket
import sys
import threading
import client
class Server:
#initializing server socket
def __init__(self, event):
self.host = 'localhost'
self.port = 50000
self.backlog = 5
self.size = 1024
self.server = None
self.server_running = False
self.listen_threads = []
self.local_threads = []
self.clients = []
self.serverSocketLock = None
self.cmdLock = None
#here i have also declared some events for the command line input
#and the receive function respectively, not sure if correct
self.cmd_event = event
self.socket_event = event
def openSocket(self):
#binding server to port
try:
self.server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.server.bind((self.host, self.port))
self.server.listen(5)
print "Listening to port " + str(self.port) + "..."
except socket.error, (value,message):
if self.server:
self.server.close()
print "Could not open socket: " + message
sys.exit(1)
def run(self):
self.openSocket()
#making Rlocks for the socket and for the command line input
self.serverSocketLock = threading.RLock()
self.cmdLock = threading.RLock()
#set blocking to non-blocking
self.server.setblocking(0)
#making two threads always running on the server,
#one for the command line input, and one for broadcasting (sending)
cmd_thread = threading.Thread(target=self.server_cmd)
broadcast_thread = threading.Thread(target=self.broadcast,args=[self.clients])
cmd_thread.daemon = True
broadcast_thread.daemon = True
#append the threads to thread list
self.local_threads.append(cmd_thread)
self.local_threads.append(broadcast_thread)
cmd_thread.start()
broadcast_thread.start()
self.server_running = True
while self.server_running:
#connecting to "knocking" clients
try:
c = client.Client(self.server.accept())
self.clients.append(c)
print "Client " + str(c.address) + " connected"
#making a thread for each clientn and appending it to client list
listen_thread = threading.Thread(target=self.listenToClient,args=[c])
self.listen_threads.append(listen_thread)
listen_thread.daemon = True
listen_thread.start()
#setting event "client has connected"
self.socket_event.set()
except socket.error, (value, message):
continue
#close threads
self.server.close()
print "Closing client threads"
for c in self.listen_threads:
c.join()
def listenToClient(self, c):
while self.server_running:
#the idea here is to wait until the thread gets the message "client
#has connected"
self.socket_event.wait()
#then clear the event immidiately...
self.socket_event.clear()
#and aquire the socket resource
self.serverSocketLock.acquire()
#the below is the receive thingy
try:
recvd_data = c.client.recv(self.size)
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
self.socket_event.clear()
self.serverSocketLock.release()
return
print recvd_data
#I put these here to avoid locking the resource if no message
#has been received
self.socket_event.clear()
self.serverSocketLock.release()
except socket.error, (value, message):
continue
def server_cmd(self):
#this is a simple command line utility
while self.server_running:
#got to have a smart way to make this work
self.cmd_event.wait()
self.cmd_event.clear()
self.cmdLock.acquire()
cmd = sys.stdin.readline()
if cmd == "":
continue
if cmd == "close\n":
print "Server shutting down..."
self.server_running = False
self.cmdLock.release()
def broadcast(self, clients):
while self.server_running:
#this function will broadcast a message received from one
#client, to all other clients, but i guess any thread
#aspects applied to the above, will work here also
try:
send_data = sys.stdin.readline()
if send_data == "":
continue
else:
for c in clients:
c.client.send(send_data)
self.serverSocketLock.release()
self.cmdLock.release()
except socket.error, (value, message):
continue
if __name__ == "__main__":
e = threading.Event()
s = Server(e)
s.run()
And then the client file
import select
import socket
import sys
import server
import threading
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#making a lock for the socket resource
self.clientSocketLock = threading.Lock()
self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure wat daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append(listen)
self.running_threads.append(speak)
listen.start()
speak.start()
#this while-loop is just for avoiding the script terminating
while self.client_running:
dummy = 1
#closing the threads if the client goes down
print "Client operating on its own"
self.client.close()
#close threads
for t in self.running_threads:
t.join()
return
#defining "listen"-function
def listenToServer(self):
while self.client_running:
#here i acquire the socket to this function, but i realize I also
#should have a message passing wait()-function or something
#somewhere
self.clientSocketLock.acquire()
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
#releasing the socket resource
self.clientSocketLock.release()
#defining "speak"-function, doing much the same as for the above function
def speakToServer(self):
while self.client_running:
self.clientSocketLock.acquire()
try:
send_data = sys.stdin.readline()
if send_data == "close\n":
print "Disconnecting..."
self.client_running = False
else:
self.client.send(send_data)
except socket.error, (value,message):
continue
self.clientSocketLock.release()
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
I realize this is quite a few code lines for you to read through, but as I said, I think the concept and the script in it self should be quite simple to understand. It would be very much appriciated if someone could help me synchronize my threads in a proper way =)
Thanks in advance
---Edit---
OK. So I now have simplified my code to just containing send and receive functions in both the server and the client modules. The clients connecting to the server gets their own threads, and the send and receive functions in both modules operetes in their own separate threads. This works like a charm, with the broadcast function in the server module echoing strings it gets from one client to all clients. So far so good!
The next thing i want my script to do, is taking specific commands, i.e. "close", in the client module to shut down the client, and join all running threads in the thread list. Im using an event flag to notify the listenToServer and the main thread that the speakToServer thread has read the input "close". It seems like the main thread jumps out of its while loop and starts the for loop that is supposed to join the other threads. But here it hangs. It seems like the while loop in the listenToServer thread never stops even though server_running should be set to False when the event flag is set.
I'm posting only the client module here, because I guess an answer to get these two threads to synchronize will relate to synchronizing more threads in both the client and the server module also.
import select
import socket
import sys
import server_bygg0203
import threading
from time import sleep
class Client(threading.Thread):
#initializing client socket
def __init__(self,(client,address)):
threading.Thread.__init__(self)
self.client = client
self.address = address
self.size = 1024
self.client_running = False
self.running_threads = []
self.ClientSocketLock = None
self.disconnected = threading.Event()
def run(self):
#connect to server
self.client.connect(('localhost',50000))
#self.client.setblocking(0)
self.client_running = True
#making two threads, one for receiving messages from server...
listen = threading.Thread(target=self.listenToServer)
#...and one for sending messages to server
speak = threading.Thread(target=self.speakToServer)
#not actually sure what daemon means
listen.daemon = True
speak.daemon = True
#appending the threads to the thread-list
self.running_threads.append((listen,"listen"))
self.running_threads.append((speak, "speak"))
listen.start()
speak.start()
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
#closing the threads if the client goes down
print "Client operating on its own"
self.client.shutdown(1)
self.client.close()
#close threads
#the script hangs at the for-loop below, and
#refuses to close the listen-thread (and possibly
#also the speak thread, but it never gets that far)
for t in self.running_threads:
print "Waiting for " + t[1] + " to close..."
t[0].join()
self.disconnected.clear()
return
#defining "speak"-function
def speakToServer(self):
#sends strings to server
while self.client_running:
try:
send_data = sys.stdin.readline()
self.client.send(send_data)
#I want the "close" command
#to set an event flag, which is being read by all other threads,
#and, at the same time set the while statement to false
if send_data == "close\n":
print "Disconnecting..."
self.disconnected.set()
self.client_running = False
except socket.error, (value,message):
continue
return
#defining "listen"-function
def listenToServer(self):
#receives strings from server
while self.client_running:
#check if event is set, and if it is
#set while statement to false
if self.disconnected.isSet():
self.client_running = False
try:
data_recvd = self.client.recv(self.size)
print data_recvd
except socket.error, (value,message):
continue
return
if __name__ == "__main__":
c = Client((socket.socket(socket.AF_INET, socket.SOCK_STREAM),'localhost'))
c.run()
Later on, when I get this server/client system up and running, I will use this system on some elevator models we have here on the lab, with each client receiving floor orders or "up" and "down" calls. The server will be running an distribution algorithm and updating the elevator queues on the clients that are most appropriate for the requested order. I realize it's a long way to go, but I guess one should just take one step at the time =)
Hope someone has the time to look into this. Thanks in advance.
The biggest problem I see with this code is that you have far too much going on right away to easily debug your problem. Threading can get extremely complicated because of how non-linear the logic becomes. Especially when you have to worry about synchronizing with locks.
The reason you are seeing clients blocking on each other is because of the way you are using your serverSocketLock in your listenToClient() loop in the server. To be honest this isn't exactly your problem right now with your code, but it became the problem when I started to debug it and turned the sockets into blocking sockets. If you are putting each connection into its own thread and reading from them, then there is no reason to use a global server lock here. They can all read from their own sockets at the same time, which is the purpose of the thread.
Here is my recommendation to you:
Get rid of all the locks and extra threads that you don't need, and start from the beginning
Have the clients connect as you do, and put them in their thread as you do. And simply have them send data every second. Verify that you can get more than one client connecting and sending, and that your server is looping and receiving. Once you have this part working, you can move on to the next part.
Right now you have your sockets set to non-blocking. This is causing them all to spin really fast over their loops when data is not ready. Since you are threading, you should set them to block. Then the reader threads will simply sit and wait for data and respond immediately.
Locks are used when threads will be accessing shared resources. You obviously need to for any time a thread will try and modify a server attribute like a list or a value. But not when they are working on their own private sockets.
The event you are using to trigger your readers doesn't seem necessary here. You have received the client, and you start the thread afterwards. So it is ready to go.
In a nutshell...simplify and test one bit at a time. When its working, add more. There are too many threads and locks right now.
Here is a simplified example of your listenToClient method:
def listenToClient(self, c):
while self.server_running:
try:
recvd_data = c.client.recv(self.size)
print "received:", c, recvd_data
if recvd_data == "" or recvd_data == "close\n":
print "Client " + str(c.address) + (" disconnected...")
return
print recvd_data
except socket.error, (value, message):
if value == 35:
continue
else:
print "Error:", value, message
Backup your work, then toss it - partially.
You need to implement your program in pieces, and test each piece as you go. First, tackle the input part of your program. Don't worry about how to broadcast the input you received. Instead worry that you are able to successfully and repeatedly receive input over your socket. So far - so good.
Now, I assume you would like to react to this input by broadcasting to the other attached clients. Well too bad, you can't do that yet! Because, I left one minor detail out of the paragraph above. You have to design a PROTOCOL.
What is a protocol? It's a set of rules for communication. How does your server know when the client had finished sending it's data? Is it terminated by some special character? Or perhaps you encode the size of the message to be sent as the first byte or two of the message.
This is turning out to be a lot of work, isn't it? :-)
What's a simple protocol. A line-oriented protocol is simple. Read 1 character at a time until you get to the end of record terminator - '\n'. So, clients would send records like this to your server --
HELO\n
MSG DAVE Where Are Your Kids?\n
So, assuming you have this simple protocol designed, implement it. For now, DON'T WORRY ABOUT THE MULTITHREADING STUFF! Just worry about making it work.
Your current protocol is to read 1024 bytes. Which may not be bad, just make sure you send 1024 byte messages from the client.
Once you have the protocol stuff setup, move on to reacting to the input. But for now you need something that will read input. Once that is done, we can worry about doing something with it.
jdi is right, you have too much program to work with. Pieces are easier to fix.
I have a main thread that waits for connection. It spawns client threads that will echo the response from the client (telnet in this case). But say that I want to close down all sockets and all threads after some time, like after 1 connection.
How would I do it? If I do clientSocket.close() from the main thread, it won't stop doing the recv. It will only stop if I first send something through telnet, then it will fail doing further sends and recvs.
My code looks like this:
# Echo server program
import socket
from threading import Thread
import time
class ClientThread(Thread):
def __init__(self, clientSocket):
Thread.__init__(self)
self.clientSocket = clientSocket
def run(self):
while 1:
try:
# It will hang here, even if I do close on the socket
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
except:
break
self.clientSocket.close()
HOST = ''
PORT = 6000
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serverSocket.bind((HOST, PORT))
serverSocket.listen(1)
clientSocket, addr = serverSocket.accept()
print 'Got a new connection from: ', addr
clientThread = ClientThread(clientSocket)
clientThread.start()
time.sleep(1)
# This won't make the recv in the clientThread to stop immediately,
# nor will it generate an exception
clientSocket.close()
I know this is an old thread and that Samuel probably fixed his issue a long time ago. However, I had the same problem and came across this post while google'ing. Found a solution and think it is worthwhile to add.
You can use the shutdown method on the socket class. It can prevent further sends, receives or both.
socket.shutdown(socket.SHUT_WR)
The above prevents future sends, as an example.
See Python docs for more info.
I don't know if it's possible to do what you're asking, but it shouldn't be necessary. Just don't read from the socket if there is nothing to read; use select.select to check the socket for data.
change:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
to something more like this:
r, _, _ = select.select([self.clientSocket], [], [])
if r:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
EDIT: If you want to guard against the possibility that the socket has been closed, catch socket.error.
do_read = False
try:
r, _, _ = select.select([self.clientSocket], [], [])
do_read = bool(r)
except socket.error:
pass
if do_read:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
I found a solution using timeouts. That will interrupt the recv (actually before the timeout has expired which is nice):
# Echo server program
import socket
from threading import Thread
import time
class ClientThread(Thread):
def __init__(self, clientSocke):
Thread.__init__(self)
self.clientSocket = clientSocket
def run(self):
while 1:
try:
data = self.clientSocket.recv(1024)
print "Got data: ", data
self.clientSocket.send(data)
except socket.timeout:
# If it was a timeout, we want to continue with recv
continue
except:
break
self.clientSocket.close()
HOST = ''
PORT = 6000
serverSocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
serverSocket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
serverSocket.bind((HOST, PORT))
serverSocket.listen(1)
clientSocket, addr = serverSocket.accept()
clientSocket.settimeout(1)
print 'Got a new connection from: ', addr
clientThread = ClientThread(clientSocket)
clientThread.start()
# Close it down immediatly
clientSocket.close()
I must apologize for the comments below. The earlier comment by #Matt Anderson works. I had made a mistake when trying it out which led to my post below.
Using timeout is not a very good solution. It may seem that waking up for an instant and then going back to sleep is no big deal, but I have seen it greatly affect the performance of an application. You have an operation that for the most part wants to block until data is available and thus sleep forever. However, if you want to abort for some reason, like shutting down your application, then the trick is how to get out. For sockets, you can use select and listen on two sockets. Your primary one, and a special shutdown one. Creating the shutdown one though is a bit of a pain. You have to create it. You have to get the listening socket to accept it. You have to keep track of both ends of this pipe. I have the same issue with the Synchronized Queue class. There however, you can at least insert a dummy object into the queue to wake up the get(). This requires that the dummy object not look like your normal data though. I sometimes wish Python had something like the Windows API WaitForMultipleObjects.
How can I make a simple server(simple as in accepting a connection and print to terminal whatever is received) accept connection from multiple ports or a port range?
Do I have to use multiple threads, one for each bind call. Or is there another solution?
The simple server can look something like this.
def server():
import sys, os, socket
port = 11116
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
try:
listening_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listening_socket.bind((host, port))
listening_socket.listen(backlog)
except socket.error, (value, message):
if listening_socket:
listening_socket.close()
print 'Could not open socket: ' + message
sys.exit(1)
while True:
accepted_socket, adress = listening_socket.accept()
data = accepted_socket.recv(buf_size)
if data:
accepted_socket.send('Hello, and goodbye.')
accepted_socket.close()
server()
EDIT:
This is an example of how it can be done. Thanks everyone.
import socket, select
def server():
import sys, os, socket
port_wan = 11111
port_mob = 11112
port_sat = 11113
sock_lst = []
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
try:
for item in port_wan, port_mob, port_sat:
sock_lst.append(socket.socket(socket.AF_INET, socket.SOCK_STREAM))
sock_lst[-1].setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
sock_lst[-1].bind((host, item))
sock_lst[-1].listen(backlog)
except socket.error, (value, message):
if sock_lst[-1]:
sock_lst[-1].close()
sock_lst = sock_lst[:-1]
print 'Could not open socket: ' + message
sys.exit(1)
while True:
read, write, error = select.select(sock_lst,[],[])
for r in read:
for item in sock_lst:
if r == item:
accepted_socket, adress = item.accept()
print 'We have a connection with ', adress
data = accepted_socket.recv(buf_size)
if data:
print data
accepted_socket.send('Hello, and goodbye.')
accepted_socket.close()
server()
I'm not a python guy, but the function you are interested in is "select". This will allow you to watch multiple sockets and breaks out when activity occurs on any one of them.
Here's a python example that uses select.
Since Python's got so much overhead, multithreaded apps are a big point of debate. Then there's the whole blocking-operation-GIL issue too. Luckily, the Python motto of "If it seems like a big issue, someone's probably already come up with a solution (or several!)" holds true here. My favorite solution tends to be the microthread model, specifically gevent.
Gevent is an event-driven single-thread concurrency library that handles most issues for you out of the box via monkey-patching. gevent.monkey.patch_socket() is a function that replaces the normal socket calls with non-blocking variants, polling and sleeping to allow the switch to other greenlets as need be. If you want more control, or it's not cutting it for you, you can easily manage the switching with select and gevent's cooperative yield.
Here's a simple example.
import gevent
import socket
import gevent.monkey; gevent.monkey.patch_socket()
ALL_PORTS=[i for i in xrange(1024, 2048)]
MY_ADDRESS = "127.0.0.1"
def init_server_sock(port):
try:
s=socket.socket()
s.setblocking(0)
s.bind((MY_ADDRESS, port))
s.listen(5)
return s
except Exception, e:
print "Exception creating socket at port %i: %s" % (port, str(e))
return False
def interact(port, sock):
while 1:
try:
csock, addr = sock.accept()
except:
continue
data = ""
while not data:
try:
data=csock.recv(1024)
print data
except:
gevent.sleep(0) #this is the cooperative yield
csock.send("Port %i got your message!" % port)
csock.close()
gevent.sleep(0)
def main():
socks = {p:init_server_sock(p) for p in ALL_PORTS}
greenlets = []
for k,v in socks.items():
if not v:
socks.pop(k)
else:
greenlets.append(gevent.spawn(interact, k, v))
#now we've got our sockets, let's start accepting
gevent.joinall(greenlets)
That would be a super-simple, completely untested server serving plain text We got your message! on ports 1024-2048. Involving select is a little harder; you'd have to have a manager greenlet which calls select and then starts up the active ones; but that's not massively hard to implement.
Hope this helps! The nice part of the greenlet-based philosophy is that the select call is actually part of their hub module, as I recall, which will allow you to create a much more scalable and complex server more easily. It's pretty efficient too; there are a couple benchmarks floating around.
If you really wanted to be lazy (from a programmer standpoint, not an evaluation standpoint), you could set a timeout on your blocking read and just loop through all your sockets; if a timeout occurs, there wasn't any data available. Functionally, this is similar to what the select is doing, but it is taking that control away from the OS and putting it in your application.
Of course, this implies that as your sleep time gets smaller, your program will approach 100% CPU usage, so you wouldn't use it on a production app. It's fine for a toy though.
It would go something like this: (not tested)
def server():
import sys, os, socket
port = 11116
host = ''
backlog = 5 # Number of clients on wait.
buf_size = 1024
NUM_SOCKETS = 10
START_PORT = 2000
try:
socket.setdefaulttimeout(0.5) # raise a socket.timeout error after a half second
listening_sockets = []
for i in range(NUM_SOCKETS):
listening_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
listening_socket.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
listening_socket.bind((host, START_PORT + i))
listening_socket.listen(backlog)
listening_sockets.append(listening_socket)
except socket.error, (value, message):
if listening_socket:
listening_socket.close()
print 'Could not open socket: ' + message
sys.exit(1)
while True:
for sock in listening_sockets:
try:
accepted_socket, adress = sock_socket.accept()
data = sock.recv(buf_size)
if data:
sock_socket.send('Hello, and goodbye.')
sock.close()
except socket.timeout:
pass
server()