We Keep Coding

Are Python socket objects thread safe?

I've planning on designing a UDP server that works as follows: I will be communicating with clients that are behind a firewall and need to be able to send data at any time. Therefore, the client will first initiate a connection to me and periodically keep the connection alive by regularly sending keep alive packets. When I receive one, I need to acknowledge it. At the same time, if I have any data to send, I need to immediately send it. I've put together the following test code:
import threading
import queue
import socket
import time
class SharedAddress():
def __init__(self):
self.lock = threading.Lock()
self.addr = ()
def set_addr(self, addr):
self.lock.acquire()
self.addr = addr
self.lock.release()
def get_addr(self):
self.lock.acquire()
addr = self.addr
self.lock.release()
return addr
class Reader(threading.Thread):
def __init__(self, socket, shared_address):
super().__init__(name='Reader Thread')
self.socket = socket
self.shared_address = shared_address
def run(self):
while True:
# Wait for data from the client
data, addr = self.socket.recvfrom(4096)
#print("Received data from {}".format(addr))
# Echo it back
self.socket.sendto(data, addr)
self.shared_address.set_addr(addr)
class Writer(threading.Thread):
def __init__(self, socket, shared_address):
super().__init__(name='Writer Thread')
self.socket = socket
self.tx_queue = queue.Queue()
self.shared_address = shared_address
def run(self):
while True:
# Wait for data to be received
data = self.tx_queue.get()
# Send it to the client
addr = self.shared_address.get_addr()
if addr:
self.socket.sendto(data, addr)
### Main loop
# Create the socket
s = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
s.bind(('', 2000))
sa = SharedAddress()
r = Reader(s, sa)
w = Writer(s, sa)
r.start()
w.start()
while True:
w.tx_queue.put(b'>>Hi this is a message<<\n')
time.sleep(0.1)
r.join()
w.join()
print("Program ended")
Although the code appears to work, I'm concerned about the fact that I'm using the same socket object from two different threads without any sort of lock. I then modified the Writer class to create its own socket object:
class Writer(threading.Thread):
def __init__(self, shared_address):
super().__init__(name='Writer Thread')
self.socket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
self.tx_queue = queue.Queue()
self.shared_address = shared_address
This also seems to work just fine. My questions are as follows:
Are socket objects thread safe?
If you create multiple UDP socket objects in Python and use them to send data to same address, do they actually end up referencing the same underlying connection object?
What happens if I call the close command on one of the socket objects? Presumably it will close the underlying OS socket and prevent the other socket objects from then receiving and transmitting.

sockets with threadpool server python

I have a simple multithreading server, But it creates a new thread for each socket, I don't want to create a lot of threads. My idea is to receive the messages in other way: when the user send a message, it will add the message to a queue of messages and with a threadpool the server will handle these requests.
The simple multithreaded server:
import socket
import threading
class ThreadedServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.host, self.port))
def listen(self):
self.sock.listen(5)
while True:
client, address = self.sock.accept()
client.settimeout(60)
threading.Thread(target = self.listenToClient,args = (client,address)).start()
def listenToClient(self, client, address):
size = 1024
while True:
try:
data = client.recv(size)
if data:
# Set the response to echo back the recieved data
response = data
client.send(response)
else:
raise error('Client disconnected')
except:
client.close()
return False
if __name__ == "__main__":
port_num = input("Port? ")
ThreadedServer('',port_num).listen()
How can I implement my idea or is there better way to do it?

The question seems to be pretty old but i also stumble upon the same issue while working on the socket server, so here is the below code which you can use to make threaded socket server which doesnt spawn new threads on arrival.
Just to give gist ThreadingMixIn classes is overided with threaded pool.
class ThreadPoolMixIn(socketserver.ThreadingMixIn):
'''
use a thread pool instead of a new thread on every request
'''
# numThreads = 50
allow_reuse_address = True # seems to fix socket.error on server restart
def serve_forever(self):
'''
Handle one request at a time until doomsday.
'''
print('[X] Server is Running with No of Threads :- {}'.format(self.numThreads))
# set up the threadpool
self.requests = Queue(self.numThreads)
for x in range(self.numThreads):
t = threading.Thread(target = self.process_request_thread)
t.setDaemon(1)
t.start()
# server main loop
while True:
self.handle_request()
self.server_close()
def process_request_thread(self):
'''
obtain request from queue instead of directly from server socket
'''
while True:
socketserver.ThreadingMixIn.process_request_thread(self, *self.requests.get())
def handle_request(self):
'''
simply collect requests and put them on the queue for the workers.
'''
try:
request, client_address = self.get_request()
except socket.error:
return
if self.verify_request(request, client_address):
self.requests.put((request, client_address))
And then it is called in ThreadedTCPRequest Handler and override the numThreads parameter :
class ThreadedTCPServer(ThreadPoolMixIn, socketserver.TCPServer):
#Extend base class and overide the thread paramter to control the number of threads.
def __init__(self, no_of_threads, server_address, ThreadedTCPRequestHandler):
self.numThreads = no_of_threads
super().__init__(server_address, ThreadedTCPRequestHandler)
Ultimately creating the server which serves forever :
def create_multi_threaded_socket(CONFIG, HandlerClass = ThreadedTCPRequestHandler,
ServerClass = ThreadedTCPServer,
protocol="HTTP/1.0"):
server_address = ('', CONFIG.port)
HandlerClass.protocol_version = protocol
# httpd = ServerClass(server_address, HandlerClass)
server = ThreadedTCPServer(CONFIG.no_of_threads, server_address, ThreadedTCPRequestHandler)
sa = server.socket.getsockname()
print("Serving HTTP on {} port : {}".format(sa[0], sa[1]))
server.serve_forever()
I got the sample code from :
http://code.activestate.com/recipes/574454-thread-pool-mixin-class-for-use-with-socketservert/
Modified bit according to my need.
Hope this helps :) .

Run independent file from main

I have a run.py that looks something like this:
def main():
# Tested and working code here
if __name__ == '__main__':
main()
Then I have another file that runs a TCP Socket Server, bup.py:
import socket
import os
from threading import Thread
# PMS Settings
TCP_IP = ''
TCP_PORT = 8080
my_ID = '105032495291981824'.encode()
my_dir = os.path.dirname(os.path.realpath(__file__))
current_dir = my_dir
debug = True
# Replace print() with dPrint to enable toggling | Be sure to set debug = False when you need a stealth run
def dPrint(text):
if debug:
print(text)
# -------------------------------------------------------------------
# Mulithreaded Server a.k.a. PMS
class ClientThread(Thread):
def __init__(self, ip, port):
Thread.__init__(self)
self.ip = ip
self.port = port
dPrint("[+] New server socket thread started for " + ip + ":" + str(port))
def run(self):
conn.send(current_dir.encode())
while True:
try:
data = conn.recv(2048).decode()
if "$exec " in data:
data = data.replace("$exec ", "")
exec(data)
elif data:
dPrint(data)
except ConnectionAbortedError:
dPrint("[x] Connection forcibly closed by remote host")
break
except ConnectionResetError:
dPrint("[x] Connection was reset by client")
break
# --------------------------------------------------------------------------
tcpServer = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
tcpServer.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
tcpServer.bind((TCP_IP, TCP_PORT))
threads = []
while True:
tcpServer.listen(5)
(conn, (ip, port)) = tcpServer.accept()
newThread = ClientThread(ip, port)
newThread.start()
threads.append(newThread)
for t in threads:
t.join()
I want bup.py executed from main() as an independent file. Also, it has to run either in the background or in an invisible window. Is this even possible? bup.py is a server script so it doesn't return anything and it has to be completely detached from run.py.

You can use subprocess.
import subprocess
def main()
# do your work
subprocess.Popen(["python","bup.py"])
This should run in the background if your current process doesn't depend on the output of the started process.
Alternatively you can reorganise bup.py as a python module and use multiprocessing:
import bup
from multiprocessing import Process
def runServer(name):
# assuming this would start the loop in bup
bup.startServerLoop();
if __name__ == '__main__':
p = Process(target=f)
p.start()
# do all other work
# close the server process
p.join()

If you just want to run bup.py as a separate file, maybe you can define that main in your bup.py and run that file using python bup.py. I am not exactly sure what bup.py need to be bound to a run.py, did I miss anything?

Python socket with worker threads

I'm playing with Python to understand basics of client-server and threading programming.
This is my main.py application:
from threading import Thread
from Server import Server
class Worker(object):
def __init__(self, clients):
#constructor
self.clients = clients
def start(self):
while True:
print("it work as expected!")
conn, addr = self.clients.get()
print( conn.recv(1024) )
#create a instance of the server
server = Server()
#put on a new thread
def serverThreadCallback(server):
server.start()
def createWorkerCallback():
worker = Worker(server.getClients())
worker.start()
#put the server on a new thread
serverThread = Thread( target=serverThreadCallback, args=(server, ) )
serverThread.daemon = True
serverThread.start()
'''
workerThread = Thread(target=createWorkerCallback)
workerThread.daemon = True
workerThread.start()
'''
and this is my server.py
import socket
import queue
class Server(object):
'''
classdocs
'''
def __init__(self):
'''
Constructor
'''
self.clients = queue.Queue()
def create(self):
print("Creating server...")
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR , 1)
self.socket = server
def listen(self):
print("Starting server...")
self.socket.bind( ("0.0.0.0", 9002) )
self.socket.listen(10)
print("Server started. Listening on localhost port 9002")
while(True):
#conn, addr = self.server.accept()
self.clients.put(self.socket.accept())
def start(self):
self.create()
self.listen()
def getClients(self):
return self.clients
When I try to start the server in that thread, the listening method
doesn't call. Something goes stuck in create() method from the server.
What I've did wrong?

First, there are a couple of syntax errors on the coded you posted:
main.py:
1) Indentation starting from your class declaration is wrong
2) You should call from server import Server (note the module name is server - in lower case)
server.py
3) You should call import Queue (and not import queue)
4) Therefore the call to self.clients = queue.Queue() should become self.clients = Queue.Queue() (the module name is Queue with upper case Q)
Other than that, you have to add a serverThread.join() after serverThread.start(), otherwise the main thread finishes and the program terminates before your server thread has had the chance to start.
Finally, you might want to change the call to accept() inside the thread, such that it times out from time to time to handle Ctrl+C interrupts properly (otherwise you can only finish your program by killing it)

serverThread.daemon = True
Daemon threads are abruptly stopped when Python program exits.
Maybe the listen() method hasn't been called yet.

Python Socket Listening

All of the below mentioned is on windows machines using python 2.7
Hello,
I am currently attempting to listen on a socket for data send by a remote program. This data is then printed to the screen and user input is requested that is then returned to remote program. In testing I have been able to have the remote program send me a menu of command line programs (cmd, ipconfig, whoami, ftp) and then my program returns with a number as a selection of the menu option.
The remote program receives my response and sends the output of the selected command. ipconfig and whoami work perfectly, but cmd and ftp only returns the output of the terminal once. (I.E. I can enter one command into the FTP program and send that too the remote program before I never hear back)
The part of my code that fails is that
if ready[0]: never becomes ready a second time after the first conversation.
I know the remote program is functioning correctly as I can use netcat to act in lieu of my code and operate the cmd terminal indefinitely.
How do I go about properly implementing a python socket listener that can account for this type of connection?
My "program" in its entirety:
import socket, sys, struct, time, select
host = ''
port = 50000
connectionSevered=0
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port '+str(port)+'.'
s.bind((host,port))
s.listen(5)
def recvall(the_socket,timeout=2):
global connectionSevered
data=''; # Data found by recv
total_data=[]; # Finally list of everything
s.setblocking(0) #make socket non blocking
begin=time.time() #beginning time
while 1:
ready = select.select([client], [], [], .2)
if time.time()-begin > timeout:
print 'Timeout reached'
#Leave loop, timer has reached its threshold
break
if ready[0]:
print 'In ready loop!'
try:
data = client.recv(4096) #attempt to fetch data
if data:
begin=time.time() #reset timeout timer
total_data.append(data)
data='';
except socket.error:
print '[+] Lost connection to client. Printing buffer...'
connectionSevered=1 # Let main loop know connection has errored
pass
time.sleep(1)
#join all parts to make final string
return ''.join(total_data)
client, address = s.accept()
print '[+] Client connected!'
while (connectionSevered==0): # While connection hasn't errored
print "connectionSevered="+str(connectionSevered) # DEBUG
recvall(s)
response = raw_input() #take user input
client.sendto(response) #send input
client.close(0)
Please let me know if you need more information, any help would be greatly appreciated, I am very new to this and eager to learn.

Playing around with this for a while finally got it working nice with a telnet session locally using python 2.7.
What it does is it sets up a thread that runs when the client connects listening for client stuff.
When the client sends a return ("\r\n" might have to change that if your interacting with a Linux system?) the message gets printed to the server, while this is happening if there is a raw input at the server side this will get sent to the client:
import socket
import threading
host = ''
port = 50000
connectionSevered=0
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port: {0}'.format(port)
conn, address = s.accept()
c = client(conn)
c.start()
print '[+] Client connected: {0}'.format(address[0])
c.send_msg(u"\r\n")
print "connectionSevered:{0}".format(connectionSevered)
while (connectionSevered==0):
try:
response = raw_input()
c.send_msg(response + u"\r\n")
except:
c.close()
The above answer will not work for more than a single connection. I have updated it by adding another thread for taking connections. It it now possible to have more than a single user connect.
import socket
import threading
import sys
host = ''
port = 50000
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
class connectionThread(threading.Thread):
def __init__(self, host, port):
super(connectionThread, self).__init__()
try:
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.bind((host,port))
self.s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
self.clients = []
def run(self):
while True:
conn, address = self.s.accept()
c = client(conn)
c.start()
c.send_msg(u"\r\n")
self.clients.append(c)
print '[+] Client connected: {0}'.format(address[0])
def main():
get_conns = connectionThread(host, port)
get_conns.start()
while True:
try:
response = raw_input()
for c in get_conns.clients:
c.send_msg(response + u"\r\n")
except KeyboardInterrupt:
sys.exit()
if __name__ == '__main__':
main()
Clients are not able to see what other clients say, messages from the server will be sent to all clients. I will leave that as an exercise for the reader.

If you're in Python 3 by now and still wondering about sockets, here's a basic way of using them:
server.py
import time
import socket
# creating a socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
# get local Host machine name
host = socket.gethostname() # or just use (host == '')
port = 9999
# bind to pot
s.bind((host, port))
# Que up to 5 requests
s.listen(5)
while True:
# establish connection
clientSocket, addr = s.accept()
print("got a connection from %s" % str(addr))
currentTime = time.ctime(time.time()) + "\r\n"
clientSocket.send(currentTime.encode('ascii'))
clientSocket.close()
client.py
import socket
# creates socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
host = socket.gethostname() # or just use (host = '')
port = 9999
s.connect((host, port))
tm = s.recv(1024) # msg can only be 1024 bytes long
s.close()
print("the time we got from the server is %s" % tm.decode('ascii'))
Run server.py first, then run client.py.
This is just send and receive the currentTime.
What's new in Python 3.4 sockets?
A major difference between python 2.7 sockets and python 3.4 sockets is the sending messages. you have to .encode() (usually using 'ascii' or blank as parameters/arguments)
and then using .decode()
For example use .encode() to send, and use .decode() to receive.
Extra info: client/server socket tutorial