We Keep Coding

Running an echo server as a daemon process for testing in python

I want to automate testing of an instrument and wrote a little server program to imitate the instrument which will send back the command except when it receives a special command "*IDN?". When I ran the echo server directly in its own script, and then run a client script separately, everything works great, and I am getting back the expected results. Now I wanted to run the server directly from the testing script. So I thought I would start it using multiprocessing. But the problem seems to be when the server socket gets to the s.accept() line it just waits there and never returns. So how do I accomplish automated testing if I cannot run this server in the same code as the test function?
import socket
import multiprocessing as mp
import time,sys
HOST = '127.0.0.1' # Standard loopback interface address (localhost),
PORT = 65432 # Port to listen on (non-privileged ports are > 1023),
FTP_PORT = 63217 # Port for ftp testing, change to 21 for device
def handle_connection(conn,addr):
with conn:
conn.send('Connected by', addr)
print("Got connection")
data = conn.recv(1024)
if not data:
return 'Nodata'
elif (data == b'*IDN?\n'):
print('SONY/TEK,AWG520,0,SCPI:95.0 OS:3.0 USR:4.0\n')
conn.sendall(b'SONY/TEK,AWG520,0,SCPI:95.0 OS:3.0 USR:4.0\n')
return 'IDN'
else:
conn.sendall(data)
return 'Data'
def echo_server(c_conn,host=HOST,port=PORT):
# this server simulates the AWG command protocol, simply echoing the command back except for IDN?
p = mp.current_process()
print('Starting echo server:', p.name, p.pid)
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.bind((host, port))
s.listen()
try:
while True:
print("Waiting for connection...")
c_conn.send('waiting for connection...')
conn, addr = s.accept()
handle_connection(conn,addr)
c_conn.send('serving client...')
finally:
conn.close()
c_conn.send('done')
time.sleep(2)
print('Exiting echo server:', p.name, p.pid)
sys.stdout.flush()
def test_echo_server():
print("entering client part")
with socket.socket(socket.AF_INET,socket.SOCK_STREAM) as mysock:
mysock.connect((HOST,PORT))
mysock.sendall('test\n'.encode())
data = mysock.recv(1024)
print('received:',repr(data))
if __name__ == '__main__':
parent_conn, child_conn = mp.Pipe()
echo_demon = mp.Process(name='echo', target=echo_server(child_conn, ))
echo_demon.daemon = True
echo_demon.start()
time.sleep(1)
echo_demon.join(1)
test_echo_server()
if parent_conn.poll(1):
print(parent_conn.recv())
else:
print('Waiting for echo server')

I managed to solve my own question using some code snippets I found in the book "Getting started with Python" by Romano, Baka, and Phillips. here is the code for the server:
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen()
try:
while True:
print("Waiting for connection...")
client, addr = s.accept()
with client:
data = client.recv(1024)
if not data:
break
elif (data == b'*IDN?\n'):
client.sendall(b'SONY/TEK,AWG520,0,SCPI:95.0 OS:3.0 USR:4.0\n')
else:
client.sendall(data)
finally:
time.sleep(1)
print('Exiting echo server:')
and here is the code for the testing file which runs this server in a separate process, and a couple of simple tests:
#pytest.fixture(scope="session")
def awgserver():
print("loading server")
p = subprocess.Popen(["python3", "server.py"])
time.sleep(1)
yield p
p.terminate()
#pytest.fixture
def clientsocket(request):
print("entering client part")
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as mysock:
mysock.connect((HOST, PORT))
yield mysock
mysock.close()
#pytest.mark.run_this
def test_echo(awgserver, clientsocket):
clientsocket.send(b"*IDN?\n")
#assert clientsocket.recv(1024) == b"SONY/TEK,AWG520,0,SCPI:95.0 OS:3.0 USR:4.0\n"
assert clientsocket.recv(10) == b"TEK" # deliberately make test fail
#pytest.mark.run_this
def test_echo2(awgserver, clientsocket):
clientsocket.send(b"def")
assert clientsocket.recv(3) == b"def"
Set HOST to loopback IP, and PORT to > 1024

Multithreaded socket gets closed before remote script execution over

Using Python 2.6.6, I am not able to get complete data from client script as suspecting the socket gets closed before the client script execution is over, while the client script working fine if I manually trigger the script on the remote server.
What script does -
Trigger SCRIPT
Only to transfer the client script [agent.py] and trigger remotely using multithread.
Client SCRIPT
agent.py would be transferred to target servers [1000+]
running on (1000+) remote servers [linux] to collect data and return
a list of dictionary from each remote machine back to Server
Server SCRIPT
Receives the data and convert to CSV
Client Script - agent.py
s = socket.socket() # Create a socket object
host = "<SERVER_HOST>" # server ip
port = 12345 # Reserve a port for your service.
try:
s.connect((host, port))
instlist = []
infoCollect = processInfo() #get the info in dictionary object
instlist.append(infoCollect)
data_string = str(instlist)
s.sendall(data_string)
s.send(",")
s.shutdown(socket.SHUT_WR)
print('Sent ',(instlist))
s.close()
Server Script [DataCollector]:
class ThreadedServer(object):
def __init__(self, host, port):
self.host = host
self.port = port
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
self.sock.bind((self.host, self.port))
def listen(self):
self.sock.listen(5)
print 'Started Listening....'
while True:
client, address = self.sock.accept()
print 'Got connection from', address
client.settimeout(60)
threading.Thread(target = self.listenToClient,args = (client,address)).start()
def listenToClient(self, client, address):
size = 1024
while True:
try:
l = client.recv(size)
while (l):
print ("Receiving...%s" % l)
f.write(l)
l = client.recv(size)
print "Instance Details Recieved"
client.send('Thank you for connecting')
except:
client.close()
return False
if __name__ == "__main__":
while True:
port_num = input("Port? ")
try:
port_num = int(port_num)
break
except ValueError:
pass
f = open(array_data,'wb')
ThreadedServer('',port_num).listen()
Trigger Script - This script's purpose is to initiate this all process and place the Client Script [agentScript] on 1000+ servers and execute remotely
cmd = "python agent.py"
takeLock = threading.Lock()
def worker(host):
ssh = paramiko.SSHClient() # Initiate SSH Object
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
try:
ssh.connect(host, username='user', password='pass' )
sftp = ssh.open_sftp() #sedning files to parellaly to all listed servers
sftp.put(__file__, 'excludelist')
sftp.close()
stdin, stdout, stderr = ssh.exec_command(cmd)
while not stdout.channel.exit_status_ready():
# Print Only data when available
if stdout.channel.recv_ready():
alldata = stdout.channel.recv(1024)
prevdata = b"1"
while prevdata:
prevdata = stdout.channel.recv(1024)
alldata += prevdata
print alldata
return (str(alldata))
ssh.close()
except socket.error, v:
print "Connection Refused"
def main():
with open('IPlist.txt') as ip:
hosts = ip.read().splitlines()
threads = []
for h in hosts:
t = threading.Thread(target=worker, args=(h,))
t.start()
threads.append(t)
for t in threads:
t.join()
if __name__ == "__main__":
main()
NOTE: Client Script [agent.py] takes 1-5 sec depends on the server's
configuration to generate output and return value
PROBLEM - Incomplete Data populating at server side like -
When I trigger script manually on remote machine to test it send complete dictionary object like [{commonServerData},{a,b,c,d},{Engine02Data},{tempData,tempData02}] and so server recives the same data
When agent.py is invoked using trigger script it sends incomplete data like-
[{commonServerData},{a,b,c,d}] for all 1000+ srervers.
I am suspecting something wrong with my Trigger Script which does close the socket session of the remote server even before the agent.py gets completed.

Removed extra comma s.send(',') and added it to s.sendall(data_string + ',') from the client script does the expected work -
s = socket.socket() # Create a socket object
host = "<SERVER_HOST>" # server ip
port = 12345 # Reserve a port for your service.
try:
s.connect((host, port))
instlist = []
infoCollect = processInfo() #get the info in dictionary object
instlist.append(infoCollect)
data_string = str(instlist)
s.sendall(data_string + ',')
s.shutdown(socket.SHUT_WR)
print('Sent ',(instlist))
s.close()

multiple clients cannot listen and write at the same time

I'm writing a very basic chat room in python. Clients connect and any message from a client is relayed to all clients. The problem I'm having is getting the client to listen and send messages at the same time. It seems to only do either one. I've set up a separate listening client and confirmed that the message is received but the listening server cannot send anything.
Currently the client has to send data before getting a response from the server, but I want clients to be able to receive data before sending - otherwise the chat room won't work. I attempted using clientsock.settimeout() and then use recv but it did not solve the issue as it did not move past the input part.
server.py
#!/usr/bin/python
#socket server using threads
import socket, sys, threading
from _thread import *
HOST = 'localhost'
PORT = 2222
lock = threading.Lock()
all_clients = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print ("Socket created")
#bind socket to local host and port
try:
s.bind((HOST, PORT))
except socket.error as msg:
print ("Bind failed. Error code: " + str(msg[0]) + ' Message ' + msg[1])
sys.exit(0)
print ("Socket bind complete")
#Start listening on socket
s.listen(5)
print ("Socket now listening")
#function for handling connections. This will be used to create threads
def clientthread(conn):
#sending message to connected client
conn.send("Welcome to the server. Type something and hit enter\n".encode('utf-8'))
#infinite loop so that function does not terminate and thread does not end
while True:
#receiving data from client
data = conn.recv(1024)
reply = "OK..." + str(data, "utf-8")
if not data:
break
with lock:
for c in all_clients:
c.sendall(reply.encode('utf-8'))
#came out of loop
conn.close()
#keep talking with the client
while 1:
#wait to accept a connection - blocking call
conn, addr = s.accept()
with lock:
all_clients.append(conn)
print ("Connected with " + addr[0] + ":" + str(addr[1]))
#start new thread takes 1st argument as a function name to be run, second
#is the tuple of arguments to the function
start_new_thread(clientthread ,(conn,))
s.close()
client.py
#!/usr/bin/python
import socket, sys
#client to transfer data
def main():
#create tcp stocket
clientsock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
#connect the socket to the server open port
server_address = ('localhost', 2222)
print ("connecting to %s port %s" % server_address)
clientsock.connect(server_address)
#receive data
data = clientsock.recv(1024)
print(str(data, "utf-8"))
while 1:
#send data
message = "sean: " + input()
clientsock.send(message.encode('utf-8'))
#look for the response
amount_received = 0
amount_expected = len(message)
while amount_received < amount_expected:
data = clientsock.recv(1024)
amount_received += len(data)
print ("received %s " % data)
print ("closing socket")
clientsock.close()
main()
new_client.py
#!/usr/bin/python
import socket, sys
from threading import Thread
#client for chat room
def send_msg(sock):
while True:
data = input()
sock.send(data.encode('utf-8'))
def recv_msg(sock):
while True:
stuff = sock.recv(1024)
sock.send(stuff)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('localhost', 2222)
sock.connect(server_address)
print("Connected to chat")
Thread(target=send_msg, args=(sock,)).start()
Thread(target=recv_msg, args=(sock,)).start()

Create two threads, one for receiving the other for sending. This is the simplest way to do.
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect("address")
def send_msg(sock):
while True:
data = sys.stdin.readline()
sock.send(data)
def recv_msg(sock):
while True:
data, addr = sock.recv(1024)
sys.stdout.write(data)
Thread(target=send_msg, args=(sock,)).start()
Thread(target=recv_msg, args=(sock,)).start()

Python Socket Listening

All of the below mentioned is on windows machines using python 2.7
Hello,
I am currently attempting to listen on a socket for data send by a remote program. This data is then printed to the screen and user input is requested that is then returned to remote program. In testing I have been able to have the remote program send me a menu of command line programs (cmd, ipconfig, whoami, ftp) and then my program returns with a number as a selection of the menu option.
The remote program receives my response and sends the output of the selected command. ipconfig and whoami work perfectly, but cmd and ftp only returns the output of the terminal once. (I.E. I can enter one command into the FTP program and send that too the remote program before I never hear back)
The part of my code that fails is that
if ready[0]: never becomes ready a second time after the first conversation.
I know the remote program is functioning correctly as I can use netcat to act in lieu of my code and operate the cmd terminal indefinitely.
How do I go about properly implementing a python socket listener that can account for this type of connection?
My "program" in its entirety:
import socket, sys, struct, time, select
host = ''
port = 50000
connectionSevered=0
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port '+str(port)+'.'
s.bind((host,port))
s.listen(5)
def recvall(the_socket,timeout=2):
global connectionSevered
data=''; # Data found by recv
total_data=[]; # Finally list of everything
s.setblocking(0) #make socket non blocking
begin=time.time() #beginning time
while 1:
ready = select.select([client], [], [], .2)
if time.time()-begin > timeout:
print 'Timeout reached'
#Leave loop, timer has reached its threshold
break
if ready[0]:
print 'In ready loop!'
try:
data = client.recv(4096) #attempt to fetch data
if data:
begin=time.time() #reset timeout timer
total_data.append(data)
data='';
except socket.error:
print '[+] Lost connection to client. Printing buffer...'
connectionSevered=1 # Let main loop know connection has errored
pass
time.sleep(1)
#join all parts to make final string
return ''.join(total_data)
client, address = s.accept()
print '[+] Client connected!'
while (connectionSevered==0): # While connection hasn't errored
print "connectionSevered="+str(connectionSevered) # DEBUG
recvall(s)
response = raw_input() #take user input
client.sendto(response) #send input
client.close(0)
Please let me know if you need more information, any help would be greatly appreciated, I am very new to this and eager to learn.

Playing around with this for a while finally got it working nice with a telnet session locally using python 2.7.
What it does is it sets up a thread that runs when the client connects listening for client stuff.
When the client sends a return ("\r\n" might have to change that if your interacting with a Linux system?) the message gets printed to the server, while this is happening if there is a raw input at the server side this will get sent to the client:
import socket
import threading
host = ''
port = 50000
connectionSevered=0
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
try:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
print '[+] Listening for connections on port: {0}'.format(port)
conn, address = s.accept()
c = client(conn)
c.start()
print '[+] Client connected: {0}'.format(address[0])
c.send_msg(u"\r\n")
print "connectionSevered:{0}".format(connectionSevered)
while (connectionSevered==0):
try:
response = raw_input()
c.send_msg(response + u"\r\n")
except:
c.close()
The above answer will not work for more than a single connection. I have updated it by adding another thread for taking connections. It it now possible to have more than a single user connect.
import socket
import threading
import sys
host = ''
port = 50000
class client(threading.Thread):
def __init__(self, conn):
super(client, self).__init__()
self.conn = conn
self.data = ""
def run(self):
while True:
self.data = self.data + self.conn.recv(1024)
if self.data.endswith(u"\r\n"):
print self.data
self.data = ""
def send_msg(self,msg):
self.conn.send(msg)
def close(self):
self.conn.close()
class connectionThread(threading.Thread):
def __init__(self, host, port):
super(connectionThread, self).__init__()
try:
self.s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
self.s.bind((host,port))
self.s.listen(5)
except socket.error:
print 'Failed to create socket'
sys.exit()
self.clients = []
def run(self):
while True:
conn, address = self.s.accept()
c = client(conn)
c.start()
c.send_msg(u"\r\n")
self.clients.append(c)
print '[+] Client connected: {0}'.format(address[0])
def main():
get_conns = connectionThread(host, port)
get_conns.start()
while True:
try:
response = raw_input()
for c in get_conns.clients:
c.send_msg(response + u"\r\n")
except KeyboardInterrupt:
sys.exit()
if __name__ == '__main__':
main()
Clients are not able to see what other clients say, messages from the server will be sent to all clients. I will leave that as an exercise for the reader.

If you're in Python 3 by now and still wondering about sockets, here's a basic way of using them:
server.py
import time
import socket
# creating a socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
# get local Host machine name
host = socket.gethostname() # or just use (host == '')
port = 9999
# bind to pot
s.bind((host, port))
# Que up to 5 requests
s.listen(5)
while True:
# establish connection
clientSocket, addr = s.accept()
print("got a connection from %s" % str(addr))
currentTime = time.ctime(time.time()) + "\r\n"
clientSocket.send(currentTime.encode('ascii'))
clientSocket.close()
client.py
import socket
# creates socket object
s = socket.socket(socket.AF_INET,
socket.SOCK_STREAM)
host = socket.gethostname() # or just use (host = '')
port = 9999
s.connect((host, port))
tm = s.recv(1024) # msg can only be 1024 bytes long
s.close()
print("the time we got from the server is %s" % tm.decode('ascii'))
Run server.py first, then run client.py.
This is just send and receive the currentTime.
What's new in Python 3.4 sockets?
A major difference between python 2.7 sockets and python 3.4 sockets is the sending messages. you have to .encode() (usually using 'ascii' or blank as parameters/arguments)
and then using .decode()
For example use .encode() to send, and use .decode() to receive.
Extra info: client/server socket tutorial

Python - Server and client problems

I'm trying to create a basic server and client script. The idea is that the client can connect to the server and execute commands. Kinda like SSH but very simple. Heres my server code:
import sys, os, socket
host = ''
port = 50103
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
print("Server started on port: ", port)
s.listen(1)
while (1):
conn, addr = s.accept()
print 'New connection from ', addr
try:
while True:
rc = conn.recv(2)
pipe = os.popen(rc)
rl = pipe.readlines()
fl = conn.makefile('w', 0)
fl.writelines(rl[:-1])
fl.close()
except IOError:
conn.close()
And here is my client:
import sys, socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = 'localhost'
port = input('Port: ')
s.connect((host, port))
while (1):
cmd = raw_input('$ ')
s.send(cmd)
file = s.makefile('r', 0)
sys.stdout.writelines(file.readlines())
file.close()
Here is my problem. I start the server and then run the client on the same machine. I enter the port and connect. Then I get the raw_input which is the '$'. If I type a command like 'ls' it just hangs on the client side. I have to exit the server for the client to receive the output of ls. By the way I am running Ubuntu Linux. Not sure if that matters.

When you makefile() on the socket and then use readlines() on it, it will continue until you reach an end of file, which in the socket case is that it closed from the other end.
Using makefile() in this case makes no sense to me, especially since you create it and close it after each command. Just use send() and recv() on both ends.
You probably also want to have some sort of actual "protocol" so the server tells the client "HERE COMES A RESPONSE" and "THIS IS THE END OF THE RESPONSE" so that the client knows. Otherwise it gets hard to know when to stop waiting for more response. :)
Update with an example that works:
server.py:
import sys, os, socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(('', 50500))
print("Server started")
s.listen(1)
while True:
print "Accepting"
conn, addr = s.accept()
print 'New connection from ', addr
while True:
try:
rc = conn.recv(1024)
print "Command", rc
if not rc.strip():
continue
if rc.strip() == 'END':
print "Close"
conn.send("**END**")
conn.close()
break
else:
conn.send("This is the result of command %s\n" % rc)
except Exception:
conn.close()
sys.exit()
client.py
import sys, os, socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('localhost', 50500))
while True:
cmd = raw_input('$ ')
s.send(cmd)
result = s.recv(1024)
print result
if result == "**END**":
print "Ending"
break

Well for one thing you're only connecting on the client once and on the server you're closing the socket after every read.
You should take a look at this example.
http://ilab.cs.byu.edu/python/socket/echoserver.html
You're doing quite a few things incorrectly.