I have a basic client-server script in Python using sockets. The server binds to a specific port and waits for a client connection. When a client connects, they are presented with a raw_input prompt that sends the entered commands to a subproc on the server and pipes the output back to the client.
Sometimes when I execute commands from the client, the output will hang and not present me with the raw_input prompt until I press the [enter] key.
At first I thought this might have been a buffer problem but it happens when I use commands with a small output, like 'clear' or 'ls', etc.
The client code:
import os, sys
import socket
from base64 import *
import time
try:
HOST = sys.argv[1]
PORT = int(sys.argv[2])
except IndexError:
print("You must specify a host IP address and port number!")
print("usage: ./handler_client.py 192.168.1.4 4444")
sys.exit()
socksize = 4096
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
server.connect((HOST, PORT))
print("[+] Connection established!")
print("[+] Type ':help' to view commands.")
except:
print("[!] Connection error!")
sys.exit(2)
while True:
data = server.recv(socksize)
cmd = raw_input(data)
server.sendall(str(cmd))
server.close()
Server code:
import os,sys
import socket
import time
from subprocess import Popen,PIPE,STDOUT,call
HOST = ''
PORT = 4444
socksize = 4096
activePID = []
conn = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
conn.bind((HOST, PORT))
conn.listen(5)
print("Listening on TCP port %s" % PORT)
def reaper():
while activePID:
pid,stat = os.waitpid(0, os.WNOHANG)
if not pid: break
activePID.remove(pid)
def handler(connection):
time.sleep(3)
while True:
cmd = connection.recv(socksize)
proc = Popen(cmd,
shell=True,
stdout=PIPE,
stderr=PIPE,
stdin=PIPE,
)
stdout, stderr = proc.communicate()
if cmd == ":killme":
connection.close()
sys.exit(0)
elif proc:
connection.send( stdout )
connection.send("\nshell => ")
connection.close()
os._exit(0)
def accept():
while 1:
global connection
connection, address = conn.accept()
print "[!] New connection!"
connection.send("\nshell => ")
reaper()
childPid = os.fork() # forks the incoming connection and sends to conn handler
if childPid == 0:
handler(connection)
else:
activePID.append(childPid)
accept()
The problem I see is that the final loop in the client only does one server.recv(socksize), and then it calls raw_input(). If that recv() call does not obtain all of the data sent by the server in that single call, then it also won't collect the prompt that follows the command output and therefore won't show that next prompt. The uncollected input will sit in the socket until you enter the next command, and then it will be collected and shown. (In principle it could take many recv() calls to drain the socket and get to the appended prompt, not just two calls.)
If this is what's happening then you would hit the problem if the command sent back more than one buffer's worth (4KB) of data, or if it generated output in small chunks spaced out in time so that the server side could spread that data over multiple sends that are not coalesced quickly enough for the client to collect them all in a single recv().
To fix this, you need have the client do as many recv() calls as it takes to completely drain the socket. So you need to come up with a way for the client to know that the socket has been drained of everything that the server is going to send in this interaction.
The easiest way to do this is to have the server add boundary markers into the data stream and then have the client inspect those markers to discover when the final data from the current interaction has been collected. There are various ways to do this, but I'd probably have the server insert a "this is the length of the following chunk of data" marker ahead of every chunk it sends, and send a marker with a length of zero after the final chunk.
The client-side main loop then becomes:
forever:
read a marker;
if the length carried in the marker is zero then
break;
else
read exactly that many bytes;.
Note that the client must be sure to recv() the complete marker before it acts on it; stuff can come out of a stream socket in lumps of any size, completely unrelated to the size of the writes that sent that stuff into the socket at the sender's side.
You get to decide whether to send the marker as variable-length text (with a distinctive delimiter) or as fixed-length binary (in which case you have to worry about endian issues if the client and server can be on different systems). You also get to decide whether the client should show each chunk as it arrives (obviously you can't use raw_input() to do that) or whether it should collect all of the chunks and show the whole thing in one blast after the final chunk has been collected.
Related
I have a sample client-server program that does non-blocking I/O for several sockets not using processes or threads. It uses select. Unfortunately, the server just shows lots of blank lines and that's all. Where is the mistake?
Running on MacOS.
Thanks in advance.
Server:
import socket
import select
sock = socket.socket()
sock.bind(('', 10001))
sock.listen()
conn1, _ = sock.accept()
conn2, _ = sock.accept()
conn1.setblocking(0)
conn2.setblocking(0)
epoll = select.poll()
epoll.register(conn1.fileno(), select.POLLIN | select.POLLOUT)
epoll.register(conn2.fileno(), select.POLLIN | select.POLLOUT)
conn_map = {
conn1.fileno(): conn1,
conn2.fileno(): conn2,
}
while True:
events = epoll.poll(1)
for fileno, event in events:
if event & select.POLLIN:
data = conn_map[fileno].recv(1024)
print(data.decode('utf8'))
elif event & select.POLLOUT:
conn_map[fileno].send('ping'.encode('utf8'))
Client:
import socket
from multiprocessing import Pool
def create_socket_and_send_data(number):
with socket.create_connection(('127.0.0.1', 10001)) as sock:
try:
sock.sendall(f'client {number}\n'.encode('utf8'))
except socket.error as ex:
print('data sending error', ex)
print(f'data for {number} has been sent')
if __name__ == '__main__':
with Pool(processes=2) as pool:
pool.map(create_socket_and_send_data, range(2))
Unfortunately, the server just shows lots of blank lines and that's all.
Actually this is not true.
The server prints at the beginning the lines it got from the clients. After they've send these lines the client close the connection which means that select.POLLIN gets triggered again on the socket and recv returns empty data.
This empty data is the sign that the peer has closed the connection. Once it got this sign the server should close the connection to the client and remove the fileno from the select. Instead your server prints the empty string with a newline and continues to expect new POLLIN events. These will come again and again and will always an empty buffer, thus leading to all the empty lines you see.
select is paradoxically easier to use for input than for output. For input, you receive an event each time new data arrives on a socket, so you always ask for all the sockets and have something to process for every new event.
For output, select will just say that a socket if ready to accept new data. Which is almost always true except if you have just filled a buffer. So you should only poll for an output socket when you have something to write there.
So you should register your sockets with select.POLLIN only. For the write part, you should either directly write to a socket without polling if you can hope that the peer should always be able to receive, or set up a queue with pending output per socket, modify the polling state of a socket with select.POLLIN | select.POLLOUT when there is something in its queue and modify it back with select.POLLIN back when the queue is empty again.
I am trying to modify a server and client chat script implemented in python. One of the requirement is that the client exits when the user types CTRL-D. My question is how do I read that the user typed (ctrl-D) and implement it within this code. Should I just close the socket like c_socket.close() without any message back to the server that I am exiting?
Thanks!
# telnet program example
import socket, select, string, sys
import readline
def prompt() :
sys.stdout.write('<You> ')
sys.stdout.flush()
#main function
if __name__ == "__main__":
if(len(sys.argv) < 3) :
print 'Usage : python MClient hostname port'
sys.exit()
host = sys.argv[1]
port = int(sys.argv[2])
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(2)
# connect to remote host
try :
s.connect((host, port))
except :
print 'Unable to connect'
sys.exit()
print 'Connected to remote host. Start sending messages'
prompt()
while 1:
socket_list = [sys.stdin, s]
# Get the list sockets which are readable
read_sockets, write_sockets, error_sockets = select.select(socket_list , [], [])
for sock in read_sockets:
#incoming message from remote server
if sock == s:
data = sock.recv(4096)
if not data :
print '\nDisconnected from chat server'
sys.exit()
else :
#print data
sys.stdout.write(data)
prompt()
#user entered a message
else :
msg = sys.stdin.readline()
s.send(msg)
prompt()
Last thing first - whether you need to send something to the server before closing the socket or not depends purely on your protocol. Given the simplistic nature of the presented client code, I'd guess that closing of the socket should be enough (and the server should treat unannounced disconnects anyway as one shouldn't consider network I/O as persistent).
Second, CTRL+D will cause the returned message from sys.stdin.readline() to return as empty so you can test against that, e.g.:
msg = sys.stdin.readline()
if msg:
s.send(msg)
prompt()
else:
s.close()
sys.exit()
If you really need to send something to the server (again, depends on the protocol) you can use socket's shutdown() method before calling close().
However, keep in mind that the code presented here doesn't account for pressing the CTRL+D while reading from the socket (i.e. receiving data) so if there is a particularly long data stream coming in your CTRL+D won't be registered. You can solve this by checking the user input (and writing it to buffer for later sending) during the retrieval procedure, or you can just place your receiving code in a separate thread and leave the main thread just for user input.
You can then use the atexit module to cleanly exit from your client at any time.
I run my script on computer "A". Then I connect to computer "A" from computer "B" through my script. I send my message to computer "A" and my script runs it with an exec() instruction.
I want to see the result of executing my message on computer "A", through a socket on computer "B".
I tried to change sys.stdout = socket_response but had a error: "Socket object has no attribute write()"
So, how can I redirect standard output (for print or exec()) from computer "A" to computer "B" through socket connection?
It will be some kind of 'python interpreter' into my script.
SORRY, I CAN'T ANSWER MY OWN QUESTION WITHOUT REPUTATION
Thanks to all!
I use a simple way, which #Torxed advised me of. Here's my pseudo-code (it's just an example, not my real script)
#-*-coding:utf-8-*-
import socket
import sys
class stdout_():
def __init__(self, sock_resp):
self.sock_resp = sock_resp
def write(self, mes):
self.sock_resp.send(mes)
MY_IP = 'localhost'
MY_PORT = 31337
srv = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
print("Start server")
old_out = sys.stdout
srv.bind((MY_IP, MY_PORT))
srv.listen(0)
sock_resp, addr_resp = srv.accept()
new_out = stdout_(sock_resp)
sys.stdout = new_out
#sys.stdout = sock_resp ### sock_object has no attribute 'write'
while 1:
try:
a = sock_resp.recv(1024)
exec(a)
except socket.timeout:
#print('server timeout!!' + '\n')
continue
I connected to script with Putty and sent "print 'abc'" and then I received the answer 'abc'.
There is the makefile function in Python's socket class:
socket.makefile(mode='r', buffering=None, *, encoding=None,
errors=None, newline=None)
Return a file object associated with the socket. The exact returned
type depends on the arguments given to makefile(). These arguments are
interpreted the same way as by the built-in open() function.
Closing the file object won’t close the socket unless there are no
remaining references to the socket. The socket must be in blocking
mode; it can have a timeout, but the file object’s internal buffer may
end up in a inconsistent state if a timeout occurs.
You can read how to use it in Mark Lutz's book (chapter 12, "Making Sockets Look Like Files and Streams").
An example from the book (the idea is simple: make a file object from a socket with socket.makefile and link sys.stdout with it):
def redirectOut(port=port, host=host):
"""
connect caller's standard output stream to a socket for GUI to listen
start caller after listener started, else connect fails before accept
"""
sock = socket(AF_INET, SOCK_STREAM)
sock.connect((host, port)) # caller operates in client mode
file = sock.makefile('w') # file interface: text, buffered
sys.stdout = file # make prints go to sock.send
return sock # if caller needs to access it raw
Server side:
from subprocess import Popen, STDOUT, PIPE
from socket import socket
from time import sleep
server_sock = socket()
server_sock.bind(('', 8000))
server_sock.listen(4)
def close_process(p):
p.stdin.close()
p.stdout.close()
while 1:
try:
client, client_address = server_sock.accept()
data = client.recv(8192)
except:
break
# First, we open a handle to the external command to be run.
process = Popen(data.decode('utf-8'), shell=True, stdout=PIPE, stdin=PIPE, stderr=STDOUT)
# Wait for the command to finish
# (.poll() will return the exit code, None if it's still running)
while process.poll() == None:
sleep(0.025)
# Then we send whatever output the command gave us back via the socket
# Python3: sockets never convert data from byte objects to strings,
# so we'll have to do this "manually" in case you're confused from Py2.X
try:
client.send(bytes(process.stdout.read(), 'UTF-8'))
except:
pass
# And finally, close the stdout/stdin of the process,
# otherwise you'll end up with "to many filehandles openened" in your OS.
close_process(process)
client.close()
server_sock.close()
This assumes Python3.
If no one else have a better way of just redirecting output to a socket from a process, this is a solution you could work with.
I am writing a client-sever program based on Python socket.
The client sends a command to the server and the server responds.
But now, some client can broadcast a message to other clients, so the client can receive more than one response at the same time.
data = s.recv(1024)
the line of code above will retrieve only one response from the server.
but if I use a while loop like this
while True:
data = s.recv(1024)
if not data: break
actually, data=s.recv(1024) will block the program when there is no data left.
I don't want to block the program and want to retrieve all the responses available in the connection at one time. Can anyone find a solution? Thank you.
You can use the select module to wait until the socket is readable or until a timeout has elapsed; you can then perform other processing. For example:
while True:
# If data can be received without blocking (timeout=0), read it now
ready = select.select([s], [], [], 0)
if s in ready[0]:
data = s.recv(1024)
# Process data
else:
# No data is available, perform other tasks
You could make the socket (s) non-blocking. This way, it will retrieve all the received responses and when there is none, it will return back. Of course, with non-blocking, you will have to periodically retry.
You could make the socket (s) non-blocking using the setblocking() method:
s.setblocking(0)
The other option is to use another thread to handle the receive part. This way, your main thread can continue doing its main task and act upon the message only if it receives one.
You can use socket.setblocking or socket.settimeout:
import socket
import sys
HOST = 'www.google.com'
PORT = 80
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((HOST, PORT))
s.setblocking(0)
s.sendall('Hello, world')
try:
data = s.recv(1024)
except:
print 'Oh noes! %s' % sys.exc_info()[0]
s.close()
socket.recv takes two parameters, the second is a set of flags. If you're on a Linux system, you can do man recv for a list of flags you can supply, and their corresponding errors.
Lastly, in general, you can't really know that the other side is done with sending you data (unless you're controlling both sides), even if you're both following a protocol. I believe the right way to go about it is to use timeouts, and quit after sending a reset (how you do this will depend upon what protocol you're using).
i got this code from http://www.evolt.org/node/60276 and modified it to listen for a single "1" coming from the other side
but whenever i run this program it stops and python IDLE goes to non-responding on "data1,addr = UDPSock.recvfrom(1024)"
def get1():
# Server program, receives 1 if ball found
# ff1 is file w/ received data
import socket
import time
# Set the socket parameters
host = "mysystem"
port = 21567
#buf = 1024
addr = (host,port)
# Create socket (UDP) and bind to address
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
UDPSock.bind(addr)
# Receive messages
while 1:
print "waiting..............."
data1,addr = UDPSock.recvfrom(1024)
print "got 1"
if not data1:
print "Client has exited!"
break
else:
print "\nReceived message '", data1,"'"
UDPSock.close() # Close socket
print "socket closed\n"
#call some other function that uses 1
and client side
def send1():
# Client program, sends 1 if ball found
# mf1 is file with data to be sent
import socket
# Set the socket parameters
host = "mysystem"
port = 21567
buf = 1024
addr = (host,port)
# Create socket (UDP)
UDPSock = socket.socket(socket.AF_INET,socket.SOCK_DGRAM)
mf1=1
print mf1
# Send messages
if(UDPSock.sendto(str(mf1),addr)):
print "Sending message '",str(mf1),"'....."
# Close socket
UDPSock.close()
does anyone know what might be the cause of this? (sorry for long post)
As a second guess (I replaced my first guess with this) I suspect that you are running the receiver in IDLE and then IDLE is hanging so you can't run the client. I don't know exactly how IDLE works as I never use it, but the line containing recvfrom will stop the Python thread its running in until data is sent. So you need to start the client in a separate instance of IDLE or from the command line or something.
At any rate, I have tested the program in question on my Python with 127.0.0.1 as the host, and it worked fine, for some values of fine. The recvfrom does hang, but only until some data is sent, then it comes back with the data and prints it out and everything. You do have a bug that happens after that though. :-)