I am working on a TCP chatroom with python 2.7, my server should manage multiple clients (I used select module) and used threads for clients, I have a problem on what I am printing.
Since my raw_input in the client has text in it, it gets easily ugly when the other thread in charge of printing other clients messages gets triggered .
My code's results looks like this : (on client's window)
WELCOME TO CHATROOM
you are connected
your pseudo is PSEUDO
other_user joined the chatroom
and the my raw_input() pops out and it goes like this :
* < your message > * :e chatroom
Or sometimes I get another user's message over Raw_input text
because it replaces the last line printed, I figured I could delete the last printed line if it's equal to my raw_input text to print an unexpected message over it and then reprint my Raw_input but I don't know to implement it.
I have tried moving my cursor with "\x1b["-like commands and deleting last printed line but those commands are not working for me.
class Sender(Thread):
"""would send messages to the server"""
def __init__(self, socket, username):
Thread.__init__(self)
self.socket = socket
self.username = username
def run(self):
while 1:
message = raw_input("* < your message > * :")
self.socket.send(message)
class Receiver(Thread):
import sys
def __init__(self,socket):
Thread.__init__(self)
self.connexion = socket
def run(self) :
while 1:
message_received = self.connexion.recv(4096)
message_received = message_received.decode()
sys.stdout.write("\r" + message_received)
sys.stdout.flush()
if message_recu =='' or message_recu.upper() == "OFF":
break
self._Thread__stop()
print "\nClient arrete. Connexion interrompue."
self.connexion.close()
For \b and \r to work you need to be ending your print statements with a space instead of a newline.
EG:
print 'hello there', the , at the end of the print statement defines that it wont go forwards one line, but instead end it with a space. Then you can use another print statement to delete the text.
EG:
print '\r ' this would go to the beginning of the line and replace it with a space, then you can output text over it.
Related
i made a messenger that works with the socket library. it has 2 sides : server and client.
i later decided to make a GUI for it , too. ( with tkinter )
when i was making it , i realized that programs does not work correctly. here :
import socket
from tkinter import *
win = Tk()
win.geometry("300x300")
win.resizable(False,False)
def disc() :
s = socket.socket()
ip = "0.0.0.0"
port = 9999
s.bind((ip,port))
s.listen()
print ('please wait...')
c , addr =s.accept()
print ('someone has joined!')
while True :
msg = input('your message : ' )
c.send(msg.encode('utf8'))
print (c.recv(1024).decode())
lbl_1 = Label(win,text="mychat",bg="light blue")
lbl_1.place(x=130,y=20)
lbl_2 = Label(win,text="your message: ")
lbl_2.place(x=130,y=50)
lbl_3 = Label(win,text="recieved message: ")
lbl_3.place(x=130,y=70)
btn_1 = Button(win,text="make your device discoverable",command=disc)
btn_1.pack()
txt_1 = Text(win)
txt_1.pack()
word = "messages"
txt_1.insert(END, word)
win.mainloop()
here , you can see what i've tried. i have two parts : the socket part and the gui part.
the socket part is in the def block.
but this does not work correctly. if you put the mainloop before socket part , it will never be executed because mainloop is not finished until i close the program.
if you put main loop after socket part , the GUI will not be displayed until someone join the server.(because socket part is not finished)
here , you see i've tried another thing. i put the socket part in def and then made a button for it. but this doesn't work either. when you press the button , the program stops and gives a ( not responding ) error on title bar . ( so until someone has joined , it will not respond. )
i want a solution for this code that the GUI part works and doesn't care to socket part(dismissing it). in other words , python executes the 2 parts in one time.
Sounds like you want to start using threads.
You basically have two "while True" loops, the first is the win.mainloop() that stops executing when the user exists the GUI. The second is the while True in the disc function, that waits for input from the user (msg = input(...)) and then waits for data from the client (s.recv)
import threading
def disc():
...
connection = threading.Thread(target=disc).start()
The rest of the GUI code goes below the connection thread without the make your device discoverable part as it is now unnecessary:
# remove this part
btn_1 = Button(win,text="make your device discoverable",command=disc)
btn_1.pack()
The Problem
I want to interact with interactive terminal programs from Python scripts, these programs might not always be written in Python. I already managed to do it with pexpect and the class in the code snippet below but I struggle to find a way to capture the whole output after each instruction.
The Context
I cannot capture the whole output of the command (all the lines) and keep the program alive for future inputs.
Let's say I want to do this:
terminal.start("/path/to/executable/repl/file") # on start returns 3 lines of output
terminal.run_command("let a = fn(a) { a + 1 }") # this command return 1 line of output
terminal.run_command("var") # this command will return 2 lines of output
terminal.run_command("invalid = invalid") # this command returns 1 line of output
note that the amount of lines on each output might vary because I want to be able to run multiple interactive terminal programs.
What I have tried
Attempt 1
I tried using readlines but as the documentation states
Remember, because this reads until EOF that means the child process should have closed its stdout.
It means that when once I run that it will close my process for future instructions, which is not my expected behaviour. Anyways when I try it I get the following.
def read(self):
return list(self.process.readlines())
For a reason unknown to me the program just does nothing, prints nothing, raises no error, just stays paused with no output whatsoever.
Attempt 2
Read each line until finding an empty line like this
def read(self):
val = self.process.readline()
result = ""
while val != "":
result += val
val = self.process.readline()
return result
Once again the same problem, the program pauses, prints no input, does nothing for a few seconds then it prints the error pexpect.exceptions.TIMEOUT: Timeout exceeded.
Attempt 3
using read_nonblocking method causes my program to read only a few characters, so I use the first parameter size as follows.
def read(self):
return self.process.read_nonblocking(999999999)
Only then I get the expected behavior but only for a few commands, then it reads nothing, besides, If I put an even bigger number, an error on memory overflow is raised.
The Code
This is the implementation of the Terminal class.
import pexpect
class Terminal:
process: pexpect.spawn
def __init__(self):
self.process = None
def start(self, executable_file: str):
'''
run a command that returns an executable TUI program, returns the output,
(if present) of the initialization of program
'''
self.process = pexpect.spawn(executable_file, encoding="utf-8", maxread=1)
return self.read()
def read(self):
'''return entire output of last executed command'''
return self.process.readline() # when executed more than amoutn of output program breaks
def write(self, message):
'''send value to program through keyboard input'''
self.process.sendline(message)
def terminate(self):
'''kill process/program and restart property value to None'''
self.process.kill()
self.process.wait()
self.process = None
def run_command(self, command: str):
'''
run an instruction for the executed program
and get the returned result as string
'''
self.write(command)
return self.read()
How I consume the class. This is what I run to test on each attempt mentioned above
from terminal import Terminal
term = Terminal()
print(term.start("/path/to/executable/repl/file"), end="")
print(term.run_command("let a = fn(a) { a + 1 }"), end="")
print(term.run_command("a(1)"), end="")
print(term.run_command("let b = [1,2,4]"), end="")
print(term.run_command("b[0]"), end="")
print(term.run_command("b[1]"), end="")
print(term.run_command("a(2)"), end="")
If you want to know what kind of specific programs I want to run, its just these two 1 and 2 at the moment but I expect to add more in the future.
The crux of the problem comes from detecting when the command you sent to the console program has finished writing output.
I started by creating a very simple console program with input and output : echo. It just writes back what you wrote.
Here it is :
echo.py
import sys
print("Welcome to PythonEcho Ultimate Edition 2023")
while True:
new_line = sys.stdin.readline()
print(new_line, file=sys.stdout, end="") # because the line already has an \n at the end
print("") # an empty line, because that's how the webshell detects the output for this command has ended
It is a slightly modified version, that prints a line at the start (because that's what your startProgram() expects when doing terminal.read()) and an empty line after each echo (because that's how your runCommand detected that the output finished).
I did without Flask, because it is not needed for making the communication with the console program to work, and helps with debug (see Minimal Reproducible Example). So here is the code I used :
main.py
import subprocess
class Terminal:
def __init__(self):
self.process = None
def start(self, executable_file):
self.process = subprocess.Popen(
executable_file,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE
)
def read(self):
return self.process.stdout.readline().decode("utf-8")
# no `.strip()` there ^^^^^^^^
def write(self, message):
self.process.stdin.write(f"{message.strip()}\n".encode("utf-8"))
self.process.stdin.flush()
def terminate(self):
self.process.stdin.close()
self.process.terminate()
self.process.wait(timeout=0.2)
terminal = Terminal()
def startProgram():
terminal.start(["python3", "echo.py"]) # using my echo program
return terminal.read()
def runCommand(command: str):
terminal.write(command)
result = ""
line = "initialized to something else than \\n"
while line != "\n": # an empty line is considered a separator ?
line = terminal.read()
result += line
return result
def stopProgram():
terminal.terminate()
return "connection terminated"
if __name__ == "__main__": # for simple testing
result = startProgram()
print(result)
result = runCommand("cmd1")
print(result, end="")
result = runCommand("cmd2")
print(result, end="")
result = stopProgram()
print(result)
which gives me
Welcome to PythonEcho Ultimate Edition 2023
cmd1
cmd2
connection terminated
I removed the split() at the end of the Terminal.read otherwise the lines would not be correctly concatenated later in runCommand (or was it voluntary ?).
I changed the runCommand function to stop reading when it encounters an \n (which is an empty line, different from an empty string which would indicate the end of the stream). You did not explain how you detect that the output from your programs ended, you should take care to this.
Is it possible to print something without overwriting the already written but not sent input?
This is the client side for a socket and I'm wanting to print messages from the server without overwriting/appending the currently written input.
def listen():
while True:
data = s.recv(1024)
print(data.decode('utf-8'))
def write():
while True:
text = input("Text > ")
s.sendall(bytes(text, 'utf-8'))
listener = threading.Thread(target=listen)
listener.start()
writer = threading.Thread(target=write)
writer.start()
I'd want to print the received data above or below the current input line, but right now it just writes it on the input line.
Add time.sleep(10) after s.sendall(bytes(text, 'utf-8')), because you immediately do input while the data is received and printed later. Having time.sleep you give time for the data to be received and printed.
But this won't help you if you enter the data for input too slow. Generally the code is strange, because why would you do input and print on the same terminal? In real life it doesn't make sense.
For academic purposes you can try this:
is_input_active = False
def listen():
while True:
data = s.recv(1024)
while is_input_active:
time.sleep(0.2)
print(data.decode('utf-8'))
def write():
global is_input_active
while True:
is_input_active = True
text = input("Text > ")
is_input_active = False
s.sendall(bytes(text, 'utf-8'))
With this version listen() will block until input function finishes. If you don't want this, try something like:
def listen():
data = b''
while True:
data += s.recv(1024) # accumulate
while not is_input_active:
print(data.decode('utf-8'))
data = b'' # reset
Interactive terminal input is complex and is usually handled with libraries, it's hard to handle it directly. For line input, readline and its alternatives such as libedit are popular. Most shells and REPLs use such libraries.
Python's standard library has readline module, which is interface to readline or libedit. Importing it makes input use realine/libedit, so it obtains cursor navigation and even auto-completion and history capabilities. And it allows to redraw input line after you draw something in terminal.
To print in previous line, you can use ANSI escape codes.
import threading
import sys
from time import sleep
import readline # importing readline makes input() use it
def listen():
i = 0
while True:
i += 1
sleep(1)
# <esc>[s - save cursor position
# <esc>[1A - move cursor up
# \r - carriage return
# print message
# <esc>[u - restore cursor position
sys.stdout.write("\x1b[s\x1b[1A\rOutput: "+str(i)+"\x1b[u")
readline.redisplay()
def write():
while True:
print "" # make space for printing messages
text = input("Text > ")
listener = threading.Thread(target=listen)
listener.daemon = True
listener.start()
write()
readline library also has rl_message and other useful capabilities, but its' not exported by python's readline module.
Example: A simple program that prints the value of a list every 10 seconds
import argparse
import time
import sys
myList = []
def parseArguments():
parser = argparse.ArgumentParser(description="example")
parser.add_argument('-a', '--addElement', help='adds an element to the list')
args = parser.parse_args()
if args.addElement:
myList.append(args.addElement)
def main():
parseArguments()
while(True):
print(myList)
time.sleep(10)
The problem is that the program only reads the arguments passed at the start, I want it to read arguments passed at any time while it is running.
I want to run the program in the background like a service, and pass arguments to the program every once in a while.
I understand that what you are asking for looks like a service (or daemon process) able to accept asynchonous commands.
External interface:
prog foo
=> ok repeatedly prints ['foo']
later:
prog bar
=> second instance exits and first instance repeatedly prints ['foo', 'bar']
Internal design
That's far from being simple! You need to setup an IPC mechanisme to allow second instance to communicate with first one, with non blocking IO (or multithreading) in first instance. Under Unix, you could use os.mkfifo, but is you want a portable solution, your will have to use IP sockets on localhost
Structure in high level pseudo code
get argument via argparse
bind to a fix port on localhost, in UDP protocol
if success:
# ok it is the first prog
initialize list from argument
loop:
get command from UDP socket, with timeout = 10s
if cmd is add param:
add parameter to list
elif cmd is exit: # not asked in question but should exist
exit
print list
else:
# another prog has taken the socket, pass it the arg
send the arg to the UDP port with proper protocol
Caveats on this simple design: there is a race condition is there is already a prog waiting on the socket that exits between the first try to bind and the send. To deal with that, you should use TCP protocol, with a select with timeout on listening socket, and a graceful shutdown to ensure that the message was received on the other side. In case of an error, you iterate (a maximum number of time) because the first server could have exited in the while.
Here is an implementation example:
import socket
import select
import argparse
import time
import sys
TIMEOUT=10
IFACE='127.0.0.1'
PORT=4000
DEBUG=False
myList = []
old = ""
def parseArguments():
parser = argparse.ArgumentParser(description="example")
parser.add_argument('-a', '--addElement',
help='adds an element to the list')
parser.add_argument('-q', '--quit', action='store_true',
help='closes main service')
parser.add_argument('-d', '--debug', action='store_true',
help='display debug information')
args = parser.parse_args()
if args.quit:
senddata("QUIT\n")
sys.exit(0)
if args.debug:
DEBUG=True
if args.addElement:
myList.append(args.addElement)
def read(s):
global old
data = old
while True:
block = s.recv(1024)
if len(block) == 0: return data
if b'\n' in block:
block,o = block.split(b'\n', 1)
old = o.decode()
data += block.decode()
return data
data += block.decode()
def gracefulclose(s, msg):
s.send(msg.encode())
s.shutdown(socket.SHUT_WR)
try:
read(s)
finally:
s.close()
def server(s):
if DEBUG:
print("SERVER")
s.listen(5)
while True:
sl = select.select([s], [], [], TIMEOUT)
if len(sl[0]) > 0:
s2, peer = s.accept()
try:
data = read(s2)
print(data)
gracefulclose(s2, "OK")
finally:
s2.close()
if data.startswith("QUIT"):
return
elif data.startswith("DATA:"):
myList.append(data[5:])
print(myList)
def senddata(data):
s = socket.socket(socket.AF_INET)
try:
s.connect((IFACE, PORT))
s.send(data.encode())
data = read(s)
if (data.startswith("OK")):
return True
except:
pass
finally:
s.close()
return False
def client():
return senddata("DATA:" + myList[0] + "\n")
def main():
end = False
MAX = 5
while not end and MAX > 0:
s = socket.socket(socket.AF_INET)
try:
s.bind((IFACE, PORT))
except Exception:
s.close()
s = None
if s:
try:
server(s)
finally:
s.close()
return
else:
if DEBUG:
print("CLIENT", " ", 6 - MAX)
end = client()
MAX -= 1
time.sleep(1)
if __name__ == "__main__":
parseArguments()
main()
import argparse
import time
import sys
myList = []
def parseArguments():
parser = argparse.ArgumentParser(description="example")
parser.add_argument('-a', '--addElement', help='adds an element to the list')
args = parser.parse_args()
if args.addElement:
myList.append(args.addElement)
def main():
parseArguments()
import select
while(True):
while select.select([sys.stdin], [], [], 0)[0]:
myList.append(sys.stdin.readline().strip())
print(myList)
time.sleep(10)
If you are passing more arguments during execution, you must read them from the stdin. Using the select module you can check if there is any new line in stdin and then add them to myList.
Basically what you're asking is how to do Inter-process communication (IPC).
Why did I say that? Well, answer yourself: how would you like to pass these arguments to your background service? By hand? I don't think so (because that way you'd have a simple interactive program which should just wait for user input). You probably want some other script/program which sends these arguments via some kind of commands on-demand.
Generally there are several several ways to communicate two or more programs, the most popular being:
Shared file - you could simply check contents of a file on your disk. Advantage of this solution is that you could probably edit this file with your favourite text editor, without the need of writing a client application.
Pipes - one program reads its input which is the other program's output. You should simply read sys.stdin.
# receiver
def read_input():
for l in sys.stdin:
yield l
Sockets - a data stream sent over a network interface (but it can be sent locally on the same machine). Python docs have very nice introduction to sockets programming.
Shared memory - your programs read/write the same memory block. In Python you can use mmap module to achieve this.
Whichever way to communicate your processes you choose, you should establish some kind of interface between them. It can be very simple text-based interface like this one:
# command syntax
<command> SPACE <parameter> NEWLINE
SPACE := 0x20 # space character
NEWLINE := 0x0A # '\n' character
# a command adding element to receiver's list
ADD SPACE <element> NEWLINE
# a command removing element from receiver's list:
REMOVE SPACE <element> NEWLINE
# examples:
ADD first element\n
REMOVE first element\n
So for example if you send a message over a socket (which I recommend), your receiver (server) should read a buffer until a newline character, then check if the first word is "ADD" and then add remaining characters (minus newline) to your list. Of course you should be prepared for some kind of "attacks" - like you should specify that your messages cannot be longer than e.g. 4096 bytes. This way you can discard your current buffer after it reached its limitation, meaning that you won't allocate memory indefinitely while waiting for a newline character. That's one very important rule: don't trust user input.
Good luck! :)
The next script I'm using is used to listen to IMAP connection using IMAP IDLE and depends heavily on threads. What's the easiest way for me to eliminate the treads call and just use the main thread?
As a new python developer I tried editing def __init__(self, conn): method but just got more and more errors
A code sample would help me a lot
#!/usr/local/bin/python2.7
print "Content-type: text/html\r\n\r\n";
import socket, ssl, json, struct, re
import imaplib2, time
from threading import *
# enter gmail login details here
USER="username#gmail.com"
PASSWORD="password"
# enter device token here
deviceToken = 'my device token x x x x x'
deviceToken = deviceToken.replace(' ','').decode('hex')
currentBadgeNum = -1
def getUnseen():
(resp, data) = M.status("INBOX", '(UNSEEN)')
print data
return int(re.findall("UNSEEN (\d)*\)", data[0])[0])
def sendPushNotification(badgeNum):
global currentBadgeNum, deviceToken
if badgeNum != currentBadgeNum:
currentBadgeNum = badgeNum
thePayLoad = {
'aps': {
'alert':'Hello world!',
'sound':'',
'badge': badgeNum,
},
'test_data': { 'foo': 'bar' },
}
theCertfile = 'certif.pem'
theHost = ('gateway.push.apple.com', 2195)
data = json.dumps(thePayLoad)
theFormat = '!BH32sH%ds' % len(data)
theNotification = struct.pack(theFormat, 0, 32,
deviceToken, len(data), data)
ssl_sock = ssl.wrap_socket(socket.socket(socket.AF_INET,
socket.SOCK_STREAM), certfile=theCertfile)
ssl_sock.connect(theHost)
ssl_sock.write(theNotification)
ssl_sock.close()
print "Sent Push alert."
# This is the threading object that does all the waiting on
# the event
class Idler(object):
def __init__(self, conn):
self.thread = Thread(target=self.idle)
self.M = conn
self.event = Event()
def start(self):
self.thread.start()
def stop(self):
# This is a neat trick to make thread end. Took me a
# while to figure that one out!
self.event.set()
def join(self):
self.thread.join()
def idle(self):
# Starting an unending loop here
while True:
# This is part of the trick to make the loop stop
# when the stop() command is given
if self.event.isSet():
return
self.needsync = False
# A callback method that gets called when a new
# email arrives. Very basic, but that's good.
def callback(args):
if not self.event.isSet():
self.needsync = True
self.event.set()
# Do the actual idle call. This returns immediately,
# since it's asynchronous.
self.M.idle(callback=callback)
# This waits until the event is set. The event is
# set by the callback, when the server 'answers'
# the idle call and the callback function gets
# called.
self.event.wait()
# Because the function sets the needsync variable,
# this helps escape the loop without doing
# anything if the stop() is called. Kinda neat
# solution.
if self.needsync:
self.event.clear()
self.dosync()
# The method that gets called when a new email arrives.
# Replace it with something better.
def dosync(self):
print "Got an event!"
numUnseen = getUnseen()
sendPushNotification(numUnseen)
# Had to do this stuff in a try-finally, since some testing
# went a little wrong.....
while True:
try:
# Set the following two lines to your creds and server
M = imaplib2.IMAP4_SSL("imap.gmail.com")
M.login(USER, PASSWORD)
M.debug = 4
# We need to get out of the AUTH state, so we just select
# the INBOX.
M.select("INBOX")
numUnseen = getUnseen()
sendPushNotification(numUnseen)
typ, data = M.fetch(1, '(RFC822)')
raw_email = data[0][1]
import email
email_message = email.message_from_string(raw_email)
print email_message['Subject']
#print M.status("INBOX", '(UNSEEN)')
# Start the Idler thread
idler = Idler(M)
idler.start()
# Sleep forever, one minute at a time
while True:
time.sleep(60)
except imaplib2.IMAP4.abort:
print("Disconnected. Trying again.")
finally:
# Clean up.
#idler.stop() #Commented out to see the real error
#idler.join() #Commented out to see the real error
#M.close() #Commented out to see the real error
# This is important!
M.logout()
As far as I can tell, this code is hopelessly confused because the author used the "imaplib2" project library which forces a threading model which this code then never uses.
Only one thread is ever created, which wouldn't need to be a thread but for the choice of imaplib2. However, as the imaplib2 documentation notes:
This module presents an almost identical API as that provided by the standard python library module imaplib, the main difference being that this version allows parallel execution of commands on the IMAP4 server, and implements the IMAP4rev1 IDLE extension. (imaplib2 can be substituted for imaplib in existing clients with no changes in the code, but see the caveat below.)
Which makes it appear that you should be able to throw out much of class Idler and just use the connection M. I recommend that you look at Doug Hellman's excellent Python Module Of The Week for module imaplib prior to looking at the official documentation. You'll need to reverse engineer the code to find out its intent, but it looks to me like:
Open a connection to GMail
check for unseen messages in Inbox
count unseen messages from (2)
send a dummy message to some service at gateway.push.apple.com
Wait for notice, goto (2)
Perhaps the most interesting thing about the code is that it doesn't appear to do anything, although what sendPushNotification (step 4) does is a mystery, and the one line that uses an imaplib2 specific service:
self.M.idle(callback=callback)
uses a named argument that I don't see in the module documentation. Do you know if this code ever actually ran?
Aside from unneeded complexity, there's another reason to drop imaplib2: it exists independently on sourceforge and PyPi which one maintainer claimed two years ago "An attempt will be made to keep it up-to-date with the original". Which one do you have? Which would you install?
Don't do it
Since you are trying to remove the Thread usage solely because you didn't find how to handle the exceptions from the server, I don't recommend removing the Thread usage, because of the async nature of the library itself - the Idler handles it more smoothly than a one thread could.
Solution
You need to wrap the self.M.idle(callback=callback) with try-except and then re-raise it in the main thread. Then you handle the exception by re-running the code in the main thread to restart the connection.
You can find more details of the solution and possible reasons in this answer: https://stackoverflow.com/a/50163971/1544154
Complete solution is here: https://www.github.com/Elijas/email-notifier