We Keep Coding

Python while loop is pausing

I am trying to do multiple while loops but somehow they don't work. I already searched the internet but none of the problems I found has the same issue.
So here is the code containing only the necessary information. I am basically opening a socket, giving an in input (i\n) and receiving the output in the first step. I want to continue receiving the output until I have some specific characters xxx in the output. Then I want to go to the elif statement in the next loop.
def netcat(h, p):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((h,p))
i = 0
a = True
while a == True:
socket_list=[sys.stdin, s]
r,w,x = select.select(socket_list, [], [])
if i==0:
time.sleep(1)
message = s.recv(1024)
print(message)
s.send("i\n")
sys.stdout.flush()
while "xxx" not in message:
message = s.recv(1024)
print(message)
i+=1
elif i==1:
print("+++++++++++++++++++++++++++")
i+=1
print("hello")
server.close()
What I would expect the code to do is to print the message from the if statement, then print hello, then the message from the elif statement and then hello over and over again because the while loop is still active. So in summary this is the expected output:
message
hello
+++++++++++++++++++++++++++
hello
hello
hello
hello...
What it really prints is
message
hello
and then it finishes.
What I found out is that if I comment out the following lines:
while "xxx" not in message:
message = s.recv(1024)
print(message)
it works as expected. The hello at the end of the code gets printed to the screen over and over again. I just don't get it why this second while loop has anything to do with it. I would really appreciate help here.
Since the working code was requested, here is also the full code. The hostname and port are from a CTF which is still working so you will be interacting with the CTF-server:
#!/usr/bin/env python
import socket
import time
import select
import sys
base64="ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/ ="
hostname = "18.188.70.152"
port = 36150
def netcat(h, p):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((h,p))
i = 0
a = True
b = True
while a == True:
socket_list=[sys.stdin, s]
r,w,x = select.select(socket_list, [], [])
if i==0:
time.sleep(1)
message = s.recv(1024)
print(message)
s.send("i\n")
sys.stdout.flush()
while "flag" not in message:
message = s.recv(1024)
print(message)
txtfile = message[9:38]
print(txtfile)
i+=1
elif i==1:
print("+++++++++++++++++++++++++++")
i+=1
print("hello")
server.close()
netcat(hostname, port)

You're mixing event-based code (select.select()) with blocking synchronous code (your small while loop with the s.recv()).
If you want your code not to block, every recv() needs to be paired up with a preceding select().
Not only that, but you must also check the returned values from select(). Only s.recv() if s was in the first returned list. If you s.recv() in any other case, the code will also block on the receive call.
Update:
Try something along the lines of:
not_done = True
while not_done:
read_sockets, _, _ = select.select([sys.stdin, s], [], [])
if s in read_sockets:
message = s.recv(1024)
print(message)
... more code ...
if 'flag' in message:
... react on flag ...
if 'quit' in message:
not_done = False
... processing of other sockets or file descriptors ...
The important point being that there is only this one s.recv() in the if branch that checks for whether select detected something was received.
The outer while will just come back to the same if branch later when additional data was received.
Note that processing stdin alongside socket code is tricky and will likely also block at some point. You will likely have to put the terminal into raw mode or something first and then be ready to process partial lines yourself as well as maybe also echoing the input back to the user.
Update:
If you want to do something while no message was received, you can give a timeout to select() and then do other processing if there was nothing received on the socket. Something like this:
say_hello_from_now_on = False
not_done = True
while not_done:
read_sockets, _, _ = select.select([s], [], [], 1)
if s in read_sockets:
message = s.recv(1024)
print(message)
say_hello_from_now_on = True
elif say_hello_from_now_on:
print("hello")

I'd check your indentation, try installing and running autopep8 on your code and see if that fixes any of your issues.
[edit] user has updated their question and it's clear that this isn't the answer.

IRC client application, how to use select() in python

Hi am working on network application that reads from standard input and sends messages to server both in infinite loops. For every loop I have thread, that calls function with what should be done. I need to be able to catch SIGINT and close both threads properly. My program is stucked on both reading functions. I think I should use select() function, but I dont know how.
def writing_mode():
while 1:
var = raw_input()
# some options with what to do with the var
def listening_mode():
readbuffer = ""
while 1:
readbuffer = readbuffer + s.recv(1024)
temp = string.split(readbuffer, "\r\n")
readbuffer = temp.pop()
for line in temp:
line = string.strip(line)
prefix, command, rest, trailing = parse(line)
# do something with the options u have
s=socket.socket()
try:
s.connect((args.h, args.p))
except Exception, e:
print("Something is wrong with %s:%d, Exception type is:\n%s" % (args.h, args.p, e))
sys.exit()
s.send("NICK AAA\r\n")
s.send("USER AAA AAA AAA AAA\r\n")
thread.start_new_thread(writing_mode)
thread.start_new_thread(listenning_mode)
#main program waiting for signals

I've solved this with non-blocking reading through select().
while condition:
# s is socket to read from
# for stdin use sys.stdin instead
ready = select.select([s], [], [], 0)
if ready[0]:
#u can start read
The select function, goes on the reading function only if there is anything to read. Otherwise it goes to the while loop so the condition can stop it at any time.

Keep console input line below output

[EDIT:]
I'm currently trying to make a small tcp chat application. Sending and receiving messages already works fine... But the problem is:
When i start typing a message while i receive one... it appears after the text I'm writing
Screenshot: http://s7.directupload.net/images/140816/6svxo5ui.png
[User sent > "hello", then I started writing "i am writing..." then user wrote " i sent a..." before i sent my message... so it has been placed after my input...
I want the incoming message always to be before my input !
this is my current code:
Client.py
con = connect.User()
server = raw_input("Type in the server adress \n[leave blank to use xyr.no-ip.info]\n>:")
nick =""
while nick == "":
nick = raw_input("Type in your nickname\n>:")
con.connect(server, nick)
def sender():
print("Sender started")
while 1:
msg = raw_input()
if msg == "q":
break
con.send(msg)
con.disconnect()
def receiver(server):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if server == "":
server="xyr.no-ip.info"
sock.connect((server, 8000))
sock.send("%pyreceiver\n")
print("Receiver started")
while 1:
msg_in = sock.recv(1024)
if not str(msg_in).startswith("[py]" + nick):
if str(msg_in).startswith("/ua"):
print(str(msg_in)[3:])
elif str(msg_in).startswith("/u "):
print(str(msg_in)[2:])
else:
print(str(msg_in[:-1]))
#
if nick == "":
nick = "guest"
print("Name changed to ""guest""")
time.sleep(.5)
thread.start_new_thread(receiver, (server, ))
time.sleep(.5)
thread.start_new_thread(sender())
Connect.py
import socket
import time
class User():
nickel =""
def connect(self, server="xyr.no-ip.info", nick="guest"):
nickel = nick
self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
if server == "":
server="xyr.no-ip.info"
print("server changed to xyr.no-ip.info")
time.sleep(.5)
print("Connecting...")
self.sock.connect((server, 8000))
print("Connected")
time.sleep(.4)
self.sock.send("[py]" + nick + "\n")
self.sock.send(nick + " connected with a python client\n")
print("registered as " + nick)
time.sleep(.3)
def send(self, msg):
self.sock.send(msg + "\n")
def disconnect(self):
self.sock.close()
print("disconnected")

Your code writes everything to stdout. Whenever something arrives to either of your sender/receiver threads, it prints to stdout. The issue with that is, due to the fundamental nature of output streams, you cannot accomplish the following :
place incoming messages above the stuff currently being typed/echoed.
Things happen strictly in the order of occurrence. The moment something comes in, wherever the cursor is, the print statement dumps that data over there. You cannot modify that behaviour without using fancier / more powerful constructs.
In order to do what you want, I would use ncurses. You seem to be using python on Windows, so you're going to have to do some digging on how to get equivalent functionality. Check out this thread : Curses alternative for windows

I had a similar problem and I found that a simpler solution (for me) was to get input via readchar (https://github.com/magmax/python-readchar/tree/master/readchar).
Using readchar, I would buffer each keystroke (checking for key.BACKSPACE and CR - see code snippet below).
All output I would prepend with "/033[1A" (make the cursor move up), print the output line, and then a "/n"...
after each output line, I move the cursor to the beginning and re-print the self.input_buff
while the user is doing input, this handles console input, displaying what they are typing:
keypress = readkey()
if keypress == key.BACKSPACE:
self.input_buff = self.input_buff[:-1]
print("\033[0A%s " % self.input_buff)
continue
if keypress != key.CR:
self.input_buff = "%s%s" % (self.input_buff, keypress)
print("\033[0A%s" % self.input_buff)
continue
This kept the input line at the bottom and all terminal output above it.
I know it comes a year late and if you are a wiz with curses, maybe that is the way to go...

Half Closing the socket prints bad characters

My client sends a string "abcd" and half closes the socket (the write part). My server reads the data and appends it inside the list (collection) until end of file is recieved (half closed socket detected). Then it iterates through the list and sends the data.
My server code:
while True:
try:
sock,address = self.__mySocket.accept()
except:
print "Client is dead"
break
print "Client connect: " + str(address)
collection = []
while True:
data = sock.recv()
if len(data) == 0:
break
data = str(data[::-1])
collection.append(data)
for val in collection:
sock.send(val)
sock.close()
The client:
sslsock.sendall('abcd\n')
time.sleep(1)
sslsock.shutdown(socket.SHUT_WR)
data = ""
while True:
data = sslsock.recv()
if len(data) == 0:
sslsock.close()
sys.exit(1)
print data
Now when I print the data on the client it just print garbage. I've tried using pickle and that didn't work either. Now, when I comment out the shutdown on the client and work my server around it just works fine. It prints the reverse of the sent data.

In server Code. I put the for loop inside the if len(data) ==0 . And, It works. I 'm guessing that break statement was breaking out of even the outside While True. So, it never got to the point of sending.

python socket file transfer

I'm trying to write transfer files or chunks of data over a socket. I feel as if I'm reinventing the wheel, but my searches for a simple solution have failed (everything I find is either too simple or too complex). The server would run on a phone running python 2.5.4. The intended application would be to sync music files between the phone and a host computer.
This is the guts of what I have, which appears to work. I send and receive 'ok' to break up streams.
Is sending 'ok' back and forth essentially as stop bits to break up streams of data a reasonable technique?
Is there a standard way to do this?
Running any sort of library server (ftp, http) on the phone is not a useful solution given the limits of the phone's memory and processing power.
server:
import socket
c = socket.socket(socket.AF_INET,socket.SOCK_STREAM)
c.bind(('', 1234))
c.listen(1)
s,a = c.accept()
while True:
data = s.recv(1024)
cmd = data[:data.find('\n')]
if cmd == 'get':
x, file_name, x = data.split('\n', 2)
s.sendall('ok')
with open(file_name, 'rb') as f:
data = f.read()
s.sendall('%16d' % len(data))
s.sendall(data)
s.recv(2)
if cmd == 'end':
s.close()
c.close()
break
client:
import socket
s = socket.socket()
s.connect(('192.168.1.2', 1234))
def get_file(s, file_name):
cmd = 'get\n%s\n' % (file_name)
s.sendall(cmd)
r = s.recv(2)
size = int(s.recv(16))
recvd = ''
while size > len(recvd):
data = s.recv(1024)
if not data:
break
recvd += data
s.sendall('ok')
return recvd
print get_file(s, 'file1')
print get_file(s, 'file2')
s.sendall('end\n')

Is sending 'ok' back and forth essentially as stop bits to break up
streams of data a reasonable technique?
Most protocols use some terminator or another. Popular alternatives are '\r\n', '\r\n\r\n' or EOF (ctrl+d), but these are just arbitrarily chosen and no worse or better than your 'ok', as long as your client and server know how to handle it.

Your code looks good.
You don't actually need to send across the size of the file. You can use while True, as the check if not data: break will stop the loop.
while True:
data = s.recv(1024)
if not data: print " Done "; break
recvd += data
Also, why are you sending 'ok' is the other side doesn't check for it? You are just skipping 2 bytes at each side.
Don't you need to cater to multiple clients? No need for multi-threading?

Is there a standard way to do this?
Yes. http://www.faqs.org/rfcs/rfc959.html
Describes the standard way to do this.
Here is an implementation: http://docs.python.org/library/ftplib.html

U may look at this implementation. It also take care of if the file is in a sub-directory. Here is the link!
server
import socket
import os
print('Waiting for clinet to connect...')
c = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
c.bind(('', 1234))
c.listen(1)
s, a = c.accept()
print('Connected. Going to receive file.')
s.sendall('getfilename')
filename = s.recv(1024)
if '/' in filename:
dir = os.path.dirname(filename)
try:
os.stat(dir)
except:
print('Directory does not exist. Creating directory.')
os.mkdir(dir)
f = open(filename, 'wb')
print('Filename: ' + filename)
while True:
s.sendall('getfile')
size = int(s.recv(16))
print('Total size: ' + str(size))
recvd = ''
while size > len(recvd):
data = s.recv(1024)
if not data:
break
recvd += data
f.write(data)
#print(len(recvd))
break
s.sendall('end')
print('File received.')
s.close()
c.close()
f.close()
client
import socket
import sys
if len(sys.argv) > 1 :
print('Trying to connect...')
s = socket.socket()
s.connect(('127.0.0.1', 1234))
print('Connected. Wating for command.')
while True:
cmd = s.recv(32)
if cmd == 'getfilename':
print('"getfilename" command received.')
s.sendall(sys.argv[1])
if cmd == 'getfile':
print('"getfile" command received. Going to send file.')
with open(sys.argv[1], 'rb') as f:
data = f.read()
s.sendall('%16d' % len(data))
s.sendall(data)
print('File transmission done.')
if cmd == 'end':
print('"end" command received. Teminate.')
break

rsync is the standard way to sync files between two computers. You could write it in Python like this http://code.activestate.com/recipes/577518-rsync-algorithm/ or you could wrap the C library like this http://freshmeat.net/projects/pysync/ with some tweaks like replacing MD4 with MD5.
Or, if you want to do this at the socket level, you really should be using asynchat with asyncore. Here is an FTP server written with asynchat http://pyftpdlib.googlecode.com/svn-history/r20/trunk/pyftpdlib/FTPServer.py but you should start by reading http://www.doughellmann.com/PyMOTW/asynchat/ Pay attention to the part about Message Terminators point 2. A lot of network protocols do odd stuff like this, i.e. sometimes they send and receive full line commands and responses, and sometimes they send and receive chunks of arbitrary data preceded by the count of how many bytes are in the chunk. You can handle this much more easily with asynchat, and your program will scale much better too.