We Keep Coding

New to python need to send data over already connected TCP port without waiting for request from client

7 socket listener setup. It works great and keeps the connection open, non blocking, all that. From time to time a file will show up that I need to hand back to the client. That works to, but it only send the data in the file if the client sends a character first. I need to have it send the data when the file shows up and not wait. I am coming from php and know what I am doing there. Python is new to me so there are some nuances I don't understand about this code.
while True:
try:
#I want this bit here to fire without waiting for the client to send anything
#right now it works except the client has to send a character first
#check for stuff to send back
for fname in os.listdir('data/%s/in' % dirname):
print(fname)
f = open('data/%s/in/%s' % (dirname, fname), "r")
client.send(f.readline())
data = client.recv(size)
if data:
bucket=bucket+data
else:
raise error('Client disconnected')
except Exception as e:
client.close()
print(e)
return False

Output incoming UDP data to new terminals with Python3.7 on Windows

I have a client/server UDP program running on python3.7 on Windows. After establishing a connection, the client listens for incoming data from the server. Each time a client receives data/message (from server), a new terminal should open displaying that message and the client should be able to respond to that message on that terminal. So if a client receives 4 incoming messages from the server, 4 terminals should open with each one displaying their corresponding message.
As of now, my program works just fine but on one terminal.
Could someone please help me? I was able to make a new terminal open for each incoming message with os.system("start cmd") but that's as far as I was able to get. These terminal instances just point to the project's directory with nothing running on them and I understand why, this is just as much 'progress' as I was able to make.
Here is the sample of my code I am referring to :
def clientListen():
while 1:
try:
data, server = client_listen_socket.recvfrom(1024)
if data:
reply = pickle.loads(data)
if str(reply) == 'Connection Successful':
print('\n ~~~ INCOMING MESSAGE FROM ' + str(serverName) + ' ~~~\n') #REPLACE->localhost
print(reply)
clientServerConnectionOutput()
else:
# *OPEN TERMINAL DISPLAYING REPLY*
print('\n ~~~ INCOMING MESSAGE FROM ' + str(serverName) + ' ~~~\n') #REPLACE->localhost
print(reply)
except socket.error:
pass
So each time it receives data, (if data), under the else clause, this is where I want a new terminal to open displaying the reply.
Any help would be greatly appreciated!
Thanks

Continous receiving & sending on a socket

I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.

How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.

Why doesn't the server receive my second command from the client?

The server is already set up. I connected to it and got it to return any data. The received data printed for the first piece of data that I sent. But then I tried to send something else, which, in theory, should return another string, but it just says that it's still expecting it.
At the moment, this is my code:
import socket
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
clientsocket.connect(('ServerHostHere', PortNumberHere))
clientsocket.send('hello')
dataOne = clientsocket.recv(2048)
print(dataOne)
This returned : "Hello to you too!"
Then I tried sending another piece of information:
clientsocket.send('How are you?')
dataTwo = clientsocket.recv(1024)
print(dataTwo)
Instead of returning a line, it just says "Error, expecting command question") meaning that it didn't receive my second command.
Is my code wrong or am I supposed to add something else?

How to detect if user has entered any data as a console input

Is there a way to find out if the user has entered any data in the terminal window without having to use the blocking stdin.
I am implementing a chat client using twisted python and the client code should display messages from other connected clients. As soon as the client enters a message and hits enter, i want it to run an event driven loop, which sends the message to the server, which then broadcasts it to every other client.
In short, I am trying to look for a way to detect when the user hits ENTER or enters some text in the terminal without having to block the program.
UPDATE: Client code so far..
class MyClientProtocol( protocol.Protocol ):
def sendData( self ):
message = raw_input( 'Enter Message: ' )
if message and ( message != "quit()" ):
logging.debug( " ...Sending %s ...", message )
self.transport.write( str( message ) )
else:
self.transport.loseConnection()
def connectionMade( self ):
print "Connection made to server!"
def dataReceived( self, msg ):
print msg
self.sendData()
class MyClientFactory( protocol.ClientFactory ):
protocol = MyClientProtocol
clientConnectionLost = clientConnectionFailed = lambda self, connector, reason: reactor.stop()
reactor.connectTCP( HOST, PORT, MyClientFactory() )
reactor.run()
This code only currently only accepts the user input after receiving something from back from the server, as i am calling sendData in dataReceived. Any suggestions on how i can make this pick up user entered data, plus also keep getting data from the server?

If you're already using Twisted, they have plugins for hooking almost anything into the event loop.
But for stdin, you don't even need a plugin; it's built in. One of the stock examples even shows exactly what you're trying to do. It's the one named stdin.py.

I also recently played around with this. What I did was just start a separate thread (using the threading module) that was waiting for user input, and the main thread was receiving and printing broadcast messages, like:
def collect_input():
while True:
msg = raw_input()
handle(msg) # you'll need to implement this
# in client code
import threading
t = threading.Thread(target=collect_input)
t.start()
I'm not sure if it's a good idea, but it was the first that came to my mind and it seemed to work.
Note: I wasn't using Twisted, just sockets. As you can see from the other answer, you don't need to implement it with Twisted.