We Keep Coding

Continous receiving & sending on a socket

I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.

How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.

Infinite wait for reply from server using sockets in Python

I'm trying to send a message to the API of a server in order to get a reply. I am using the following code:
import socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_address = ('h68.p.ctrader.com',5211)
sock.connect(server_address)
message = "8=FIX.4.4|9=87|35=0|49=theBroker.12345|56=cServer|57=QUOTE|50=BVN's Message|34=1|52=20180322-21:26:01|10=101"
sock.send(bytes(message,'utf-8'))
data = sock.recv(3)
print(data)
sock.close()
However, when executing it, the message is sent to the server, but at the moment of receiving the reply from the server [data = sock.recv (3)], the program does not go on. It keeps the cursor blinking, like in an infinite loop. What is the cause of this problem? Was it the script? The message sent to the server? The server itself? How to solve the problem?
Note: This message is in a format required by the server API, which consists of "tag" = "value" | "tag" = "value" | "tag" = "value" ...

A quick look at the documentation for this API you're using turns up a couple of problems:
You're missing a | at the very end of the message.
You shouldn't actually be sending |s at all, that's just a human-readable translation of the actual message. All of them need to be replaced with \x01 or \u0001 (depending on whether you're creating a byte string or a Unicode string), the actual tag separator character.

Prevent print() statement from overwriting text from input() statement

So I'm working on a simple chat server in python.
The server is running fine and on the client I have one thread for receiving incoming data and one for sending messages.
I can send a message from client_1 to the server which passes it to all other clients which will then print the message.
Even though everything technically works fine, there is still one thing that is VERY annoying whenever it happens:
Say client_1 is typing text into the console.
At the same time client_2 sends a message to the server, the server sends it to client_1 and client_1 prints the message.
Now the text client_1 was originally typing into the console is no longer in the line it was supposed to be in.
This is what the consoles looked like before client_2 sent the string "test test": https://ibb.co/hFdeo7
and this is what they looked like after sending: https://ibb.co/mEWAvn
NOTE: If I were to press Enter on client_1 the message "TEST TEST TEST" would still be sent correctly. The problem only lies in the conflict between the text that is being printed and the text in the input() statement.
My code looks like This:
Server.py
connections = []
while True:
readable, writeable, exception = select.select(connections, [], [], 0)
for sock in readable:
if sock == server:
conn, addr = server.accept()
connections.append(conn)
else:
data = str(sock.recv(1024), 'utf8')
if data:
for s in connections:
if s != server_socket and s != sock:
s.send(bytes(data, 'utf8'))
else:
connections.remove(sock)
Client.py
def receive():
while True:
readable, writeable, exception = select.select([0, client], [], [])
for sock in readable:
if sock == client:
data = str(sock.recv(1024), 'utf8')
if data:
print(data)
def send():
while True:
readable, writeable, exception = select.select([0, client], [], [])
for sock in readable:
if sock == client:
pass
else:
msg = input()
client.send(bytes(msg, 'utf8'))
Thread(target=receive).start()
Thread(target=send).start()
Is there any way to solve this problem without running the send() and receive() functions in separate scripts, or using a GUI module like Tkinter?
EDIT: I would like to print the incoming message as soon as it is received but then display the input() prompt and typed text again afterwards.

Here's one approach to showing the incoming message and then redisplaying the input prompt and partial input string (as specified in a comment). It uses readline.get_line_buffer to read the currently-input string and redisplay it. A warning, though: reading and writing to the same stream from different threads without locking is going to be prone to glitches.
It just requires a small modification to the receive function:
if data:
print('\n' + data)
sys.stdout.write(readline.get_line_buffer())
sys.stdout.flush()
The '\n' in the print is so the incoming message doesn't land right on top of whatever's being typed. The flush is necessary so that you can write the current input without a newline but without it getting buffered. I might suggest adding a prompt (like '> ') to the input and the sys.stdout.write, just to make it clearer to the user what's happening.
Lastly, running this may mess up your terminal output. You might need to run reset afterwards. There's probably a way to clean up that prevents that but I don't know offhand what it is.

Cant receive data from socket

I'm making a client-server program, and there is problem with client part.
Problem is in infinite receiving data. I've tested this particular class, listed below, in a python interpreter. I've succesfuly(maybe not) connected to google, but then program stoped in function recvData() in data = self.socket.recv(1024)
class client():
def __init__(self, host, port):
self.host = host
self.port = port
self.socket = self.connect()
self.command = commands()
def connect(self):
'''
Connect to a remote host.
'''
try:
import socket
return socket.create_connection((self.host, self.port))
except socket.error:
print(":: Failed to connect to a remote port : ")
def sendCommand(self, comm):
'''
Send command to remote host
Returns server output
'''
comman = comm.encode()
# for case in switch(comman):
# if case(self.command.RETRV_FILES_LIST.encode()):
# self.socket.send(b'1')
# return self.recvData()
# if case():
# print(":: Got wrong command")
if (comman == b'1'):
self.socket.send(b'1')
return self.recvData()
def recvData(self):
'''
Receives all the data
'''
i = 0
total_data = []
while(True):
data = self.socket.recv(1024)
if not data: break
total_data.append(data)
i += 1
if i > 9:
break
return total_data
about commented part :
I thought problem in Case realization, so used just if-then statement. But it's not.

Your problem is that self.socket.recv(1024) only returns an empty string when the socket has been shut down on the server side and all data has been received. The way you coded your client, it has no idea that the full message has been received and waits for more. How you deal with the problem depends very much on the protocol used by the server.
Consider a web server. It sends a line-delimited header including a content-length parameter telling the client exactly how many bytes it should read. The client scans for newlines until the header is complete and then uses that value to do recv(exact_size) (if large, it can read chunks instead) so that the recv won't block when the last byte comes in.
Even then, there a decisions to make. The client knows how large the web page is but may want to send a partial data to the caller so it can start painting the page before all the data is received. Of course, the caller needs to know that is what happens - there is a protocol or set of rules for the API itself.
You need to define how the client knows a message is complete and what exactly it passes back to its caller. A great way to deal with the problem is to let some other protocol such as [zeromq](http://zeromq.org/ do the work for you. A simple python client / server can be implemented with xmlrpc. And there are many other ways.
You said you are implementing a client/server program then you mentioned "connected to google" and telnet... These are all very different things and a single client strategy won't work with all of them.

Simple client program

server.py is fine and I use array to send data:
for i in range(0,len(outputdata)):
connectionSocket.send(outputdata[i])
connectionSocket.close()
But my client don't work(print nothing):
#import socket module
from socket import *
serverName = '127.0.0.1'
serverPort = 9999
clientSocket = socket(AF_INET,SOCK_STREAM)
clientSocket.connect((serverName,serverPort))
request = raw_input('Input the filename:')
clientSocket.send(request)
while (clientSocket.recv(1024)):
print clientSocket.recv(1024)
clientSocket.close()
Why doesn't this work?

The command clientSocket.recv(1024) is a command that will fetch available data.
from Python Docs:
Receive data from the socket. The return value is a string
representing the data received. The maximum amount of data to be
received at once is specified by bufsize. See the Unix manual page
recv(2) for the meaning of the optional argument flags; it defaults to
zero.
To understand it better, your server is sending some data, the client receives that data and holds it in a buffer, the command will go to that buffer and pop the data (with max of 1024 at your code) so if you have 1024 bytes waiting in the buffer the first recv(1024) command will get all the data in the buffer and the second recv(1024) will be empty.
A good solution was suggested by #gnibbler.
Remember calling clientSocket.recv(1024) without saving it's return value it's like popping from a stack without saving the value.

The first clientSocket.recv(1024) is throwing away those bytes
while (clientSocket.recv(1024)):
print clientSocket.recv(1024)
You need to save them into a variable like this
while True:
data = clientSocket.recv(1024)
if not data:
break
print data