We Keep Coding

Continous receiving & sending on a socket

I'm working on assignment where I need to connect to a server (no details of server are disclosed), capture the reply, modify it and send it back for verification.
I have created following code which does what I need, but the issue is that after 1st correct reply, server sends another.
Code:
# -*- encoding: utf-8 -*-
import socket
from decryptmsg import decryptmsg
from cleanmsg import cleanmsg
#connection
ip="<IP>"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
def recvall(sock):
BUFFER = 8192
data = b''
while True:
part = sock.recv(BUFFER)
data += part
if len(part) < BUFFER:
break
return data
while True:
print "[+] start communication"
data = recvall(sock)
print data
data = cleanmsg(data)
if data != None:
valmis = decryptmsg(str(data))
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis)
continue
When I hit the second question, I get the input captured fine with this code and processed as expected, but when I try to send the 2nd reply back I get error:
Traceback (most recent call last):
File "challenge.py", line 28, in <module>
sock.send(valmis)
socket.error: [Errno 32] Broken pipe
If I do not close or shutdown the socket, no reply is ever sent to server.
How can I tell my client to send the message and wait for reply without socket.shutdown? Or if I need to open new socket for each loop, how should the loop be constructed? The reply from server changes each time so if I open new connection completely and request for data, I get new reply and the process starts from beginning again.
UPDATE:
the issue seems to be when trying to receive the second reply from server, only the first line of message is received by client.

How do you know it does not send anything? I modified your code a bit (there is something odd in the else: clause, I will come back to that later).
import socket
#connection
ip="localhost"
port=4000
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect((ip,port))
while True:
data = sock.recv(8192)
if not data: break;
print data
if data != None:
valmis = data
if valmis == None:
print "[-] no results"
break
else:
print "[+] sending message... "
sock.send(valmis) #this never gets sent without
continue
Basically this is a stripped version of your code - no decrypting or external functionality. It just sends back whatever it receives from the server.
Then I ran a "server" with ncat:
ncat -l 4000
start your program and start typing in lines (1, 2, 3, 4 etc) and this happens at "server". The client promptly echoes my messages back:
test#xyzzy:/tmp$ ncat -l 4000
1
1
2
2
3
3
And this happens at the client:
test#xyzzy:/tmp$ python so.py
1
[+] sending message...
2
[+] sending message...
3
[+] sending message...
To me it looks this code works fine. If the server does not receive your reply, it might be that there is a problem on the server side. It might for example expect a terminator character in the response. Does your cleanmsg clean the message too much and for example remove a trailing newline and the server expects to receive one?
There is a problem in your original else clause as you do another sock.recv() there. Which means after receiving a reply, you block there to wait for the next message from server and when you do receive one, you will continue your loop and hit sock.recv() again. The second message was consumed already in your else clause.
This may be intentional if your server somehow acknowledges your decryption. If your protocol is this:
server -> client (request)
client -> server (decrypted message)
server -> client (some kind of acknowledgement - unclear from your code)
server -> client (request 2)
etc.
Then you have probably hit the issue in Jason's comment. TCP sockets are completely agnostic to the concept of a message. They just transmit data. When your code hits sock.recv(), one of five things can happen:
There is nothing in the socket and the call blocks
There is a full "message" and only that in the socket and you receive that
There is a partial message and you will receive that. Either because the message is over 8192 bytes, or your code just decides to read when the server has only transmitted some of the message data.
There are two or more complete "messages" waiting and you will receive them all.
As four, but the last message is partial
Always when operating with TCP sockets, you must cater for scenarios 2-5. You must parse the data, make sure everything is there, and if not, wait for more. And if there was more than you expected, process them accordingly. If they are complete messages, process them. If the last message is partial, process everything else and wait for more.
If messages seem to "disappear" in a self-made communication protocol using TCP sockets, 99% of the problems are caused by making the assumption that sockets would know or care what is the structure of your "message". A very common mistake is to read your socket empty and ignore everything you received after your first message.
Hope this is helpful. Your code without the additional recv seems to work fine from socket communication perspective - no need to shut down the socket. It may be a server side issue, protocol issue or message parsing problem.
For this reason, always have only one recv call for your socket. Even if you are expecting some kind of an acknowledgement instead of a new message, have only one place where you process socket data. Then do some kind of a conditional there to detect what kind of a message you received and then decide what to do with it.

Sending a string to a program

I am trying to send a string from a program 1 to another program program 2, both in python 3
e.g.
#_____________________________________1.py
a = input('Type in a string: ')
# send somehow a string a from this program
# to the second program
I want to somehow send a string a to my second program so it will print out a:
#_____________________________________2.py
# receive somehow a string from the first
# program and store it in a
print(a)
How do I do this?
I am still a beginner programmer and would love it if you could help me.
I need to be able to enter the string in 1.py
I need then be able to access the string that I entered from 2.py.
I have to have them as two separate files.
ANSWER:
I found a way to solve this.
import subprocess
username = input()
subprocess.Popen(['python.exe', 'file.py', username])

You have many way communicate between two or N python program, Ex:
Socket
Database - MySQL, Mongodb, SQL Server... etc
or maybe you can try ZeroMQ

There are multiple ways to do that, you could use socket, file, pipe, shared-memory, message, ... to transfer a string from one process to another.
As an example of using messages, ZeroMQ provides an easy messaging library to do that smarter, than with system (raw, low level) sockets:
for more details look into http://zguide.zeromq.org/
A HelloWorld server example:
import time
import zmq
context = zmq.Context()
socket = context.socket(zmq.REP)
socket.bind("tcp://*:5555")
while True:
# Wait for next request from client
message = socket.recv()
print("Received request: %s" % message)
# Do some 'work'
time.sleep(1)
# Send reply back to client
socket.send(b"World")
A HelloWorld client example:
import zmq
context = zmq.Context()
# Socket to talk to server
print("Connecting to hello world server…")
socket = context.socket(zmq.REQ)
socket.connect("tcp://localhost:5555")
# Do 10 requests, waiting each time for a response
for request in range(10):
print("Sending request %s …" % request)
socket.send(b"Hello")
# Get the reply.
message = socket.recv()
print("Received reply %s [ %s ]" % (request, message))
With files, you write a file with program A then poll on it with program B.

# file_1.py
def get_input():
return input('Type in a string: ')
# file_2.py
from file_1 import get_input
print(get_input())

Most common way of two programs communicating together is through http, tcp or other protocol. The same way as your browser (one program) communicates with the web server (another program).
You can send http request from one program and the second has to listen for that.
If you want more info, look for SOA. It is a bit more complicated than that, so if you have any questions, ask.

I found the answer.
import subprocess
username = input()
subprocess.Popen(['python.exe', 'file.py', username], subprocess.creationflags=CREATE_NEW_CONSOLE)

python based server client for dynamic calculater

i created a python based client for netcat server which runs on port 20122 ,which runs a service for a game (dynamic calculator) it produce random mathematical expression (1+34+9+0=?) but due to time restriction the connection will terminate automatically so it have to replay quickly so i created a socket program on python which will store the out from server on string and this computer calculate and feed back to server it self,.but i did so far and executed the connection sticks in " you are connected to game server" nothing else kindly help me to correct my code .
the client code
My client code
import socket
def netcat(replay):
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(('192.168.56.101', 20055))
s.shutdown(socket.SHUT_WR)
rec_data = []
while 1:
data=s.recv(1024)
exp=str(data)
replay = input(str(exp))
print replay
if not data:
break
rec_data.append(data)
return rec_data
s.sendall(replay)
if __name__ == '__main__':
replay = ''
recved = netcat(replay)
print recved[1]
any one can tell me how i can extract mathematical expression only from data and execute it
thanks in advance

The s.shutdown(socket.SHUT_WR) immediately after the s.connect() is out of place, since you want to s.sendall() later.

Drop Incoming 'packets' for Datagram Socket

this is question is really focused on my problem and not relative to any of the other question I could find on this topic.
PSA: When I say "packet" I mean a full string received in a single socket.recv(maxsize)
I developed similar code for the same result on Java (my pref language) and it is ok, now I have to do in python.
I have two processes that run in parallel:
1-Normal client socket connected to a specific IP
2-A "client" Datagram socket binded to "ALL" IPs.
The normal socket is working correctly as I expect, while the datagram not.
I continuosly receive packets from a server (not mine and not opensource) at a rate of more than 5 per second, but I want to process only one of them every 3 seconds. In java I did just a "sleep" and it was ok, I was getting only the last live packet, while in Python with a "time.sleep(3)" the packets are queued (I don't know how and where) and not dropped.
I HAVE to drop them because those are not need and I have to do an HTTP call between one and the other so I can't fire an HTTP post for every set of data received at that rate!
here it is my "code" for the listening socket, some comments are for private code:
def listenPositions():
lsSocket = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
lsSocket.bind(("0.0.0.0", 8787))
lsSocket.setblocking(0)
try:
while True:
ready = select.select([lsSocket], [], [], 1)
if ready[0]:
lsSocket.settimeout(1)
recvData = lsSocket.recv(16384)
if len(recvData) != 0:
recv = recvData[0:len(recvData)].decode("utf-8")
#print("LS: Received: " + recv)
strings = filter(None, str(recv).split('\n'))
print("Strings count=" + str(len(strings))+ ": " + str(strings))
for item in strings:
#parse the received strings as json and get the items
jsonPosition = json.loads(item)
strId = jsonPosition["id"]
coordinates = jsonPosition.get("coordinates")
if coordinates is None:
continue
print("coordinates not null:" + str(coordinates))
#DO THE HTTP POST REQUEST
time.sleep(3) #Pause the system for X seconds, but other packets are queued!
else:
print("LS: Received empty")
else:
print("LS: No data, timeout")
except Exception as e:
print(e)
#handle exceptions...
print("Exception, close everything")

When you have an open socket, all correctly addressed packets should be delivered to the application. We want to have our network connections as realiable as possible, don't we? Dropping a packet is an action of last resort.
If you want to get a packet only from time to time, you could create a listening socket, get a packet and close the socket.
However there is nothing easier than ignoring a packet. Just skip its processing and move on. The code below is incomplete, but hopefully expresses what I mean.
TIMEOUT = 1.0
INT = 3.0 # interval in seconds
# create udp_socket
last = time.time() - INT
udp_socket.settimeout(TIMEOUT)
while True:
try:
packet = udp_socket.recv(MAXSIZE)
except socket.timeout:
# handle recv timeout
continue # or break, or return
except OSError:
# handle recv error (Python 3.3+)
break # or continue, or return
now = time.time()
if now - last >= INT:
# process the packet
last = now
Please note that the select is not needed if you read only from one source.

Python: socket.recv() doesn't receive push messages

Python: socket.recv() doesn't receive push messages
Hello,
I'm coding a socket based IMAP client in Python3 which successfully establishes a connection to the server, succussfully transmits the IDLE command but then fails to receive incoming data from the server.
If you are wondering why I do not use libimap or sth., the answer is easy: I just want to implement an IDLE command-supporting python client which must be written without that library.
An extract:
import socket
def runIMAPPeek():
#socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.settimeout(29 * 60)
#connection
s.connect((IMAP_SERVER , 443))
#login
data = b"a1 LOGIN " + USER + b" " + PASSWORD + b"\n"
s.sendall(data)
reply = read(s)
#Idle loop
#As in RFC 3501 connection will be reinitialized every 29 minutes
while True:
# Idle command
print("#Sending IDLE...")
data = b"a2 IDLE\n"
s.sendall(data)
reply = read(s)
if reply.startswith("+ idling"):
print(" #idling.")
else:
print(" #Unexpected answer: {}".format(reply))
#sys.exit()
# waiting for incoming mails ----------------------------------
try:
push_msg = read(s)
# got push message = new message arrived
getNewEnvelope(s, push_msg)
except socket.timeout:
# timeout
print(" #timeout. Reinitializing IDLE...")
#TODO: except (KeyboardInterrupt, SystemExit)
# Quit Idle
data = b"DONE\n"
write(s, data)
reply = read(s)
if reply.startswith(prefix_str + " OK"):
print(" #quit idling.")
else:
print(" #Unexpected answer: {}".format(reply))
#sys.exit()
def read(s):
"""Read socket data, print it, convert to string, replace new lines
and return it.
"""
print("#Receiving...", end=" ")
reply = s.recv(4096)
reply = str(reply)[2:-1] #convert and remove byte indicators
reply = reply.replace("\\r\\n", "\n")
print(reply)
return reply
The problem is marked with the "----". Although messages are received in the mailbox, python does not react but remains in the idling/receiving state. In fact, the print line above the s.recv() command isn't even printed.
I tried everything successfully with Telnet, so there is no server problem.

In addition to my comment above, you have never selected INBOX. You will not receive any push messages, because you haven't told it what folder you want. Technically, IDLE is not valid in the unselected state.

Constructs like this one:
if reply.startswith("+ idling"):
are completely non-compliant. The IDLE RFC specifies that the client shall expect a continuation request, not this particular string (which also happens to be a continuation request).