I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()
Related
I am trying to send TCP packets using socket in Python, and I can successfully send packets outside, I can see them using WireShark. However, as you can see that inside the while loop the code wait for ENTER key to be entered in order to send a packet. When I press ENTER one by one, I can see that my data (18 byte starts with 0xEFEB000000 in bytearray format) which is read from file, but not given here because I think it unnecessary and confusing, is sent packet by packet as I wanted. However, when I press and hold the ENTER button, it consecutively sends packets way faster than one by one. In this case, I can see in WireShark that some of the data are combined into a single TCP packet. I do not want this, and this is very strict unfortunately. How can I force socket to send the payload I gave packet by packet separately?
from socket import socket, AF_INET, SOCK_STREAM
if __name__ == "__main__":
CLIENT_SOCK = socket(AF_INET, SOCK_STREAM)
CLIENT_SOCK.connect(("192.168.1.10", 42000))
while True:
input("Press ENTER to send a TCP packet...")
# some file IO here to construct data variable,
# 18 byte bytearray format. starts with 0xEFEB000000
CLIENT_SOCK.sendall(data)
Single Packet with single payload
Single Packet with 5 payloads combined
TCP isn't message-based. It just returns the bytes written into the socket in the order sent, but due to network buffer can break the bytes up in whatever packets it chooses. Wrap the receiver in a socket.makefile object which will buffer read data, then .read(n) will read exactly n bytes unless the socket closes early. Example:
server.py
import socket
s = socket.socket()
s.bind(('', 5000))
s.listen()
c, a = s.accept()
# 'with' will close the client socket and makefile object
# when it exits.
with c, c.makefile('rb') as f:
while True:
data = f.read(18)
if not data:
break
print(data)
client.py
import socket
s = socket.socket()
s.connect(('localhost', 5000))
s.sendall(b'abcdefghijklmnopqr')
s.sendall(b'abc')
s.sendall(b'defghijklm')
s.sendall(b'nopqr')
s.sendall(b'abcdefghijklm')
s.sendall(b'nopqrabcdefghijklm')
s.sendall(b'nopqr')
s.close()
Server output:
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
b'abcdefghijklmnopqr'
Note: This may appear to work with a direct c.recv(18) but in a busy/complex network environment can fail to receive 18 bytes every time.
While trying to attempt to go for python based socket, I have 2 questions which I am not able to resolve. Kindly help me. Here is my sample socket code:
import socket
import threading
import chardet
bind_ip = '0.0.0.0'
bind_port = 9999
server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server.bind((bind_ip, bind_port))
server.listen(1) # max backlog of connections
print (('Listening on {}:{}').format(bind_ip, bind_port))
def handle_client_connection(client_socket):
request = client_socket.recv(4096 )
result = chardet.detect(request)
print(result)
print (request.decode(result['encoding']))
client_socket.send('ACK!'.encode(result['encoding']))
client_socket.close()
while True:
client_sock, address = server.accept()
print (('Accepted connection from {}:{}').format(address[0], address[1]))
client_handler = threading.Thread(
target=handle_client_connection,
args=(client_sock,) # without comma you'd get a... TypeError: handle_client_connection() argument after * must be a sequence, not _socketobject
)
client_handler.start()
The above one is server and the below is client:
import socket
client = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client.connect(('127.0.0.1', 9999))
client.send(str('test data').encode("utf-16"))
response = client.recv(4096)
print(response.decode("utf-16"))
Now the questions:
1) What is the meaning of the number in this statement: client.recv(4096)? What is 4096, is it bit or byte of kilobyte for data receiving?
2) Can I receive infinite data stream through the socket? Meaning, as this statement client.recv(4096), whatever is the menaing of 4096, may be byte, then the socket will receive the 4096 bytes of data only. I do not have a control on the size of data received through the socket, hence, can I generalize it to accept any size of data through socket?
Please help me get the answers to above queries. I tried the python documentation for socket but didn't found much. I guess I miss something. Please help me get through it.
According to documentation
"If a message is too long to fit in the supplied buffer, excess bytes may be discarded depending on the type of socket the message is received from."
found here
"https://linux.die.net/man/2/recv"
which was from python 3.6 docs
"he maximum amount of data to be received at once is specified by bufsize. See the Unix manual page recv(2)"
found here
"https://docs.python.org/3.6/library/socket.html"
so it is a bytes object and it may actually truncate the message depending on the protocol. So message integrity is handled by the next layer up. So presumably you would loose part of a UDP packet, and you would get either a retry or an additional packet for TCP if the message was too large to fit in the buffer.
I'm trying to create a simple server to client based chat program and the issue is that when I try to execute c.sendto(data,client) this error appears saying that Client is an int but it's a tuple containing the port number and the address. I'm I supposed to convert the tuple to bytes so I can send the message to the clients?
Server Script
import socket
clients = []
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1",7999))
s.listen()
print("Waiting for connection")
c, addr = s.accept()
while True:
data , addr = c.recvfrom(1024)
print(data)
if addr not in clients:
clients.append(addr)
print(clients[0])
if data:
for client in clients:
print(client)
c.sendto(data,client)
s.close()
Client Script
import socket
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
addr = ("127.0.0.1",7999)
s.connect(addr)
while True:
string = input(">>")
s.send(string.encode("utf-8"))
data =s.recv(1024)
s.close()
Server Output
The problem is that you're using sendto() with a connection-mode socket. I think you want c.send(data) instead.
Details:
The Python docs for sendto say "The socket should not be connected to a remote socket, since the destination socket is specified by address." Also the man page for sendto says "If sendto() is used on a connection-mode (SOCK_STREAM, SOCK_SEQPACKET) socket, the arguments dest_addr and addrlen are ignored (and the error EISCONN may be returned when they are not NULL and 0)." I somewhat suspect that this is happening and Python is misreporting the error in a confusing way.
The sockets interface and networking in general can be pretty confusing but basically sendto() is reserved for SOCK_DGRAM which is UDP/IP type internet traffic, which you can think of as sending letters or postcards to a recipient. Each one goes out with a recipient address on it and there's no guarantee on order of receipt. On the other hand, connection-mode sockets like SOCK_STREAM use TCP/IP which is a bit more like a phone call in that you make a connection for a certain duration and and each thing you send is delivered in order at each end.
Since your code seems to be designed for communication over a connection I think you just want c.send(data) and not sendto.
You must first convert the string that contains the IP address into a byte or a string of bytes and then start communicating.
According to the code below, your error will be resolved.
Make sure your code is working correctly overall.
string = '192.168.1.102'
new_string = bytearray(string,"ascii")
ip_receiver = new_string
s.sendto(text.encode(), (ip_receiver, 5252))
Referencing this example (and the docs): https://pymotw.com/2/socket/tcp.html I am trying to achieve bidirectional communication with blocking sockets between a client and a server using TCP.
I can get one-way communication to work from client->server or server->client, but the socket remains blocked or "hangs" when trying to receive messages on both the server and client. I am using a simple algorithm(recvall), which uses recv, to consolidate the packets into the full message.
I understand the sockets remain blocked by design until all the data is sent or read(right?), but isn't that what sendall and recvall take care of? How come disabling recv on either the client or server "unblocks" it and causes it to work? And ultimately what am I doing wrong that is causing the socket to stay blocked?
Here is my code, the only fundamental difference really being the messages that are sent:
recvall(socket)(shared between client and server):
def recvall(socket):
data = ''
while True:
packet = socket.recv(16)
if not packet: break
data += packet
return data
server.py (run first):
import socket
host = 'localhost'
port = 8080
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
s.listen(5)
while True:
(client, address) = s.accept()
print 'client connected'
try:
print recvall(client)
client.sendall('hello client')
finally:
client.close()
client.py:
import socket
s = socket.create_connection((args.ip, args.port))
try:
s.sendall('hello server')
print recvall(s)
finally:
s.close()
From my understanding (epiphany here), the main problem is that recv inside recvall is only concerned with retrieving the stream (in the same way send is only concerned with sending the stream), it has no concept of a "message" and therefore cannot know when to finish reading. It read all the bytes and did not return any additional bytes, but that is NOT a signal that the message is finished sending, there could be more bytes waiting to be sent and it would not be safe to assume otherwise.
This requires us to have an explicit indicator for when to stop reading. recv and send are only concerned with managing the stream and therefore have no concept of a message (our "unit"). This article has some great solutions to this problem. Since I am sending fixed-length messages, I opted to check that the length is as expected before finishing recv. Here is the updated version of recvall, note MSG_LENGTH must be defined and enforced in order for recvall to not block the socket.
def recvall(socket):
data = ''
while len(data) < MSG_LENGTH:
packet = socket.recv(BUFFER_SIZE)
if not packet: break
data += packet
return data
Bidirectional communication now works, the only catch being the client and server must know the length of the message they will receive, again this is not an issue in my case. This is all new to me so someone please correct me on terminology and concepts.
Possibly related questions that seem close but don't describe my issue as I understand it:
Reading all the data from a UDP socket
Python UDP socket semi-randomly failing to receive
Python raw socket listening for UDP packets; only half of the packets received
problem
Long file sent line by line doesn't go all the way through UDP over loopback.
long story
I have a long file consisting of lines and breaks that is identical to what I will get from another program over UDP locally. Let me emphasize that the program sending the packets will do so over UDP (there is no choice here), and cannot be feasibly modified to process ACK requests etc. while sending.
It looks like this (this is the tail):
StimulusTime 56398
Signal(0,2) -79.5457
Signal(0,4) -81.7426
Signal(0,6) -83.9978
Signal(0,9) -63.3755
Signal(0,11) -15.6045
Signal(0,13) 31.1299
Signal(0,16) 75.7539
Signal(0,18) 98.301
Signal(0,20) 98.301
Signal(0,22) 48.4546
Signal(0,25) 3.73159
Signal(0,27) -49.9798
Signal(0,29) -77.8449
Signal(1,0) -22.0332
Signal(1,2) -60.6384
Signal(1,4) -98.0858
Signal(1,6) -86.4579
Signal(1,9) -68.9173
Signal(1,11) -31.5552
Signal(1,13) 35.2906
Signal(1,16) 77.0686
Signal(1,18) 96.3836
Signal(1,20) 95.7246
Signal(1,23) 25.6074
Signal(1,25) -20.2101
Signal(1,27) -60.2933
Signal(1,29) -83.8169
Signal(2,0) -31.8826
Signal(2,2) -53.5045
Signal(2,4) -89.9895
Signal(2,7) -84.4503
Signal(2,9) -59.7016
Signal(2,11) -12.8569
Signal(2,13) 28.857
Signal(2,15) 58.0577
Signal(2,18) 96.4222
Signal(2,20) 79.783
Signal(2,22) 58.6463
Signal(2,25) -3.24883
Signal(2,27) -45.5
Signal(2,29) -88.8937
Signal(3,0) -18.6625
Signal(3,2) -53.3978
Signal(1,16) 58.784
Signal(1,17) 44.7782
Signal(1,18) 6.247
Signal(1,19) -12.0855
Signal(1,20) -33.7644
Signal(1,21) -49.4406
Signal(1,22) -67.5791
Signal(1,23) -92.0336
Signal(1,24) -93.9841
END
I wrote code that takes this file and sends it a line at a time over UDP locally, and then code that receives it and parses it based on the data type.
Sender:
import socket
import sys
# Sends udp test data piped in from STDIN to the listener.
# ex: cat sampleoutput.txt | python testsender.py
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
# Send from stdin
if len(sys.argv) < 2:
while True:
line = sys.stdin.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
# get from file arg
else:
myfile = open(str(sys.argv[1]), "r")
while True:
line = myfile.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
sock.close()
Listener:
import socket
from array import array
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size arg
print data
# write to file for later testing
# file = open("testdummy.txt", "a")
# file.write(data)
if data == "END\n":
break
I was able to use the above listener to produce the test file from the original program, so it should work. Unfortunately, it fails around 500 lines of payload, as tested by tail -n 500 testdummy.txt | python testsender.py, although it's somewhat random. Specifically, the listener does not receive all of the sent lines before the sender exits, leaving it hanging, waiting for the "END\n" string.
As I understand it, the socket is already in blocking mode--how can I prevent this from occurring?
My first advice to you, Don't use UDP if you want to transfer files with the sequence of lines preserved, Use TCP if you don't want to code alot. The reasons are;
UDP is an unreliable Protocol, in the sense a Packet sent is not guaranteed to be received by the recipient.
UDP doesn't guarantee the sequence of packets being received, This is because UDP packets may go to recipient via several routes (Hops between computers). So latter sent packets can take a short route and reach the recipient before former sent packets. ("End\n" packet can come before other packets)
TCP on the other hand is reliable and sequence of packets received is guaranteed. Ideal for file transfer.
But don't worry File sharing applications like Bear Share, Bit Torrents make use of UDP but there are some additional coding you have to do.
You need to implement an Acknowledgement protocol, as in you need to have a unique id for each packet you send to the recipient and when the packet is received recipient should send an Acknowledgement packet back to the sender with that id saying the packet was received.
If in case the packet got lost and didn't reach the recipient (No Acknowledgement from recipient) you must resend the packet again (and again) until you get the Acknowledgement.
You can control the order by not sending the next packet until you get the Acknowledgement for the previous one.