I'm trying to send a stream of data via socket in Python. So far I manage to create a dummy_data_gen.py which sends a line containing 4 floats to the server.py. However, I'm still having issues in the stability of the all setup.
server.py:
import sys
import time
import socket
import numpy as np
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Bind the socket to the port
server_address = ('localhost', 5002)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
# Listen for incoming connections
sock.listen(1)
# Create a list to store the incoming data
data = []
while True:
# Wait for a connection
print >>sys.stderr, 'waiting for a connection'
connection, client_address = sock.accept()
try:
print >>sys.stderr, 'connection from', client_address
while True:
incoming_data = connection.recv(48).split(',')
print incoming_data
event = float(incoming_data[0]), float(incoming_data[1]), float(incoming_data[2]), float(incoming_data[3])
data += [event]
time.sleep(0.01)
finally:
# Clean up the connection
connection.close()
dummy_data_gen.py
import sys
import time
import socket
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Connect the socket to the port where the server is listening
server_address = ('localhost', 5002)
sock.connect(server_address)
file = '../data/myfile.txt'
# Simulating a real-time data stream at 100 Hz
try:
with open(file) as f:
for line in f:
sock.sendall(line)
time.sleep(0.01)
finally:
print >>sys.stderr, 'closing socket'
sock.close()
My problem is that sometimes the communication is working properly, however, I have situations where I receive more data per line than I should. In the following output example the first 7 lines are correct, however the following lines are incorrect and therefore problematic:
['391910745379', '24.134277', '-1.9487305', '-117.373535', '\n']
['391920745379', '24.434082', '-1.3491211', '-117.373535', '\n']
['391930745379', '23.68457', '-0.5996094', '-116.62402', '\n']
['391940745379', '24.434082', '-1.0493164', '-115.57471', '\n']
['391950745379', '24.134277', '-1.0493164', '-116.47412', '\n']
['391960745379', '23.234863', '-1.0493164', '-116.47412', '\n']
['391970745379', '24.583984', '-0.89941406', '-116.92383', '\n']
['391980745379', '23.384766', '-0.2998047', '-116.62402', '\n39']
['1990745379', '23.68457', '-0.5996094', '-115.72461', '\n39200']
['0745379', '23.834473', '-0.44970703', '-115.87451', '\n392010']
['745379', '23.534668', '-1.0493164', '-114.9751', '\n392020745']
['379', '23.384766', '-1.7988281', '-115.72461', '\n39203074537']
['9', '22.935059', '-0.44970703', '-114.9751', '\n392040745379', '']
I tried to play with the connection.recv bytes but I'm still facing this issue.
EDIT1: Following some suggestions I modified the server.py as follows:
del_message = '\n'
del_stream = ','
while True:
_buffer += connection.recv(1)
if del_message in _buffer:
incoming_data = _buffer.split(del_stream)
event = float(incoming_data[0]), \
float(incoming_data[1]), \
float(incoming_data[2]), \
float(incoming_data[3])
This approach seems to solve my issue, however the performance is extremely slow. My files contains approximately 6300 lines that were actually sent in 70 seconds (time interval at which the socket was closed on my dummy data generator). However, I took almost 10 minutes to receive all of the 6300 lines. It seems also that I receive more samples per second on the beginning rather that on the end of the stream.
If you have a message protocol that terminates messages with a newline, you need to write some code to implement that protocol. It won't work by magic.
You need a "receive a message" function, where "message" is defined as "a sequence of bytes delimited by a newline". You've never written any such function, so you're not receiving messages but just the chunks of bytes you're sending.
Related
From WireShark I can see that a UDP packet is being sent to my server containing data that I need.
Raw packet data from wireshark:
55aa001e03840000c864da6ea1a613422f7b18835780f2ac1f657fb39264405cc8a9f6f09291a7532b69ca128d549112002674027daf6fedf73ce5bd73e6e1e31a108970b8881da0c4a19149eed46a3a40870ed89e10a5e197283afa554a91d5ee6a24c96d375862da4a909832890049ce01778d63dddf3ae17d800421032f971d2d671232d1b64d45589b44663a39763575236160ca6bc26e66523cef74ffa1d248ed35329232132341442a0095ce012f7cdfacf71be1bd534d639e2ea78f6a293d46f7066886518cd6777d9b33d0b0d1f2c55701842c88740d0251ff6964487b5b26d949021229124900b26f012b84e2abdffac3bff9c4433d76079d03cdabc303a22045464f80cd25dd7f0bd2f59bf139c61484847621533df77092b06bb59a69a40a4448934800558d012774e29cf7dbd5bd657c87997a53be922e82a42d0736664caf04b66bbcdf6d2cbf3746eb041116768081a4b6f99092a96ed312dad9129110952400928f01757ce79bf71bd4bd2a41ef49a8bc12c940d96e44045032ea5ed74f6b5f76640fa9741ae80df8a4726d909082fb18da3259c4313365921425c9848019ca00e773635ad63791f95340d098253a6474c0cf05c8590402240140bd8286f7a7be0cab23e67dbb1d5feadbbe5225bbc9fe93b23532a49192c95044a59401006b093ae7f85679d779aa4124980000004190822c6441a38230783f0d7fb72d796fb284d74a93f6eda6ad09a741decafecce4a852499a484d8cd4424600805c685a6119d7f9de37b557ee34e978d6d288fe3d06bca9ff8e2a265eca4328c141b000440dd86a1f8499be010a18ff58d9112691c9646529582624c500980d02ef84ef8cdfbdd6bb3e84abdee3c36adfa9ac736e5b656d1de1eb7573e42cb2d30c2f28400cb8164080892a06ff28b591ad724924242551a9310a0019ae01ab6363abf73de3bd8ed73857beeab76ccd9a671cd761680070585e3e2297e3bc10d425094a1634308d2a8bd4ff6de9a2d32508538889244993a400338e012563ef69f73bd379a9cf47c9cb2a169c64cc6d81232c78125f4978265e31d3758f1b5f11dadabc72ffb3254ffecae5a93251a9d84809c998a844004dad01a56aeb6af75cd7bd2fd6b383467e23b5a3bac5c2d800288dd7e661451a4e887a906563a894d5a6b445c96d26df40d5a6a63125e94d06102504990069ad012754f32bf75ce7bfb18b8cffd72cc18eed938d4e02761a260fa466a129b06af27602a422fab2b6a705da8ca2deaad4ab5e63cc
Using python, how do I remove the first 20 characters:
55aa001e03840000c864
and save the rest?
Current code:
import socket
import sys
# Create a UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Bind the socket to the port
server_address = ('0.0.0.0', 8484)
sock.bind(server_address)
print >>sys.stderr, 'starting up on %s port %s' % server_address
while True:
print('\nwaiting to receive message')
data, address = sock.recvfrom(10240)
Since the within the data variable is of type bytes, which I think can be translated to something like b'<string_with_your_packet_data>', I think you can use slicing just as you would do on a standard string and it should work just fine.
So something among the lines of
while True:
print('\nwaiting to receive message')
data, address = sock.recvfrom(10240)
sliced_packet = data[20:] # trimming your first 20 characters from your data
I created socket for two PC, one is Raspberry Pi and the other one is my laptop. I just connected two then I send string to test the connection. If I send a character "q" from the RPi, my PC should break out of the loop and close the connection but it does not. The part print("Listening") is still running. Why? See code below.
import socket
import time
# IP address of this PC.
TCP_IP = '192.168.137.1'
# Port.
TCP_PORT = 5005
# Size of buffer.
BUFFER_SIZE = 1024
# Create a socket, connect and listen to it.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print('Connection address:', addr)
while 1:
print("Listening")
data = conn.recv(BUFFER_SIZE)
data = data.decode()
if data=='q':
break
if data:
print ("Received data:", data)
# Echo back.
conn.send(data.encode())
time.sleep(1)
print("It breaks.")
conn.close()
s.close()
TCP is a stream oriented protocol. So data transmitted is a stream not a sequence of messages. So when you expect data to be q it actually is some_data_sent_before_q_and_finally_q.
The simplest way to repair the code is to use if data.endswith('q') instead of if data=='q'. May work and may not depending on how you actually use the connection. For example, this approach may fail with some_data_sent_before_q pretty long pause more_data_and_q and with some_data_sent_before_q_and_finally_q_plus_something_else_why_not.
Little bit more advanced way to solve the problem is to divide the stream into messages with separators - message_1<separator>message_2<separator>q<separator>. This method will allow you to treat every message separately.
here is my code, the client should be blocking in recv (it expect 256 characters), cause the server just send 5 character to it, but recv return, any idea?
#-----------------------------------------------------------
# server.py
import socket
import sys
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(('localhost', 10000))
sock.listen(1)
while True:
connection, client_address = sock.accept()
try:
connection.send('hello'.encode('utf-8'))
except Exception:
connection.close()
#-----------------------------------------------------------
# client.py
import socket
import sys
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.connect(('localhost', 10000))
sock.setblocking(True)
try:
data = sock.recv(256)
print('received "%s"' % data.decode('utf-8'))
finally:
sock.close()
sock.recv() returns because something was received on the socket, or if len(data) is 0, because the server closed the connection.
Expecting 256 bytes do not mean they will be received in one go. In your example, 256 is the maximum number of bytes.
The blocking behavior means: wait until something is received, not to wait until all that is expected is received.
If you are sure your server will send 256 bytes you can make a loop:
data = ''
while len(data) < 256:
data += socket.recv(256)
If you cannot receive 256 bytes in one go, it means your network is maybe quite unstable (wifi ?) or the server or your client runs on an embedded platform with very small buffers ? Or the server is sending small chunks...
I need to simulate a UDP server, which sends content of a text file line by line in an endless loop. I have written the code below but at the other end I do not receive anything (The other side is Qt code and I am sure it works):
import socket
import time
# setup UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sa = ('localhost', 9763)
sock.bind(sa)
filename = '/home/saeid/Documents/data.txt' # file to read
numlines = sum(1 for line in open(filename)) # get number of lines in file
# get line of a file
def get_line(line):
with open(filename) as fp:
for i, l in enumerate(fp):
if i == line:
return l
# main loop
while True:
currline = 0
while currline < numlines:
sock.sendto(get_line(currline), sa)
currline += 1
time.sleep(1)
I am no python pro and cant figure out the problem :(
For starters this is messed up:
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sa = ('localhost', 9763)
sock.bind(sa)
...
sock.sendto(get_line(currline), sa)
By binding you basically say "I want to listen on that host/port". But then you send the data to the same host/port. I assume that there's some other destination address, e.g. sock.sendto(get_line(currline), ('my_host', 1234)). By the way, why are you binding to the address anyway? sock.bind(sa) line is unnecessary, remove it.
The other thing is that your file reading code is extremely inefficient and difficult to read (it took me a while to understand what's going on). Try something like this:
with open(filename, 'r') as fo:
while True:
for line in fo:
sock.sendto(line, DEST_ADDRESS)
time.sleep(1)
fo.seek(0) # go back to the begining of the file and repeat
and get rid of get_line function.
That's at least what I've came up with after reading your description. If you don't want to inifinitely send the same file then you can get rid of while True: loop and fo.seek(0) call.
Uncle google came up with User Datagram Client and Server
User Datagram Client and Server
The user datagram protocol (UDP) works differently from TCP/IP. Where TCP is a stream oriented protocol, ensuring that all of the data is transmitted in the right order, UDP is a message oriented protocol. UDP does not require a long-lived connection, so setting up a UDP socket is a little simpler. On the other hand, UDP messages must fit within a single packet (for IPv4, that means they can only hold 65,507 bytes because the 65,535 byte packet also includes header information) and delivery is not guaranteed as it is with TCP.
Echo Server
Since there is no connection, per se, the server does not need to listen for and accept connections. It only needs to use bind() to associate its socket with a port, and then wait for individual messages.
import socket
import sys
# Create a TCP/IP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Bind the socket to the port
server_address = ('localhost', 10000)
print >>sys.stderr, 'starting up on %s port %s' % server_address
sock.bind(server_address)
Messages are read from the socket using recvfrom(), which returns the data as well as the address of the client from which it was sent.
while True:
print >>sys.stderr, '\nwaiting to receive message'
data, address = sock.recvfrom(4096)
print >>sys.stderr, 'received %s bytes from %s' % (len(data), address)
print >>sys.stderr, data
if data:
sent = sock.sendto(data, address)
print >>sys.stderr, 'sent %s bytes back to %s' % (sent, address)
Echo Client
The UDP echo client is similar the server, but does not use bind() to attach its socket to an address. It uses sendto() to deliver its message directly to the server, and recvfrom() to receive the response.
import socket
import sys
# Create a UDP socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
server_address = ('localhost', 10000)
message = 'This is the message. It will be repeated.'
try:
# Send data
print >>sys.stderr, 'sending "%s"' % message
sent = sock.sendto(message, server_address)
# Receive response
print >>sys.stderr, 'waiting to receive'
data, server = sock.recvfrom(4096)
print >>sys.stderr, 'received "%s"' % data
finally:
print >>sys.stderr, 'closing socket'
sock.close()
Client and Server Together
Running the server produces:
$ python ./socket_echo_server_dgram.py
starting up on localhost port 10000
waiting to receive message
received 42 bytes from ('127.0.0.1', 50139)
This is the message. It will be repeated.
sent 42 bytes back to ('127.0.0.1', 50139)
waiting to receive message
and the client output is:
$ python ./socket_echo_client_dgram.py
sending "This is the message. It will be repeated."
waiting to receive
received "This is the message. It will be repeated."
closing socket
$
You need to change the destination address or port in your call to sendto:
peer_address = ('localhost', peer_port)
sock.sendto(get_line(currline), peer_address)
To which address and port does the Qt application bind?
I am learning socket programming using python. my first assignment is to a write a client.py and a server.py. The client sends a message to server. The server receives the message of 16 bytes each time. After it has received the entire message, it will send the same message back to client.
so it is very simple. The server has backlog of 1. After the server sends the message to client, the connection to client close and the server should be open to receive new connection.
my current code fails in the last step. It is not open to receive new connections. It is throwing error. I even figured out the error. but I do not know how to fix this.
The error comes from server.py because I call for sock.accept() but I have closed the sock.
Let me explain my server.py code: I have two while loops. The outer loop looks for new connection, and the inner loop looks process request from connections i.e it simply receives data, wait till everything is received and send it back to client and finally close the connection.
I am asked not to change the structure of two while loops but just implement them.
Any thoughts or ideas on this:
client.py
import socket
import sys
def client(msg, log_buffer=sys.stderr):
server_address = ('localhost', 10000)
sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM,socket.IPPROTO_IP)
sock.connect(server_address)
print >>log_buffer, 'connecting to {0} port {1}'.format(*server_address)
try:
print >>log_buffer, 'sending "{0}"'.format(msg)
sock.sendall(msg)
chunk = ''
done=False;
while not done:
chunk+=sock.recv(16)
if chunk==msg:
done=True
print >>log_buffer, 'received "{0}"'.format(chunk)
finally:
print >>log_buffer, 'closing socket'
sock.close()
if __name__ == '__main__':
if len(sys.argv) != 2:
usg = '\nusage: python echo_client.py "this is my message"\n'
print >>sys.stderr, usg
sys.exit(1)
msg = sys.argv[1]
client(msg)
Server.py
import socket
import sys
def server(log_buffer=sys.stderr):
# set an address for our server
address = ('127.0.0.1', 10000)
sock=socket.socket(socket.AF_INET,socket.SOCK_STREAM,socket.IPPROTO_IP)
sock.setsockopt(socket.SOL_SOCKET,socket.SO_REUSEADDR,1)
# log that we are building a server
print >>log_buffer, "making a server on {0}:{1}".format(*address)
sock.bind(address)
sock.listen(1)
try:
# the outer loop controls the creation of new connection sockets. The
# server will handle each incoming connection one at a time.
while True:
print >>log_buffer, 'waiting for a connection'
conn,add=sock.accept()
addr=(conn,add)
try:
print >>log_buffer, 'connection - {0}:{1}'.format(*addr)
# the inner loop will receive messages sent by the client in
# buffers. When a complete message has been received, the
# loop will exit
data = ''
while True:
recvdata=conn.recv(16)
print recvdata
data+=recvdata
print >>log_buffer, 'received "{0}"'.format(data)
print >>log_buffer, "len of received data: {0}".format(len(recvdata))
if len(recvdata)<16:
print >>log_buffer,"sending data"
conn.sendall(data)
break
conn.close()
finally:
sock.close()
except KeyboardInterrupt:
sock.close()
if __name__ == '__main__':
server()
sys.exit(0)
I runpython server.py in one terminal andpython client.py "This is the first message. send me back"` in a different terminal. The client connection is lost normally as expected. But I get the following error at server side (towards the end):
making a server on 127.0.0.1:10000
waiting for a connection
connection - <socket._socketobject object at 0x100849c20>:('127.0.0.1', 50626)
sairam hopefully
received "sairam hopefully"
len of received data: 16
this works lets
received "sairam hopefully this works lets"
len of received data: 16
c
received "sairam hopefully this works lets c"
len of received data: 2
sending data
waiting for a connection
Traceback (most recent call last):
File "echo_server.py", line 89, in <module>
server()
File "echo_server.py", line 39, in server
conn,add=sock.accept()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 202, in accept
sock, addr = self._sock.accept()
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/socket.py", line 170, in _dummy
raise error(EBADF, 'Bad file descriptor')
socket.error: [Errno 9] Bad file descriptor
You are closing sock inside your while loop. Don't do that. sock is your long-lasting server socket, which needs to remain open to listen for new connections. conn is your ephemeral socket, which needs to remain open only the length of a single connection.
Close conn after each connection, close sock when the server needs to terminate.
More simply, replace these lines:
finally:
sock.close()
with
finally:
conn.close()
What you are trying to do is a simple echo server, which I believe you can implement much more simply.
Server:
import socket
host = ''
port = 50000
backlog = 5
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host,port))
s.listen(backlog)
while 1:
client, address = s.accept()
data = client.recv(size)
if data:
client.send(data)
client.close()
Client:
import socket
host = 'localhost'
port = 50000
size = 1024
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((host,port))
s.send('Hello, world')
data = s.recv(size)
s.close()
print 'Received:', data