I have a central data feed that I want to redistribute to many clients. The data feed produces approx. 1.8 kB/s. Currently I'm writing the feed to a file and each client reads off the end of the file. Something about this just seems wrong. Here is pseudo code for what I have now...
The feed:
o = open('feed.txt','a',0) #no buffering, maybe line buffer would be better
while 1:
data = feed.read(8192)
data = parse_data(data)
o.write(data)
time.sleep(0.01)
The server (each client connects in a new thread):
feed = open('feed.txt','r')
feed.seek(-1024,2)
while 1:
dat = feed.read(1024)
if len(dat)==0:
# For some reason if the end of the file is reached
# i can't read any more data, even there is more.
# some how backing off seems to fix the problem.
self.feed.seek(-1024,2)
self.feed.read(1024)
buffer += dat
idx = buffer.rfind('\n')
if idx>0:
data = buffer[:idx]
buffer = buffer[idx+1:]
for msg in data.split('\n'):
client.send(msg)
time.sleep(0.01)
What I'd like to do is just replace the file with a socket and write the messages directly to multicast packets. Any time a new client connects to the server I just spin up a new thread and start listening for the multicast packets. Are there any standard design patterns to handle this case?
Even simpler, just have all clients multicast on the same port. Then your server doesn't even need to track pseudo-connections.
We use a similar scheme for some of the software on our internal network, based on the fact that multicasting is "mostly reliable" on our networking infrastructure. We've stress tested the load and don't start dropping packets until there's over 30K messages/sec.
#!/usr/bin/python
import sys
import socket
ADDR = "239.239.239.9"
PORT = 7999
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((ADDR,PORT))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
while True:
data, addr = sock.recvfrom(2048)
print data
sys.stdout.flush()
Related
I am working on a test tool for an existing piece of SW that outputs messages on multicast ip of 240.240.240.1 port 2349. The IP interface on which I am receiving the multicast messages is 192.168.0.4. I can observe these messages being pushed out onto the network via wireshark. Even netstat -ng shows the subscriptions. However, the receiver I wrote in python is not picking them up. I did borrow the code below from another thread, with the attempt to modify it to work in my situation.
import socket
import struct
import sys
multicast_group = '240.240.240.1'
server_address = (multicast_group, 2345)
# Create the socket
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
# Bind to the server address
sock.bind(server_address)
# Tell the operating system to add the socket to
# the multicast group on all interfaces.
group = socket.inet_aton(multicast_group)
src = bytearray(socket.inet_aton("192.168.0.4"))
mreq = bytearray(group)
mreq.extend(src)
sock.setsockopt(
socket.IPPROTO_IP,
socket.IP_ADD_MEMBERSHIP,
mreq)
# Receive/respond loop
while True:
print('\nwaiting to receive message')
data, address = sock.recvfrom(1024)
print('received {} bytes from {}'.format(
len(data), address))
print(data)
print('sending acknowledgement to', address)
sock.sendto(b'ack', address)
Any help would be appreciated in figuring out what I am doing wrong.
So I figured out my own problem. I had checked iptables as being an issue early on, with no luck. But I fixed/modified several things along the way to address the issues being seen. Turns out the code above works just fine, it was firewalld/iptables rules blocking the receipt of multicast.
I'm trying to create some kind of client monitor, like a terminal, to receive data from a serial device over ethernet. I'm trying to use a socket with python, but the problem comes when I create the connection. I'm supposed to receive only one message from the server, and I get the whole message but split into two packets, like this:
Message expected:
b'-- VOID MESSAGE--'
Message received:
b'-- VOID'
b' MESSAGE--'
I don't know if is this a problem of buffer size, decoding or any other function
import socket
TCP_IP = '192.168.#.#'
TCP_PORT = ###
BUFFER_SIZE = 1024
data1=' '
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
while(1):
data = s.recv(BUFFER_SIZE)
print(data.decode('ASCII'))
s.close()
I've already tried with some codecs options like UTF-8, UTF-16 and ASCII but I still get the same result.
This function helped me to solve the issue.
while(1):
cadena += s.recv(1)
if (((cadena)[i])=='\n'):
print(cadena.decode('ASCII'))
cadena=b''
i=-1
i+=1
As it already was said - that's how sockets works.
Sent data could be splitted to chunks. So if you want to be sure, that you've received whole message that was sent you need to implement some kind of protocol, the part of which will be contain length of your message. For example:
First four bytes (integer) represents length of the message
Other bytes - content of the message
In such case algorithm to send a message will be look like:
Count length of the message
Write to socket integer (4 bytes) with message length
Write to socket content of the message
And reading algorithm:
Read bytes from socket and write read data to accumulator-buffer
Read first four bytes from buffer as integer - it will be message length
Check if buffer length is greater or equal "{message length} + 4"
If it's then read required amount of bytes and that will message that was sent.
Drop first "{message length} + 4" bytes from buffer
Repeat from second point
If it's not enough bytes to read message content repeat from first point.
One solution is to use UDP instead of TCP if you can live with the limitations:
There is a size limit, the data must fit into one packet
UDP is "unreliable".
A TCP connection transfer one single stream of bytes. OTOH UDP transfers individual datagrams (messages). If the sender sends N datagrams, the recipient shall receive the same N datagrams. Maybe out of order, maybe some will get lost, but each datagram is independent of all others.
Regarding the limitations, these are not so simple questions. There is plenty of information on these topics, just search.
The max size depends on factors like IPv4 or IPv6, fragmentation etc. and there is a best case and a worst case. Typically you can assume that one ethernet frame (for all headers + payload) is absolutely without problems.
The "unreliability" does not mean the quality of transfer is terrible. The network should work on "best effort" basis. It means there are no ACKs, timeouts and retransmits. You can live without it or you can add simple ACKs to your protocol.
You can use this example.
Server code: (read from client)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.bind((host, port))
s.listen(5)
while True:
print("Listening for connections...")
connection, addr = s.accept()
try:
buffer = connection.recv(1024)
response = ''
while buffer:
response += buffer.decode('ASCII')
buffer = connection.recv(1024)
print(response)
connection.close()
except KeyboardInterrupt:
if connection:
connection.close()
break
Client code: (send message)
#!/usr/bin/python3
from socket import socket, gethostname
s = socket()
host = gethostname()
port = 3399
s.connect((host, port))
print("Sending text..")
s.sendall(b'-- VOID MESSAGE--')
print("Done sending..")
s.close()
I have this sort of a catch-the-flag assignment where I need to connect to a server and a port, receive data, extract the number that was given in this data and resend it. I'm guessing I need to continue doing that until some flag will arrive.
I've tried doing that with python sockets (which I barely understand), and came up with this code:
import socket
import re
#AF_INET for IPv4, SOCK_STREAM for TCP
clientsocket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
# Tell the socket what IP and port number to connect to
clientsocket.connect(('35.157.111.68', 10172))
while 1:
# Recieve 1024 bytes of data.
data = clientsocket.recv(1024)
if not data: break
# Get only numbers from string
data = re.sub('\D',"", data)
# Send our result to the server.
clientsocket.send(str(data))
It's establishing a connection and receiving the data, but when it sends the number back it doesn't accept it properly and it does it for only one round (doesn't loop).
I am working on a Wi-Fi thermostat with an app running on my iPhone. It uses sockets to communicate with a python program using the built in socket library. The problem I'm having though, is that I would like to be able to change the temperature when the phone is not connected however the server will search for 1 second then time out (minimum time for the iPhone to connect) this doesn't allow me to adjust the temperature with a rotary encoder smoothly through. is there a way to listern in the background?
import sys
import socket
import os
import time
temp = 15
while True:
try:
HOST = '192.168.1.22'
PORT = 10000
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((HOST, PORT))
s.listen(1)
s.settimeout(1)
conn, addr = s.accept()
#conn.send(str(9).encode())
conn.send(str(temp).encode())
while True:
data = conn.recv(1024)
if not data: break
print(data)
print(data.decode())
data2 = data.decode()
if int(data2) in range(5, 31):
print(data2)
print("Setting the temperature to " + str(data2) + "°")
conn.send(("Setting the temperature to " + str(data2)).encode())
temp = data2
else:
print("Not in range")
conn.send("Not in range!\n".encode())
except:
print("No Connection!")
Thanks!
Your terminology is a bit confusing, but I think I see what you're trying to do. When your program executes accept with a 1 second timeout, that's not "searching" -- it's simply waiting for a client.
It sounds like you should split your code into three pieces:
actually adjusts the thermometer
listens for a connection from your iPhone client (TCP listener)
waits for ("listens for") adjustments from the rotary encoder.
I would put each in a separate thread (using the python threading module). Create a FIFO queue (with the queue module). Have the first (thermostat-adjuster) thread wait on the queue (Queue.get), and have the other two accept instructions (from TCP and rotary encoder, respectively) and feed commands through the queue (Queue.put).
Then you can get rid of the timeout in your TCP listener, and just have it block on the accept indefinitely. Likewise, your rotary encoder listener can simply wait for adjustments. And the thermostat-adjuster thread can just block waiting on instructions from the queue. This makes all three much easier to program, understand and troubleshoot.
Possibly related questions that seem close but don't describe my issue as I understand it:
Reading all the data from a UDP socket
Python UDP socket semi-randomly failing to receive
Python raw socket listening for UDP packets; only half of the packets received
problem
Long file sent line by line doesn't go all the way through UDP over loopback.
long story
I have a long file consisting of lines and breaks that is identical to what I will get from another program over UDP locally. Let me emphasize that the program sending the packets will do so over UDP (there is no choice here), and cannot be feasibly modified to process ACK requests etc. while sending.
It looks like this (this is the tail):
StimulusTime 56398
Signal(0,2) -79.5457
Signal(0,4) -81.7426
Signal(0,6) -83.9978
Signal(0,9) -63.3755
Signal(0,11) -15.6045
Signal(0,13) 31.1299
Signal(0,16) 75.7539
Signal(0,18) 98.301
Signal(0,20) 98.301
Signal(0,22) 48.4546
Signal(0,25) 3.73159
Signal(0,27) -49.9798
Signal(0,29) -77.8449
Signal(1,0) -22.0332
Signal(1,2) -60.6384
Signal(1,4) -98.0858
Signal(1,6) -86.4579
Signal(1,9) -68.9173
Signal(1,11) -31.5552
Signal(1,13) 35.2906
Signal(1,16) 77.0686
Signal(1,18) 96.3836
Signal(1,20) 95.7246
Signal(1,23) 25.6074
Signal(1,25) -20.2101
Signal(1,27) -60.2933
Signal(1,29) -83.8169
Signal(2,0) -31.8826
Signal(2,2) -53.5045
Signal(2,4) -89.9895
Signal(2,7) -84.4503
Signal(2,9) -59.7016
Signal(2,11) -12.8569
Signal(2,13) 28.857
Signal(2,15) 58.0577
Signal(2,18) 96.4222
Signal(2,20) 79.783
Signal(2,22) 58.6463
Signal(2,25) -3.24883
Signal(2,27) -45.5
Signal(2,29) -88.8937
Signal(3,0) -18.6625
Signal(3,2) -53.3978
Signal(1,16) 58.784
Signal(1,17) 44.7782
Signal(1,18) 6.247
Signal(1,19) -12.0855
Signal(1,20) -33.7644
Signal(1,21) -49.4406
Signal(1,22) -67.5791
Signal(1,23) -92.0336
Signal(1,24) -93.9841
END
I wrote code that takes this file and sends it a line at a time over UDP locally, and then code that receives it and parses it based on the data type.
Sender:
import socket
import sys
# Sends udp test data piped in from STDIN to the listener.
# ex: cat sampleoutput.txt | python testsender.py
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
print "UDP target IP:", UDP_IP
print "UDP target port:", UDP_PORT
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
# Send from stdin
if len(sys.argv) < 2:
while True:
line = sys.stdin.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
# get from file arg
else:
myfile = open(str(sys.argv[1]), "r")
while True:
line = myfile.readline()
if line:
sock.sendto(line, (UDP_IP, UDP_PORT))
else:
break
sock.close()
Listener:
import socket
from array import array
UDP_IP = "127.0.0.1"
UDP_PORT = 5000
sock = socket.socket(socket.AF_INET, # Internet
socket.SOCK_DGRAM) # UDP
sock.bind((UDP_IP, UDP_PORT))
while True:
data, addr = sock.recvfrom(1024) # buffer size arg
print data
# write to file for later testing
# file = open("testdummy.txt", "a")
# file.write(data)
if data == "END\n":
break
I was able to use the above listener to produce the test file from the original program, so it should work. Unfortunately, it fails around 500 lines of payload, as tested by tail -n 500 testdummy.txt | python testsender.py, although it's somewhat random. Specifically, the listener does not receive all of the sent lines before the sender exits, leaving it hanging, waiting for the "END\n" string.
As I understand it, the socket is already in blocking mode--how can I prevent this from occurring?
My first advice to you, Don't use UDP if you want to transfer files with the sequence of lines preserved, Use TCP if you don't want to code alot. The reasons are;
UDP is an unreliable Protocol, in the sense a Packet sent is not guaranteed to be received by the recipient.
UDP doesn't guarantee the sequence of packets being received, This is because UDP packets may go to recipient via several routes (Hops between computers). So latter sent packets can take a short route and reach the recipient before former sent packets. ("End\n" packet can come before other packets)
TCP on the other hand is reliable and sequence of packets received is guaranteed. Ideal for file transfer.
But don't worry File sharing applications like Bear Share, Bit Torrents make use of UDP but there are some additional coding you have to do.
You need to implement an Acknowledgement protocol, as in you need to have a unique id for each packet you send to the recipient and when the packet is received recipient should send an Acknowledgement packet back to the sender with that id saying the packet was received.
If in case the packet got lost and didn't reach the recipient (No Acknowledgement from recipient) you must resend the packet again (and again) until you get the Acknowledgement.
You can control the order by not sending the next packet until you get the Acknowledgement for the previous one.