We Keep Coding

Python socket loop does not break

I created socket for two PC, one is Raspberry Pi and the other one is my laptop. I just connected two then I send string to test the connection. If I send a character "q" from the RPi, my PC should break out of the loop and close the connection but it does not. The part print("Listening") is still running. Why? See code below.
import socket
import time
# IP address of this PC.
TCP_IP = '192.168.137.1'
# Port.
TCP_PORT = 5005
# Size of buffer.
BUFFER_SIZE = 1024
# Create a socket, connect and listen to it.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print('Connection address:', addr)
while 1:
print("Listening")
data = conn.recv(BUFFER_SIZE)
data = data.decode()
if data=='q':
break
if data:
print ("Received data:", data)
# Echo back.
conn.send(data.encode())
time.sleep(1)
print("It breaks.")
conn.close()
s.close()

TCP is a stream oriented protocol. So data transmitted is a stream not a sequence of messages. So when you expect data to be q it actually is some_data_sent_before_q_and_finally_q.
The simplest way to repair the code is to use if data.endswith('q') instead of if data=='q'. May work and may not depending on how you actually use the connection. For example, this approach may fail with some_data_sent_before_q pretty long pause more_data_and_q and with some_data_sent_before_q_and_finally_q_plus_something_else_why_not.
Little bit more advanced way to solve the problem is to divide the stream into messages with separators - message_1<separator>message_2<separator>q<separator>. This method will allow you to treat every message separately.

How to trigger Raspberry Pi 3 to take action from Server

I am currently developing a system where I need to send notification to Raspberry to run a Python file. It is much like a observer pattern design where my server is publisher and Raspberry is the observer. Worth to note that, I actually need to interact with one Raspberry at the time (even I have dozens of them). Specifically, on a specific event, I need to warn a single Raspberry that it has to take an action.
I searched for it literally for all the night but I could not find anything coming handy. Nothing really give me a clue how to implement this.
The most close answer I could find is this technology firm's product called PubNub which can actually work. However, as I need is a one-to-one interaction, this might be unnecessary because it is designed to publish a data to multiple clients.
Long story short, I need to trigger Raspberry to take some action in accordance to the some data coming from the server, whenever it receives the data.
Server is running on Amazon and implemented with Python 2.7.
Please do not hesitate to ask me for further detail, if I am missing any.
Thanks for all the supports,
EDIT
Just a recent update with an improvement to my answer. As far as I understand, sockets are able to manage this process. Such as from client (Raspberry in my case) listening for the server and server sending some data. Taken from this site, I managed to make a sample run on my computer, from my local. I used Port 5000 as their 'meeting point'.
Below is the code:
client.py
#!/usr/bin/env python
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 5000
BUFFER_SIZE = 1024
MESSAGE = b"Hello, World!"
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((TCP_IP, TCP_PORT))
s.send(MESSAGE)
data = s.recv(BUFFER_SIZE)
s.close()
print("received data:", data)
server.py
#!/usr/bin/env python
import socket
TCP_IP = '127.0.0.1'
TCP_PORT = 5000
BUFFER_SIZE = 20 # Normally 1024, but we want fast response
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((TCP_IP, TCP_PORT))
s.listen(1)
conn, addr = s.accept()
print('Connection address:', addr)
while 1:
data = conn.recv(BUFFER_SIZE)
if not data: break
print("received data:", data)
conn.send(data) # echo
conn.close()
However, I still have some questions.
Firstly, I want to learn whether the same thing work when I deploy the project and how. If that will work - lets say I have an url for the server like 'www.thisisanexampleurl.com' - simply assignign a port for it, will work?
Secondly, assuming first question is done, what is the way of making it continous so that it does not stop after receiving and sending data once. Because currently, when it makes the data transfer, it stops working.
Thanks again for the all support and again please do not hesitate to ask me for the further details i am missing any.

You should be able to do something this simple:
Run something like this on your pi:
import socket
s = socket.socket()
host = ""
port = 12345
s.bind((host, port))
s.listen(5)
while True:
try:
clientsock, addr = s.accept()
except OSError:
continue
message = clientsock.recv(20)
#the code you want to run
print("doing %s" % message)
clientsock.close()
And this on your server every time you want the pi to take action:
import socket
s = socket.socket()
host = "0.0.0.0"
port = 12345
s.connect((host, port))
s.send("foo")
s.close()

Have a look at Pyro4. It lets you avoid having to write network code at all and just write code that calls remote Python objects as if they were running on the same machine. In your case, the server could call a normal Python method on your Raspberry Pi to do something. It has many features but you can start with something extremely simple.
raspberry pi code:
import Pyro4
#Pyro4.expose
class Raspberry:
def something(self, arg):
print("called with:", arg)
return "hi from pi"
Pyro4.Daemon.serveSimple({Raspberry: "raspberry"})
server code to make the pi do something:
import Pyro4
rasp = Pyro4.Proxy("PYRONAME:raspberry")
print(rasp.something(42))

Python efficient socket communication

i recently started making a pure skype resolver and after doing everything fine i stuck on the socket communication.
Let me explain
I'm using python to get the user's IP and then the script opens a socket server and it sends the username to an other program written in .NET
Why is that? Well, the python skype API is not that powerfull so i'm using the axSkype library in order to gather more info.
The problem
The python socket sends the username as it should but i dont know the most efficient way to get the info back. I was thinking opening a socket server in the same script and wait for what the .NET program sends back.
I dont really kwon how to make this as fast as possible so i'm asking for your help.
The code
class api:
def GET(self, username):
skypeapi.activateSkype(username)
time.sleep(1) # because skype is ew
buf = []
print("==========================")
print("Resolving user " + username)
#This is where i'm starting the socket and sending data
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect(("127.0.0.1", 5756))
s.sendall(username)
s.close()
#at this poaint i want to get data back from the .NET app
for logfile in glob.glob('*.log'):
buf += logparse.search(logfile, username)
print("Done!")
print("==========================")
return json.dumps(buf)
class index:
def GET(self):
return render.index()
if __name__ == "__main__":
app.run()

You can bind your socket to the connection. This way, your socket stream will remain open and you will be able to send and receive information easily. Integrate this with the _thread module and you will be able to handle multiple streams. Here is some example code that binds a socket to a stream and just sends back whatever the clients sends it(Although in your case you could send whatever data is necessary)
import socket
from _thread import *
#clientHandle function will just receive and send stuff back to a specific client.
def clientHandle(stream):
stream.send(str.encode("Enter some stuff: "))
while True:
#Here is where the program waits for a response. The 4000 is a buffer limit.
data = stream.recv(4000)
if not data:
#If there is not data, exit the loop.
break
stream.senddall(str.encode(data + "\n"))
#Creating socket.
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
host = "" #In this case the host is the localhost but you can put your host
port = 80
try:
#Here the program tries to bind the socket to the stream.
s.bind((host, port))
except socket.error as e:
print("There was an error: " + str(e))
#Main program loop. Uses multithreading to handle multiple clients.
while True:
conn, addr = s.accept()
print("Connected to: " + addr[0] + ": " + str(addr[1]))
start_new_thread(clientHandle,(conn,))
Now in your case, you can integrate this into your api class(Is that where you want to integrate it? Correct me if I'm wrong.). So now when you define and bind your socket, use this code:
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind((host, port))
Where, in your case, host is 127.0.0.1, in other words, your localhost, which can also be accessed by socket.gethostbyname(socket.gethostname())(but that's a bit verbose), and then port, which for you is 5756. Once you have bounded your socket, you have to accept connections through the following syntax:
conn, addr = s.accept()
Which then you can pass conn and addr to whatever function or just use in any other code.
Regardless of what you use it in, to receive data you can use socket.recv() and pass it a buffer limit. (Remember to decode whatever you receive.) And of course, you send data by using socket.sendall().
If you combine this with the _thread module, as shown above, you can handle multiple api requests, which could come handy in the future.
Hope this helps.

I need the server to send messages to all clients (Python, sockets)

This is my server program, how can it send the data received from each client to every other client?
import socket
import os
from threading import Thread
import thread
def listener(client, address):
print "Accepted connection from: ", address
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
client.send(data)
client.close()
host = socket.gethostname()
port = 10016
s = socket.socket()
s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
s.bind((host,port))
s.listen(3)
th = []
while True:
print "Server is listening for connections..."
client, address = s.accept()
th.append(Thread(target=listener, args = (client,address)).start())
s.close()

If you need to send a message to all clients, you need to keep a collection of all clients in some way. For example:
clients = set()
clients_lock = threading.Lock()
def listener(client, address):
print "Accepted connection from: ", address
with clients_lock:
clients.add(client)
try:
while True:
data = client.recv(1024)
if not data:
break
else:
print repr(data)
with clients_lock:
for c in clients:
c.sendall(data)
finally:
with clients_lock:
clients.remove(client)
client.close()
It would probably be clearer to factor parts of this out into separate functions, like a broadcast function that did all the sends.
Anyway, this is the simplest way to do it, but it has problems:
If one client has a slow connection, everyone else could bog down writing to it. And while they're blocking on their turn to write, they're not reading anything, so you could overflow the buffers and start disconnecting everyone.
If one client has an error, the client whose thread is writing to that client could get the exception, meaning you'll end up disconnecting the wrong user.
So, a better solution is to give each client a queue, and a writer thread servicing that queue, alongside the reader thread. (You can then extend this in all kinds of ways—put limits on the queue so that people stop trying to talk to someone who's too far behind, etc.)
As Anzel points out, there's a different way to design servers besides using a thread (or two) per client: using a reactor that multiplexes all of the clients' events.
Python 3.x has some great libraries for this built in, but 2.7 only has the clunky and out-of-date asyncore/asynchat and the low-level select.
As Anzel says, Python SocketServer: sending to multiple clients has an answer using asyncore, which is worth reading. But I wouldn't actually use that. If you want to write a reactor-based server in Python 2.x, I'd either use a better third-party framework like Twisted, or find or write a very simple one that sits directly on select.

Is multicast appropriate for one to many streaming within a localhost?

I have a central data feed that I want to redistribute to many clients. The data feed produces approx. 1.8 kB/s. Currently I'm writing the feed to a file and each client reads off the end of the file. Something about this just seems wrong. Here is pseudo code for what I have now...
The feed:
o = open('feed.txt','a',0) #no buffering, maybe line buffer would be better
while 1:
data = feed.read(8192)
data = parse_data(data)
o.write(data)
time.sleep(0.01)
The server (each client connects in a new thread):
feed = open('feed.txt','r')
feed.seek(-1024,2)
while 1:
dat = feed.read(1024)
if len(dat)==0:
# For some reason if the end of the file is reached
# i can't read any more data, even there is more.
# some how backing off seems to fix the problem.
self.feed.seek(-1024,2)
self.feed.read(1024)
buffer += dat
idx = buffer.rfind('\n')
if idx>0:
data = buffer[:idx]
buffer = buffer[idx+1:]
for msg in data.split('\n'):
client.send(msg)
time.sleep(0.01)
What I'd like to do is just replace the file with a socket and write the messages directly to multicast packets. Any time a new client connects to the server I just spin up a new thread and start listening for the multicast packets. Are there any standard design patterns to handle this case?

Even simpler, just have all clients multicast on the same port. Then your server doesn't even need to track pseudo-connections.
We use a similar scheme for some of the software on our internal network, based on the fact that multicasting is "mostly reliable" on our networking infrastructure. We've stress tested the load and don't start dropping packets until there's over 30K messages/sec.
#!/usr/bin/python
import sys
import socket
ADDR = "239.239.239.9"
PORT = 7999
sock = socket.socket(socket.AF_INET, socket.SOCK_DGRAM)
sock.bind((ADDR,PORT))
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
while True:
data, addr = sock.recvfrom(2048)
print data
sys.stdout.flush()