We Keep Coding

Threading and tcp/ip connections problems

I'm new in python and threading so please be indifferent. I'm trying to do 2-players game in python. Data are send through tcp/ip protocol (client-server architecture). On server I have three threads. One comunicate with one user, second with second and in third thread I'm getting data which was send by client form two others threads. This data are used to check if game is over. And it's all working good. Problems start now. When the game is over I want to send another data to client. So Thread 3 need to send data to client, but two others threads are still working and they still have connections with clients. Generally I do not know how to do this. I tried to send information through the Queue from third thread to others that they should close theirs connections. It's thread class code:
class myThread(threading.Thread):
def __init__(self, threadID, name, conn, conn2, kto, wartosc,
wybor,kolejkaZadan,gracz1,gracz2):
threading.Thread.__init__(self)
self.threadID = threadID
self.name = name
self.conn = conn
self.conn2 = conn2
self.kto = kto
self.wartosc = wartosc
self.wybor = wybor
self.kolejkaZadan = kolejkaZadan
self.gracz1 = gracz1
self.gracz2 = gracz2
def run(self):
if self.wybor == None:
toClient(self.conn,self.conn2,self.kto,self.wartosc,self.gracz1)
else:
while True:
data,kolejkaZwrotna = self.kolejkaZadan.get() // I receive data from two others threads
time.sleep(10)
dataKolejne,kolejkaZwrotna = self.kolejkaZadan.get() // I receive data from two others threads
if data is dataKolejne: // if end
tworzenieXmla(self.gracz1, self.gracz2)
odczytywanieXmla('itemGracza1',gracz1Otrzymane)
plik = open('Marcin.xml', 'rb')
czyZamknacConnection = True
kolejkaZwrotna.put(czyZamknacConnection) // send data to two others threads
while True:
czescXmla = plik.read(10000)
#self.conn2.send(czescXmla)
And It's my send/receiv function which is executed by two other threads:
def toClient(conn, conn2, kto, wartosc,gracz):
wordsBackup = None
kolejkaZwrotna = queue.Queue()
while True:
data = conn.recv(BUFFER_SIZE)
if not data:
break
if kolejkaZwrotna.get() is True://receive form thread 3
conn2.close()
print('closed')
break
if len(data)>7:
print('WARNING', data)
words = str(data.decode()).split()
#print(words[0], words[1])
if kto==1:
conn2.send(data)
if kto==2:
conn2.send(data)
kolejkaZadan.put(words[2],kolejkaZwrotna) // send to thread 3
xmlTablicaDoZapisu(str(int(words[0])),str(int(words[1])),str(int(words[2])),gracz)
Generally there is no error and we can play but there is only one player on each computer so I think server don't send data. I would appreciate any help.

A fix for your current situation would be to change all those connection variables into an array of connections which you could iterate over. You might want to build some container classes which define their behavior since not all clients are the same ( server client, and player clients ). That way you aren't limited by the amount of variables you've declared, and threads available.
Then once a new client connects you simply add it to the array and your iterator will take care of the rest.
This is a common problem with TCP/IP though, in that you always have to have open connections to n clients, which not only takes up resources but since TCP/IP is a queued protocol it could also set the entire game back if any client has a slower connection. In practice your game will always be as laggy as the player with the worst connection.
You have a couple of options.
You can have one thread always open which handles connections. Your supervisor thread. It holds an array of open connections' data and dispenses actions to the other threads. This isn't the best option since you'll quickly encounter Race Conditions such as two threads trying to use the same data.
You can switch over to UDP which will leave your threads wide open since there's no persistent connection. You'd then need to send states to each client, and once they ACK them you can get rid of the data. The majority of games implement UDP now'a'days, even turn based ones.
Beej's guide is probably the most extensive on the internet about UDP/TCP and socket control theory.
http://beej.us/guide/bgnet/output/html/singlepage/bgnet.html
And there's also Gaffer on Games which is a fantastic resource as well.
http://gafferongames.com/networking-for-game-programmers/udp-vs-tcp/

how to make python awaitable object

In python 3.5.1 one can make use of await/async, however, to use it (as I undestand), you need to have awaitable object.
An awaitable object is an object that defines __await__() method returning an iterator. More info here.
But I can not google out any example of having this, since most examples have some sort of asyncio.sleep(x) to mimic awaitable object.
My ultimate goal is to make simple websocket serial server, however, I can't pass this first step.
This is my (non working code).
import serial
import asyncio
connected = False
port = 'COM9'
#port = '/dev/ttyAMA0'
baud = 57600
timeout=1
class startser(object):
def __init__(self, port, baud):
self.port = port
self.baud = baud
def openconn(self):
self.ser = serial.Serial(port, baud)
async def readport(self):
#gooo= await (self.ser.in_waiting > 0)
read_byte = async self.ser.read(1).decode('ascii')
self.handle_data(read_byte)
print ("42")
def handle_data(self, data):
print(data)
serr=startser(port,baud)
serr.openconn()
loop = asyncio.get_event_loop()
#loop.run_forever(serr.readport())
loop.run_until_complete(serr.readport())
loop.close()
print ("finitto")
#with serial.Serial('COM9', 115200, timeout=1) as ser:
#x = ser.read() # read one byte
#s = ser.read(10) # read up to ten bytes (timeout)
#line = ser.readline() # read a '\n' terminated line`

I guess there is still no answer because the question is not pretty clear.
You correctly said that
An awaitable object is an object that defines __await__() method returning an iterator
Not much to add here. Just return an iterator from that method.
The only thing you need to understand is how does it work. I mean, how asyncio or another similar framework achieves concurrency in a single thread. This is simple on a high level: just get all your code organized as iterators, then call them one-by-one until the values are exhausted.
So, for example, if you have two iterators, let's say first one yields letters and the second one yields numbers, event loop calls first one and gets 'A', then it calls the second one and gets 1 then it calls first one again and gets 'B' and so on and so on, until the iterators are completed. Of course, each of these iterators can do whatever you want before yielding the next value. But, the longer it takes - the longer pause between 'switching tasks' would be. You MUST keep every iteration short:
If you have inner loops, use async for - this will allow switching task without explicit yielding.
If you have a lot of code which executes for tens or even hundreds of milliseconds, consider rewriting it in smaller pieces. In a case of legacy code, you can use hacks like asyncio.sleep(0) ← this is an allowance for asyncio to switch task here.
No blocking operations! This is most important. Consider you do something like socket.recv(). All tasks will be stopped until this call ends. This is why this is called async io in the standard library: you must use theirs implementation of all I/O functions like BaseEventLoop.sock_recv().
I'd recommend you to start (if you didn't yet) with the following docs:
https://pymotw.com/3/asyncio/
https://docs.python.org/3/library/asyncio.html
https://www.python.org/dev/peps/pep-0492

How to control a simulation in Python

I have a fairly high-level question about Python and running interactive simulations. Here is the setup:
I am porting to Python some simulation software I originally wrote in Smalltalk (VW). It is a kind of Recurrent Neural Network controlled interactively from a graphical interface. The interface allows the manipulation of most the network's parameters in real time, in addition to controlling the simulation itself (starting it, stopping it, etc). In the original Smalltalk implementation, I had two processes running with different priority levels:
The interface itself with a higher priority
The neural network running forever at a lower priority
Communication between the two processes was trivial, because all Smalltalk processes share the same address space (the Object memory).
I am now starting to realize that replicating a similar setup in Python is not so trivial. The threading module does not allow its threads to share address space, as far as I can tell. The multiprocessing module does, but in a rather complex way (with Queues, etc).
So I am starting to think that my Smalltalk perspective is leading me astray and I am approaching a relatively simple problem from the wrong angle altogether. Problem is, I don't know what is the right angle! How would you recommend I approach the problem? I am fairly new to Python (obviously) and more than willing to learn. But I would greatly appreciate suggestions on how to frame the issues and which multiprocessing modules (if any!) I should delve into.
Thanks,
Stefano

I'll offer my take on how to approach this problem. Within the multiprocessing module the Pipe and Queue IPC mechanisms are really the best way to go; in spite of the added complexity you allude to, it's worth learning how they work. The Pipe is fairly straightforward so I'll use that to illustrate.
Here's the code, followed by some explanation:
import sys
import os
import random
import time
import multiprocessing
class computing_task(multiprocessing.Process):
def __init__(self, name, pipe):
# call this before anything else
multiprocessing.Process.__init__(self)
# then any other initialization
self.name = name
self.ipcPipe = pipe
self.number1 = 0.0
self.number2 = 0.0
sys.stdout.write('[%s] created: %f\n' % (self.name, self.number1))
# Do some kind of computation
def someComputation(self):
try:
count = 0
while True:
count += 1
self.number1 = (random.uniform(0.0, 10.0)) * self.number2
sys.stdout.write('[%s]\t%d \t%g \t%g\n' % (self.name, count, self.number1, self.number2))
# Send result via pipe to parent process.
# Can send lists, whatever - anything picklable.
self.ipcPipe.send([self.name, self.number1])
# Get new data from parent process
newData = self.ipcPipe.recv()
self.number2 = newData[0]
time.sleep(0.5)
except KeyboardInterrupt:
return
def run(self):
sys.stdout.write('[%s] started ... process id: %s\n'
% (self.name, os.getpid()))
self.someComputation()
# When done, send final update to parent process and close pipe.
self.ipcPipe.send([self.name, self.number1])
self.ipcPipe.close()
sys.stdout.write('[%s] task completed: %f\n' % (self.name, self.number1))
def main():
# Create pipe
parent_conn, child_conn = multiprocessing.Pipe()
# Instantiate an object which contains the computation
# (give "child process pipe" to the object so it can phone home :) )
computeTask = computing_task('foo', child_conn)
# Start process
computeTask.start()
# Continually send and receive updates to/from the child process
try:
while True:
# receive data from child process
result = parent_conn.recv()
print "recv: ", result
# send new data to child process
parent_conn.send([random.uniform(0.0, 1.0)])
except KeyboardInterrupt:
computeTask.join()
parent_conn.close()
print "joined, exiting"
if (__name__ == "__main__"):
main()
I have encapsulated the computing to be done inside a class derived from Process. This isn't strictly necessary but makes the code easier to understand and extend, in most cases. From the main process you can start your computing task with the start() method on an instance of this class (this will start a separate process to run the contents of your object).
As you can see, we use Pipe in the parent process to create two connectors ("ends" of the pipe) and give one to the child while the the parent holds the other. Each of these connectors is a two-way communication mechanism between the processes holding the ends, with send() and recv() methods for doing what their names imply. In this example I've used the pipe to transmit lists of numbers and text, but in general you can send lists, tuples, objects, or anything that's picklable (i.e. serializable with Python's pickle facility). So you've got some latitude for what you send back and forth between processes.
So you set up your connectors, invoke start() on your new process, and you're off and computing. Here we're just multiplying two numbers, but you can see it's being done "interactively" in the subprocess with updates sent from the parent. Likewise the parent process is informed regularly of new results from the computing process.
Note that the connector's recv() method is blocking, i.e. if the other end hasn't sent anything yet, recv() will wait until something is there to read, and prevent anything else from happening in the meantime. So just be aware of that.
Hope this helps. Again, this is a barebones example and in real life you'll want to do more error handling, possibly use poll() on the connection objects, and so forth, but hopefully this conveys the major ideas and gets you started.

Communication with sockets using threads

I'm currently programming a python class which acts as a client.
Because I don't want to block the main thread, receiving of packets is done in another thread and a callback function is called if a packet arrives.
The received packets are either broadcast messages or a reply for a command sent by the client. The function for sending commands is synchronous, it blocks until the reply arrives so it can directly return the result.
Simplified example:
import socket
import threading
class SocketThread(threading.Thread):
packet_received_callback = None
_reply = None
_reply_event = threading.Event()
def run(self):
self._initialize_socket()
while True:
# This function blocks until a packet arrives
p = self._receive_packet()
if self._is_reply(p):
self._reply = p
self._reply_event.set()
else:
self.packet_received_callback(p)
def send_command(self, command):
# Send command via socket
self.sock.send(command)
# Wait for reply
self._reply_event.wait()
self._reply_event.clear()
return self._process_reply(self._reply)
The problem which I'm facing now is that I can't send commands in the callback function because that would end in a deadlock (send_command waits for a reply but no packets can be received because the thread which receives packets is actually executing the callback function).
My current solution is to start a new thread each time to call the callback function. But that way a lot of threads are spawned and it will be difficult to ensure that packets are processed synchronously in heavy traffic situations.
Does anybody know a more elegant solution or am I going the right way?
Thanks for your help!

A proper answer to this question depends a lot on the details of the problem you are trying to solve, but here is one solution:
Rather than invoking the callback function immediately upon receiving the packet, I think it would make more sense for the socket thread to simply store the packet that it received and continue polling for packets. Then when the main thread has time, it can check for new packets that have arrived and act on them.

Recently had another idea, let me know how you think about it. It's just a general approach to solve such problems in case someone else has a similar problem and needs to use multi-threading.
import threading
import queue
class EventBase(threading.Thread):
''' Class which provides a base for event-based programming. '''
def __init__(self):
self._event_queue = queue.Queue()
def run(self):
''' Starts the event loop. '''
while True:
# Get next event
e = self._event_queue.get()
# If there is a "None" in the queue, someone wants to stop
if not e:
break
# Call event handler
e[0](*e[1], **e[2])
# Mark as done
self._event_queue.task_done()
def stop(self, join=True):
''' Stops processing events. '''
if self.is_alive():
# Put poison-pill to queue
self._event_queue.put(None)
# Wait until finished
if join:
self.join()
def create_event_launcher(self, func):
''' Creates a function which can be used to call the passed func in the event-loop. '''
def event_launcher(*args, **kwargs):
self._event_queue.put((func, args, kwargs))
return event_launcher
Use it like so:
event_loop = eventbase.EventBase()
event_loop.start()
# Or any other callback
sock_thread.packet_received_callback = event_loop.create_event_launcher(my_event_handler)
# ...
# Finally
event_loop.stop()

Producing content indefinitely in a separate thread for all connections?

I have a Twisted project which seeks to essentially rebroadcast collected data over TCP in JSON. I essentially have a USB library which I need to subscribe to and synchronously read in a while loop indefinitely like so:
while True:
for line in usbDevice.streamData():
data = MyBrandSpankingNewUSBDeviceData(line)
# parse the data, convert to JSON
output = convertDataToJSON(data)
# broadcast the data
...
The problem, of course, is the .... Essentially, I need to start this process as soon as the server starts and end it when the server ends (Protocol.doStart and Protocol.doStop) and have it constantly running and broadcasting a output to every connected transport.
How can I do this in Twisted? Obviously, I'd need to have the while loop run in its own thread, but how can I "subscribe" clients to listen to output? It's also important that the USB data collection only be running once, as it could seriously mess things up to have it running more than once.
In a nutshell, here's my architecture:
Server has a USB hub which is streaming data all the time. Server is constantly subscribed to this USB hub and is constantly reading data.
Clients will come and go, connecting and disconnecting at will.
We want to send data to all connected clients whenever it is available. How can I do this in Twisted?

One thing you probably want to do is try to extend the common protocol/transport independence. Even though you need a thread with a long-running loop, you can hide this from the protocol. The benefit is the same as usual: the protocol becomes easier to test, and if you ever manage to have a non-threaded implementation of reading the USB events, you can just change the transport without changing the protocol.
from threading import Thread
class USBThingy(Thread):
def __init__(self, reactor, device, protocol):
self._reactor = reactor
self._device = device
self._protocol = protocol
def run(self):
while True:
for line in self._device.streamData():
self._reactor.callFromThread(self._protocol.usbStreamLineReceived, line)
The use of callFromThread is part of what makes this solution usable. It makes sure the usbStreamLineReceived method gets called in the reactor thread rather than in the thread that's reading from the USB device. So from the perspective of that protocol object, there's nothing special going on with respect to threading: it just has its method called once in a while when there's some data to process.
Your protocol then just needs to implement usbStreamLineReceived somehow, and implement your other application-specific logic, like keeping a list of observers:
class SomeUSBProtocol(object):
def __init__(self):
self.observers = []
def usbStreamLineReceived(self, line):
data = MyBrandSpankingNewUSBDeviceData(line)
# broadcast the data
for obs in self.observers[:]:
obs(output)
And then observers can register themselves with an instance of this class and do whatever they want with the data:
class USBObserverThing(Protocol):
def connectionMade(self):
self.factory.usbProto.observers.append(self.emit)
def connectionLost(self):
self.factory.usbProto.observers.remove(self.emit)
def emit(self, output):
# parse the data, convert to JSON
output = convertDataToJSON(data)
self.transport.write(output)
Hook it all together:
usbDevice = ...
usbProto = SomeUSBProtocol()
thingy = USBThingy(reactor, usbDevice, usbProto)
thingy.start()
factory = ServerFactory()
factory.protocol = USBObserverThing
factory.usbProto = usbProto
reactor.listenTCP(12345, factory)
reactor.run()
You can imagine a better observer register/unregister API (like one using actual methods instead of direct access to that list). You could also imagine giving the USBThingy a method for shutting down so SomeUSBProtocol could control when it stops running (so your process will actually be able to exit).