I was looking how to do a multithread (2 threads) in python.
I want one of them is reading the serial port constantly. reading every frame and saving them to a database. I already have done a script to do this.
For the second one, I want it to listen a socket port. When it receives something from that port, I want it to pause the first thread, write something to the serial port and write to the socket. After that, unpause the first thread and go back to listen socket port.
I think the best idea is pausing one thread from the other to read serial port in that moment because if I read the answer by serial port in the 1th thread, I have to pass value read to the second one and it is more complicated, isn't it?
I already have the part of writing on serial port and check some tutorials for socket part so I have no problems with that. But I haven't find anything about pause one thread from another and I am thinking it is not possible.
What should I do in this case?
EDIT: Ask about shared variables: SO I can declare a global variable and make something like:
global1
global2
Thread 1:
while(global1 == 0)
do whatever
global2 = 1
thread 2:
wait socket
if dataReceived: global1 = 1
if global2 = 1 do whatever on serial port
global2 = 0
when finish global1 = 0
with 2 globals I can notify to thread1 to stop to go ahead next iteration and with global2, the second thread knows when the serial port is not being used...
How do I declare a shared variable in python? or it is just another variable....
I'm not sure you can share objects directly between processes, but since every process can share objects with the main process, you can use the main process to pass them back and forth:
http://docs.python.org/2/library/multiprocessing.html#exchanging-objects-between-processes
Related
I have a large python script with a thread that listens to a serial port and puts new data to a queue whenever it's received. I've been trying to improve the performance of the script, as right now even when nothing is happening it's using ~ 12% of my Ryzen 3600 CPU. That seems very excessive.
Here's the serial listener thread:
def listen(self):
"""
Main listener
"""
while self.doListen:
# wait for a message
if self.bus.ser.in_waiting:
# Read rest of message
msg = self.bus.read(self.bus.ser.in_waiting)
# Put into queue
self.msgQueue.put_nowait(msg)
I profiled the script using yappi and found that the serial.in_waiting call seems to be hogging the majority of the cycles. See the KCachegrind screenshot below:
I tried the trick suggested in this question, doing a blocking read(1) call to wait for data. However read(1) just continuously returns empty data and never actually blocks (and yes, I've made sure my pyserial timeout is set to None)
Is there a more elegant and CPU-friendly way to wait for data on the serial bus? My messages are of variable length, so doing a blocking read(n) wouldn't work. They also don't end in newlines or any specific terminators, so readline() wouldn't work either.
Aaron's suggestion was great. A simple time.sleep(0.01) in my serial thread dramatically cut down on CPU usage. So far it looks like I'm not missing any messages either, which was my big fear with adding in sleeps.
The new listener thread:
def listen(self):
"""
Main listener
"""
while self.doListen:
# wait for a message
if self.bus.ser.in_waiting:
# Read rest of message
msg = self.bus.read(self.bus.ser.in_waiting)
# Put into queue
self.msgQueue.put_nowait(msg)
# give the CPU a break
time.sleep(0.01)
Every UDP server example I can find uses a while True loop to listen for incoming data. I'm attempting to use a single UDP socket server as part a kivy window that's also doing other things. As soon as I implement the server's while True loop everything locks up, as I guess I would expect it to do.
How do I listen on a UDP port and also have the rest of the program continue functioning?
I've tried moving the UDP server handling to another (udp_server.py) file and then importing the function, but since I'm importing the while loop nothing changes.
I've also tried assigning the received data to a variable inside udp_server.py and then just importing that variable, with udp_server.py already running separately, but even that is locking up my main program.
I'm 99.99% sure it's just some basic thing that I should already know, but I'm new to Python. Thanks in advance for any help.
Thank you Chris!!!!!!
I'm sure I'm understating the complexity of threading, but it works great now and the only thing I had to add was:
def thread_function():
from udp_server import amx_rx
# do stuff with amx_rx...
# class TouchPanel stuff...
if __name__ == '__main__':
x = threading.Thread(target=thread_function, daemon=True)
x.start()
try:
TouchPanel().run()
except KeyboardInterrupt:
raise
Now I have a running program with a UDP socket listening in the background! Thank you!!!
I have a Python script that is used to receive data associated with a radio station audio event (such as a song or commercial) from the machine playing the audio. The script will parse and process the data and then send portions of it to various other destinations.
First the socket is set up:
client_socket_1 = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
try:
print 'trying to open socket 1'
client_socket_1.connect((TCP_RCV_IP_CR1, TCP_RCV_PORT_CR1))
client_socket_1.setblocking(0)
except socket.error, e:
print 'Error', e, TCP_RCV_IP_CR1, '\n\n\n'
else:
SOCK1 = 1
print 'Successful connection to ',TCP_RCV_IP_CR1,'\n'
Now we wait until data is available to be read. I used select() and when the socket is ready to be read, the thread that parses and processes the data is spawned.
ready_1 = select.select([client_socket_1], [], [], 1) # select tells us when data is available at the socket
if ready_1[0] and SOCK1: # Don't run this code if there is no connection on client_socket_1 or no data available
t1 = Thread(target=processdata1) # Set up the thread
t1.start() # Call the process to process available data as a thread
It is important that the data be read as quickly as possible as it will be transported via TCP or UDP (depending on the particular data chunks and program specifications) along with the associated audio, and the function of one of the data items we are handling can create an on-air 'hiccup' in the audio if not received in a timely fashion. (TMI: It causes a 'replacement' commercial to play at the receiving end which is supposed to 'cover' the commercial audio we are sending. If the replacement spot doesn't start quickly enough listeners will hear the beginning of the commercial we are sending, then the local replacement one will start when our data is received and it sounds like a hiccup on the air.)
To confirm that my script is not always receiving the data quickly enough I telnetted to the port it is listening to and watched the data as it is received in the telnet window, then look at the Python output (which sends received data to stdout as soon as it is received) and I see about a 1.5-second delay between the telnet output and the Python output. This is the same amount of delay we have observed in normal on-air operation.
I chose to use select() because I was asked to multi-thread the script and I thought that would be a good way to know when to trigger a data-processing thread. My original idea was to simply loop through attempting to read data from each of the three systems we are monitoring and, when data is found, process it.
The thought was that if data is being processed from one system when another system has data ready to be read, it might cause a delay in processing and sending out the data from that machine. However, I can't see that delay being as significant as what I am experiencing now. I am considering going back to the original plan.
I would rather stick with what I have which is working flawlessly as long as data is received in a timely fashion. Any thoughts on why the excessively long delay?
I think it has to do with your timeout parameter in combination with the wlist and xlist parameters
Consider this piece of code
write_list = []
exception_list = []
select.select([client_socket_1], write_list, exception_list)
It takes an optional timeout parameter, like you use it. The documentation says
select() also takes an optional fourth parameter which is the number
of seconds to wait before breaking off monitoring if no channels have
become active. Using a timeout value lets a main program call select()
as part of a larger processing loop, taking other actions in between
checking for network input.
It might be that the call will always wait one second before returning because of the empty lists. Try
ready_1 = select.select(
[client_socket_1],
[client_socket_1],
[client_socket_1], 1
)
Or you can use a timeout value of 0, which
specifies a poll and never blocks.
I'm using a library called BACpypes to communicate over network with a PLC. The short version here is that I need to start a BACpypes application in its own thread and then perform read/write to the plc in this separate thread.
For multiple PLC's, there is a processing loop that creates an application (providing the plc ip address), performs read writes on plc using application, kills application by calling BACpypes stop(*args) from the Core module, calls join on the thread, and then moves on to next ip address in the list until we start over again. This works for as many ip addresses (PLCs) as we have, but as soon as we are back at the first ip address (PLC) again, I get the error:
socket.error: [Errno 98] Address already in use
Here is the short code for my thread class, which uses the stop() and run() functions from BACpypes core.
class BACpypeThread(Thread):
def __init__(self, name):
Thread.__init__(self)
Thread.name = name
def run(self):
run()
def stop(self):
stop()
It seems like I'm not correctly killing the application. So, I know stop(*args) is registered as a signal handler according to BACpypes docs. Here is a snippet I pulled from this link http://bacpypes.sourceforge.net/modules/core.html
core.stop(*args)
Parameters: args – optional signal handler arguments
This function is called to stop a BACpypes application. It resets the running boolean value. This function also installed as a signal handler responding to the TERM signal so you can stop a background (deamon) process:
$ kill -TERM 12345
I feel like I need to provide a kill -term signal to make the ip address available again. I don't know how to do that. Here's my question...
1) In this example, 12345 is the process number I believe. How do I figure out that number for my thread?
2) Once I have the number, how do I actually pass the kill -TERM signal to the stop function? I just don't know how to actually write this line of code. So if someone could explain this that would be great.
Thanks for the help!
Before stopping the core, you need to free the socket.
I use :
try:
self.this_application.mux.directPort.handle_close()
except:
self.this_application.mux.broadcastPort.handle_close()
After that I call stop
then thread.join()
So here's the problem, I have a small server script in Python that is supposed to accept multiple clients and based on the message they are sending, receiving a certain command back to them. It's a simple concept and it's working like I want to, with one really big problem: I put each connection on hold and in separate thread, and I want when a certain connected users puts EXIT to close the connection...Which works, with one really big problem - the thread is kept alive and there is no way to kill it and that really bothers me.
sock = socket()
sock.bind((host,port))
sock.listen(50)
def clientthread(conn):
while True:
data = conn.recv(1024).strip()
if(data == "HELO"):
conn.send("HELO")
elif(data == "EXIT"):
conn.close()
break
return
while True:
conn,addr = sock.accept()
start_new_thread(clientthread, (conn,))
conn.close()
sock.close()
I searched of a way to terminate a thread but just couldn't find it, .join() is not working here since it detects the thread as "dummy", it does not recognize the __stop() and since a couple of searches on google for this topic I'm really out of options. Any idea? I'll be really grateful, thanks.
AFAIK, you can't kill a thread from another - you have to arrange for the thread-to-be-killed to notice some flag has changed, and terminate itself.
BTW, your socket code looks a little off - you need a loop around your send's and recv's unless you use something like twisted or bufsock. IMO, bufsock is much easier and less error prone than twisted, but I may be biased because I wrote bufsock. http://stromberg.dnsalias.org/~strombrg/bufsock.html
The problem with what I'm seeing is that TCP reserves the right to split or aggregate transmission units. Usually it won't, but under high load, or with a changing Path MTU, or even just Nagle, it probably will.
Assuming you're using Python v2.4+, you should be using the newer Threading module. Check out a tutorial on it here - It explains the use of the threading module you're using now and how and why you should use the newer Threading module.