first of all I have to mention I'm not a programmer but a mechanical engineer so please don't crucify me if I misinterpret something or say some nonsense.
I want to write a python code witch will be "simulating" a real life problem. The real life problem is something like FIFO queue, where objects are taken from on different stations and they spend there some time and then they are returned back to queue.
What I understand what I need is to write an asynchronous Programm, because I have one Function which is putting objects to queue (let say every 15 second) and then I have some stations which take only one object from this queue and then are working on it for some time again (simple timer and a print "Hi im working on object x, will return it in: minutes".
I'm not sure if I can do it with Threading? What if I had 100 stations which work asynchronous is it possible to start 100 Threads? Because as I understand every thread should have one timer?
I would ask to give me a little push to the simplest direction to solve it, it doesn't have to be pretty but functional and easy for me.
thank you in advance for each idea!
Best regards,
MM.
Of course you can use Threading to run several processes simultaneously.
You have to create a class like this :
from threading import Thread
class Work(Thread):
def __init__(self):
Thread.__init__(self)
self.lock = threading.Lock()
def run(self): # This function launch the thread
(your code)
if you want run several thread at the same time :
def foo():
i = 0
list = []
while i < 10:
list.append(Work())
list[i].start() # Start call run() method of the class above.
i += 1
Be careful if you want to use the same variable in several threads. You must lock this variable so that they do not all reach this variable at the same time. Like this :
lock = threading.Lock()
lock.acquire()
try:
yourVariable += 1 # When you call lock.acquire() without arguments, block all variables until the lock is unlocked (lock.release()).
finally:
lock.release()
From the main thread, you can call join() on the queue to wait until all pending tasks have been completed.
This approach has the benefit that you are not creating and destroying threads, which is expensive. The worker threads will run continuously, but will be asleep when no tasks are in the queue, using zero CPU time.
I hope it will help you.
Related
Maybe it's a very simple question, but I'm new in concurrency. I want to do a python script to run foo.py 10 times simultaneously with a time limit of 60 sec before automatically abort. The script is a non deterministic algorithm, hence all executions takes different times and one will be finished before the others. Once the first ends, I would like to save the execution time, the output of the algorithm and after that kill the rest of the processes.
I have seen this question run multiple instances of python script simultaneously and it looks very similar, but how can I add time limit and the possibility of when the first one finishes the execution, kills the rest of processes?
Thank you in advance.
I'd suggest using the threading lib, because with it you can set threads to daemon threads so that if the main thread exits for whatever reason the other threads are killed. Here's a small example:
#Import the libs...
import threading, time
#Global variables... (List of results.)
results=[]
#The subprocess you want to run several times simultaneously...
def run():
#We declare results as a global variable.
global results
#Do stuff...
results.append("Hello World! These are my results!")
n=int(input("Welcome user, how much times should I execute run()? "))
#We run the thread n times.
for _ in range(n):
#Define the thread.
t=threading.Thread(target=run)
#Set the thread to daemon, this means that if the main process exits the threads will be killed.
t.setDaemon(True)
#Start the thread.
t.start()
#Once the threads have started we can execute tha main code.
#We set a timer...
startTime=time.time()
while True:
#If the timer reaches 60 s we exit from the program.
if time.time()-startTime>=60:
print("[ERROR] The script took too long to run!")
exit()
#Do stuff on your main thread, if the stuff is complete you can break from the while loop as well.
results.append("Main result.")
break
#When we break from the while loop we print the output.
print("Here are the results: ")
for i in results:
print(f"-{i}")
This example should solve your problem, but if you wanted to use blocking commands on the main thread the timer would fail, so you'd need to tweak this code a bit. If you wanted to do that move the code from the main thread's loop to a new function (for example def main(): and execute the rest of the threads from a primary thread on main. This example may help you:
def run():
pass
#Secondary "main" thread.
def main():
#Start the rest of the threads ( in this case I just start 1).
localT=threading.Thread(target=run)
localT.setDaemon(True)
localT.start()
#Do stuff.
pass
#Actual main thread...
t=threading.Thread(target=main)
t.setDaemon(True)
t.start()
#Set up a timer and fetch the results you need with a global list or any other method...
pass
Now, you should avoid global variables at all costs as sometimes they may be a bit buggy, but for some reason the threading lib doesn't allow you to return values from threads, at least i don't know any methods. I think there are other multi-processing libs out there that do let you return values, but I don't know anything about them so I can't explain you anything. Anyways, I hope that this works for you.
-Update: Ok, I was busy writing the code and I didn't read the comments in the post, sorry. You can still use this method but instead of writing code inside the threads, execute another script. You could either import it as a module or actually run it as a script, here's a question that may help you with that:
How to run one python file in another file?
I want to read current results from a thread (which has a high sample rate). The thread reads sensor values from the hardware with a looptime of 10 ms -> 1 ms work and 9 ms sleep but it varies a bit (+/- 0.5 ms). In my first approach I used a Lifo-Queue. Thread writes to the queue and main program reads it and empty the queue after reading. That worked well then. Until I noticed that it came after a long time of the program to call delays.
As far as I understood the problem. Every queue.put () uses fresh memory and queue.get () + queue.clear () makages memory to be purged. The memory is then cleaned up at certain intervals. and the cleaner comes with different runtime. And that generates the sporadic call delays.
Now I have asked myself if the queue makes sense, in this application case. So I created a threading.Object and built in addition to init () and run () a get_value_function("get_io_data"), which is called by the main process. The main process has no possibility to write back to the sensor data. The writing back is not needed. Only reading in one direction from the thread into the main process. The GIL should actually prevent simultaneous reading and writing. And the worst that can happen is that an old value has been read.
Is the assumption correct? Have I overlooked something or not understood?
This code is a schematic representation to make the question easier to understand. He is not functional!
import threading
import time
from My_modul import controller_step
class IOboard(threading.Thread):
def __init__(self):
threading.Thread.__init__(self, name="IO_board")
self.__data__ = {}
def run(self):
self.init_hardware()
while True:
self.data.update(self.read_all_sensors())
self.data.update(self.read_all_aktors())
time.sleep(0.09)
def get_io_data(self):
return self.__data__
def main():
app = IOboard()
app.start()
while True:
input_data_now = app.get_io_data()
out_put_data = controller_step(input_data_now)
print(time.time() + out_put_data)
time.sleep(0.1)
if __name__ == "__main__":
main()
This is my first stack overflow question. I hope that I could understand myself. I want an answer: if with build-in-call a thread can query a varibale without violating the python-law or linux-law
I want to run two endless parallel loops. One is reading data from a server and updates an object with a number. The other is doing nothing else then reading it and in case of change, processing it. Does not have to be in sync or so. So my questions are :
In case of write from one side and read from another, does Python have issues with it ?
In case I get a sync problem, do I need to lock the read/write processes ? Any other way
I should do it ?
What is best to use, thread or threading ?
As the next step, I will read from 100 sites and update 100 objects,
and read from 100 loops for the changes. Is it recommend to use Multiprocessing from the
beginning so I can scale without problems ? Do I need at the read and write issues ?
Any help is appreciated.
Short answer is, whatever you think will be understandable for you.
Meaning, your code should make sense to you for learning purposes..
Here's an example, it's light and easy to use.
Getting values from and to the thread is easy..
It's not actual multi-threading tho (same CPU core)
from threading import *
class worker(Thread):
def __init__(self, input=0):
self.input = input
Thread.__init__(self)
self.start()
def run(self):
while 1:
self.input += 1
x = worker(-100)
y = worker(x.input)
print y.input
This is just an example to show that the Y thread can access the data in x.. in practice this can be dangerous considering that both threads will be updating the same variable :) (In short: -100 will be calculated twice per cycle, -98, -96, -94.. etc)
Will not span across multiple CPU's
Easy to use ( accessing data across threads is easy )
Logical code, if you're not familar with queue systems or distributed systems
Will raise a error if the OS can't create more threads (a "limitation")
from threading import Thread
from Queue import Queue
class producer(Thread):
def __init__(self,queue):
Thread.__init__(self)
self.queue=queue
self.start()
def run(self):
while 1:
self.queue.put(update_value())
class consumer(Thread):
def __init__(self,queue):
Thread.__init__(self)
self.queue=queue
self.start()
def run(self):
while True:
value = queue.get()
do_whatever_you_want(value)
queue = Queue()
producer(queue)
consumer(queue)
notice that you can scale by using 100 producer and one consumer (and of course one queue) 100 threads should be ok but things would be different if you wanted 10000
I have some python application with 2 threads. Each thread operates within a separate gui. The GUIs need to operate independently without blocking. I am trying to figure out how to make thread_1 trigger an event to happen in thread_2?
Below is some code I want function foo to trigger function bar in the simplest, most elegant way as quickly as possible, without consuming unnecessary resources. Below is what I've come up with.
bar_trigger=False #global trigger for function bar.
lock = threading.Lock()
class Thread_2(threading.Thread):
def run(self):
global lock, bar_trigger
while(True):
lock.acquire()
if bar_trigger==True:
Thread_2.bar() #function I want to happen
bar_trigger=False
lock.release()
time.sleep(100) #sleep to preserve resources
#would like to preserve as much resources as possible
# and sleep as little as possible.
def bar(self):
print "Bar!"
class Thread_1(threading.Thread):
def foo(self):
global lock, bar_trigger
lock.acquire()
bar_trigger=True #trigger for bar in thread2
lock.release()
Is there a better way to accomplish this? I'm not a threadding expert so any advice on how to best trigger a method in thread_2 from within thread_1 is appreciated.
Without knowing what you're doing and what GUI framework you're using, I can't get into much more detail, but from your problem's code snippet, it sounds like you're looking for something called conditional variables.
Python comes with them included by default in the threading module, under threading.Condition You might be interested in threading.Event as well.
How are these threads instantiated? There should really be a main thread that oversees the workers. For example,
import time
import threading
class Worker(threading.Thread):
def __init__(self, stopper):
threading.Thread.__init__(self)
self.stopper = stopper
def run(self):
while not self.stopper.is_set():
print 'Hello from Worker!'
time.sleep(1)
stop = threading.Event()
worker = Worker(stop)
worker.start()
# ...
stop.set()
Using a shared Event object is just one way of synchronizing and sending messages between threads. There are others, and their usages depend on the specifics.
One option would be to share a queue between the threads. Thread 1 would push an instruction into the queue and thread two would poll that queue. When Thread 2 sees the queue is non-empty, it reads off the first instruction in the queue and calls the appropriate function. This has the additional benefit of being fairly loosely couple which can make testing each thread in isolation easier.
I am using this code:
def startThreads(arrayofkeywords):
global i
i = 0
while len(arrayofkeywords):
try:
if i<maxThreads:
keyword = arrayofkeywords.pop(0)
i = i+1
thread = doStuffWith(keyword)
thread.start()
except KeyboardInterrupt:
sys.exit()
thread.join()
for threading in python, I have almost everything done, but I dont know how to manage the results of each thread, on each thread I have an array of strings as result, how can I join all those arrays into one safely? Because, I if I try writing into a global array, two threads could be writing at the same time.
First, you actually need to save all those thread objects to call join() on them. As written, you're saving only the last one of them, and then only if there isn't an exception.
An easy way to do multithreaded programming is to give each thread all the data it needs to run, and then have it not write to anything outside that working set. If all threads follow that guideline, their writes will not interfere with each other. Then, once a thread has finished, have the main thread only aggregate the results into a global array. This is know as "fork/join parallelism."
If you subclass the Thread object, you can give it space to store that return value without interfering with other threads. Then you can do something like this:
class MyThread(threading.Thread):
def __init__(self, ...):
self.result = []
...
def main():
# doStuffWith() returns a MyThread instance
threads = [ doStuffWith(k).start() for k in arrayofkeywords[:maxThreads] ]
for t in threads:
t.join()
ret = t.result
# process return value here
Edit:
After looking around a bit, it seems like the above method isn't the preferred way to do threads in Python. The above is more of a Java-esque pattern for threads. Instead you could do something like:
def handler(outList)
...
# Modify existing object (important!)
outList.append(1)
...
def doStuffWith(keyword):
...
result = []
thread = Thread(target=handler, args=(result,))
return (thread, result)
def main():
threads = [ doStuffWith(k) for k in arrayofkeywords[:maxThreads] ]
for t in threads:
t[0].start()
for t in threads:
t[0].join()
ret = t[1]
# process return value here
Use a Queue.Queue instance, which is intrinsically thread-safe. Each thread can .put its results to that global instance when it's done, and the main thread (when it knows all working threads are done, by .joining them for example as in #unholysampler's answer) can loop .getting each result from it, and use each result to .extend the "overall result" list, until the queue is emptied.
Edit: there are other big problems with your code -- if the maximum number of threads is less than the number of keywords, it will never terminate (you're trying to start a thread per keyword -- never less -- but if you've already started the max numbers you loop forever to no further purpose).
Consider instead using a threading pool, kind of like the one in this recipe, except that in lieu of queueing callables you'll queue the keywords -- since the callable you want to run in the thread is the same in each thread, just varying the argument. Of course that callable will be changed to peel something from the incoming-tasks queue (with .get) and .put the list of results to the outgoing-results queue when done.
To terminate the N threads you could, after all keywords, .put N "sentinels" (e.g. None, assuming no keyword can be None): a thread's callable will exit if the "keyword" it just pulled is None.
More often than not, Queue.Queue offers the best way to organize threading (and multiprocessing!) architectures in Python, be they generic like in the recipe I pointed you to, or more specialized like I'm suggesting for your use case in the last two paragraphs.
You need to keep pointers to each thread you make. As is, your code only ensures the last created thread finishes. This does not imply that all the ones you started before it have also finished.
def startThreads(arrayofkeywords):
global i
i = 0
threads = []
while len(arrayofkeywords):
try:
if i<maxThreads:
keyword = arrayofkeywords.pop(0)
i = i+1
thread = doStuffWith(keyword)
thread.start()
threads.append(thread)
except KeyboardInterrupt:
sys.exit()
for t in threads:
t.join()
//process results stored in each thread
This also solves the problem of write access because each thread will store it's data locally. Then after all of them are done, you can do the work to combine each threads local data.
I know that this question is a little bit old, but the best way to do this is not to harm yourself too much in the way proposed by other colleagues :)
Please read the reference on Pool. This way you will fork-join your work:
def doStuffWith(keyword):
return keyword + ' processed in thread'
def startThreads(arrayofkeywords):
pool = Pool(processes=maxThreads)
result = pool.map(doStuffWith, arrayofkeywords)
print result
Writing into a global array is fine if you use a semaphore to protect the critical section. You 'acquire' the lock when you want to append to the global array, then 'release' when you are done. This way, only one thread is every appending to the array.
Check out http://docs.python.org/library/threading.html and search for semaphore for more info.
sem = threading.Semaphore()
...
sem.acquire()
# do dangerous stuff
sem.release()
try some semaphore's methods, like acquire and release..
http://docs.python.org/library/threading.html