In python, I have two methods. In method A, I receive parameters and put them into parameter array. In procedure B, I process the data of parameter array, and put the results of processing into a log array. I want to get the data of reading log data by while loop in final of method A, and get the processing of parameters currently passed into A. As a result, I would like to ask how to pause to start method B when A is half-executed, otherwise A will endless loop.
Adding sleep method expects A to interrupt and B to execute, but it has no effect.
def A()
try:
datas=request.get_data()
data=json.loads(datas)
global queque_list,log_list
queque_list.append("data":data)
finally:
while 1:
sleep(3)
if len(log_list)>0
for logdata in log_list:
if logdata.get('uuid')==uuid:
return logdata.get('msg')
def B(task):
try:
do(task)
finally:
log_list.append({"uuid":uuid,"msg":msg})
def C():
while True:
if len(queque_list)>0：
task=queque_list.pop(0)
B(task)
t=threading.Thread(target=C)
t.start()
I expect if method A can interrupt when executing final module and wait for method B to finish executing before executing. but now method A executing final module and method B non-execution ,the mothod endless loop
You can use queue.Queue to send messages between the threads, specifically the put() method to send a message and the get() method to wait for a message in another thread. With this, you can get the threads to work in lock-step.
I'm not sure what you are trying to do, but perhaps you can get away with doing all the work in a single thread for simplicity.
Related
How can I run 2 functions in parallel and wait for the return of one of them.
Basically what I have is a keypad, which I call a function where you enter a pin and a function where i read an NFC Tag. So which one returns first I will use the return value on the rest of the code.
I've looked into threading and what I get from it, is that it hangs until all the threads are finished.
Any insight?
You could let the threads call a method when they are done that only handles one call. Hopefully the following shows what I mean:
# consider this as pseudo code, I am commuting right know
def task1(callback):
# do something
callback(*args)
class SomeCallback:
available = True
def __call__(*args):
if not SomeCallback.available: return # a second call will do nothing
# work with nfc tag response or keyphrase
SomeCallback.available = False
# maybe even end all other threads here
thread = Thread(target=task1, (callback,))
thread.start()
# start a second thread or do the other thing on the mainthread
I currently have this function, which does a api call, each api call is requesting different data. I can do up to 300 concurrent api calls at a time.
Doing this does not seem to go fast, since this is just waiting for the repl I was wondering how I would make this function faster?
from multiprocessing.pool import ThreadPool
import requests
pool = ThreadPool(processes=500)
variables = VariableBaseDict
for item in variables:
async_result = pool.apply_async(requests.get(url.json()))
result = async_result.get()
#do stuff with result
Your current code is not actually farming any real work off to a worker thread. You are calling requests.get(url.json()) right in the main thread, and then passing the object that returns to pool.apply_async. You should be doing pool.apply_async(requests.get, (url.json(),)) instead. That said, even if you corrected this problem, you are then immediately waiting for the reply to the call, which means you never actually run any calls concurrently. You farm one item off to a thread, wait for it to be done, then wait for the next item.
You need to:
Fix the issue where you're accidentally calling requests.get(...) in the main thread.
Either use pool.map to farm the list of work off to the worker threads concurrently, or continue using pool.apply_async, but instead of immediately calling async_result.get(), store all the async_result objects in a list, and once you've iterated over variables, iterate over the async_result list and call .get() on each item. That way you actually end up running all the calls concurrently.
So, if you used apply_async, you'd do something like this:
async_results = [pool.apply_async(requests.get, (build_url(item),)) for item in variables]
for ar in async_results:
result = ar.get()
# do stuff with result
With pool.map it would be:
results = pool.map(requests.get, [build_url(item) for item in variables])
I have a script with 2 threads, a basic example is shown below:
Thread 1:
value = stuff()
if value > 0:
# Code to make something() run
Thread 2:
def something():
# Thread specific task goes here
I need something() to be run in Thread 2, not Thread 1. I know how to 'sync' variables with Queue and I know I could just put a loop in Thread 2 that waits for a condition then executes something(), but that would break other stuff in Thread 2.
Sorry if I haven't made this clear, it's kind of hard to explain. Thanks.
Thread 2 needs to have a queue for delayed applications then thread 1 can simply add its call to this queue.
The contents of the queue would be either the name of or a reference to the function and the args and kwargs for use in calling the function.
In the end you would end up using the active object pattern and the promise pattern if you asynchronously wait where the promise has an event. If you wait synchronusly you would most likely lock the called thread and then pass the message and then block on the event and then the called thread would set the event and you would unblock and read the result.
I know there are a few questions and answers related to hanging threads in Python, but my situation is slightly different as the script is hanging AFTER all the threads have been completed. The threading script is below, but obviously the first 2 functions are simplified massively.
When I run the script shown, it works. When I use my real functions, the script hangs AFTER THE LAST LINE. So, all the scenarios are processed (and a message printed to confirm), logStudyData() then collates all the results and writes to a csv. "Script Complete" is printed. And THEN it hangs.
The script with threading functionality removed runs fine.
I have tried enclosing the main script in try...except but no exception gets logged. If I use a debugger with a breakpoint on the final print and then step it forward, it hangs.
I know there is not much to go on here, but short of including the whole 1500-line script, I don't know hat else to do. Any suggestions welcome!
def runScenario(scenario):
# Do a bunch of stuff
with lock:
# access global variables
pass
pass
def logStudyData():
# Combine results from all scenarios into a df and write to csv
pass
def worker():
global q
while True:
next_scenario = q.get()
if next_scenario is None:
break
runScenario(next_scenario)
print(next_scenario , " is complete")
q.task_done()
import threading
from queue import Queue
global q, lock
q = Queue()
threads = []
scenario_list = ['s1','s2','s3','s4','s5','s6','s7','s8','s9','s10','s11','s12']
num_worker_threads = 6
lock = threading.Lock()
for i in range(num_worker_threads):
print("Thread number ",i)
this_thread = threading.Thread(target=worker)
this_thread.start()
threads.append(this_thread)
for scenario_name in scenario_list:
q.put(scenario_name)
q.join()
print("q.join completed")
logStudyData()
print("script complete")
As the docs for Queue.get say:
Remove and return an item from the queue. If optional args block is true and timeout is None (the default), block if necessary until an item is available. If timeout is a positive number, it blocks at most timeout seconds and raises the Empty exception if no item was available within that time. Otherwise (block is false), return an item if one is immediately available, else raise the Empty exception (timeout is ignored in that case).
In other words, there is no way get can ever return None, except by you calling q.put(None) on the main thread, which you don't do.
Notice that the example directly below those docs does this:
for i in range(num_worker_threads):
q.put(None)
for t in threads:
t.join()
The second one is technically necessary, but you usually get away with not doing it.
But the first one is absolutely necessary. You need to either do this, or come up with some other mechanism to tell your workers to quit. Without that, your main thread just tries to exit, which means it tries to join every worker, but those workers are all blocked forever on a get that will never happen, so your program hangs forever.
Building a thread pool may not be rocket science (if only because rocket scientists tend to need their calculations to be deterministic and hard real-time…), but it's not trivial, either, and there are plenty of things you can get wrong. You may want to consider using one of the two already-built threadpools in the Python standard library, concurrent.futures.ThreadPoolExecutor or multiprocessing.dummy.Pool. This would reduce your entire program to:
import concurrent.futures
def work(scenario):
runScenario(scenario)
print(scenario , " is complete")
scenario_list = ['s1','s2','s3','s4','s5','s6','s7','s8','s9','s10','s11','s12']
with concurrent.futures.ThreadPoolExecutor(max_workers=6) as x:
results = list(x.map(work, scenario_list))
print("q.join completed")
logStudyData()
print("script complete")
Obviously you'll still need a lock around any mutable variables you change inside runScenario—although if you're only using a mutable variable there because you couldn't figure out how to return values to the main thread, that's trivial with an Executor: just return the values from work, and then you can use them like this:
for result in x.map(work, scenario_list):
do_something(result)
I've got three different generators, which yields data from the web. Therefore, each iteration may take a while until it's done.
I want to mix the calls to the generators, and thought about roundrobin (Found here).
The problem is that every call is blocked until it's done.
Is there a way to loop through all the generators at the same time, without blocking?
You can do this with the iter() method on my ThreadPool class.
pool.iter() yields threaded function return values until all of the decorated+called functions finish executing. Decorate all of your async functions, call them, then loop through pool.iter() to catch the values as they happen.
Example:
import time
from threadpool import ThreadPool
pool = ThreadPool(max_threads=25, catch_returns=True)
# decorate any functions you need to aggregate
# if you're pulling a function from an outside source
# you can still say 'func = pool(func)' or 'pool(func)()
#pool
def data(ID, start):
for i in xrange(start, start+4):
yield ID, i
time.sleep(1)
# each of these calls will spawn a thread and return immediately
# make sure you do either pool.finish() or pool.iter()
# otherwise your program will exit before the threads finish
data("generator 1", 5)
data("generator 2", 10)
data("generator 3", 64)
for value in pool.iter():
# this will print the generators' return values as they yield
print value
In short, no: there's no good way to do this without threads.
Sometimes ORMs are augmented with some kind of peek function or callback that will signal when data is available. Otherwise, you'll need to spawn threads in order to do this. If threads are not an option, you might try switching out your database library for an asynchronous one.